History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3311
Type: Bug Bug
Status: Closed Closed
Resolution: Cannot Reproduce
Priority: Major Major
Assignee: Robin Shen
Reporter: Todd Scholl
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Quickbuild Server Outage

Created: 17/Dec/18 02:12 PM   Updated: 17/Jan/19 06:01 AM
Component/s: None
Affects Version/s: 7.0.16
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
File Attachments: 1. Zip Archive console.log.zip (1012 kb)
2. GZip Archive cv-snapav-p3_QB_Issue_12_17.tra.gz (41 kb)
3. GZip Archive db_connection_gathering.gz (3.30 Mb)
4. Zip Archive issue_12_17_2018.zip (2.04 Mb)

Environment:
Operating System Linux 4.4.114-94.14-default, amd64
OS User Name root
JVM Java HotSpot(TM) 64-Bit Server VM 1.7.0_79, Oracle Corporation
QuickBuild Version 7.0.16 - Fri May 12 09:57:15 EDT 2017
Total Memory 1.99 GB
Used Memory 1.46 GB
Configurations 7717


 Description  « Hide
Support,
On this morning 12/17 at ~ 1 am the GUI for our QB server became non responsive. The logs for the time show a sharp drop in activity, but do not mention any crash. The logs show jobs still launching, but no jobs reached the agents after this time and nothing appears to have gone into the QB database after ~1:06 am. I had to initiate a server restart at 2:34 am to try and recover the server and after that information and jobs started to flow. I executed the "./server.sh dump" command twice be fore restarting.

After recovery, several of the agents displayed errors and would not reconnect to the server. I had to do a series on manual restarts on them until about ~3 am when the remaining troublesome agents just started working without intervention.

I need to know what was causing these issues and what we can do to fix it.

I am attaching logs for the issue though some of them may be too large. Is there a place I can sftp them too for the big ones?

 All   Comments   Work Log   Change History      Sort Order:
Change by Todd Scholl [17/Dec/18 02:54 PM]
Field Original Value New Value
Attachment issue_12_17_2018.zip [ 10741 ]

Change by Todd Scholl [17/Dec/18 03:17 PM]
Attachment cv-snapav-p3_QB_Issue_12_17.tra.gz [ 10742 ]

Change by Todd Scholl [03/Jan/19 06:31 PM]
Attachment db_connection_gathering.gz [ 10743 ]

Change by Todd Scholl [03/Jan/19 09:17 PM]
Attachment console.log.zip [ 10744 ]

Change by Robin Shen [17/Jan/19 06:01 AM]
Status Open [ 1 ] Closed [ 6 ]
Resolution Cannot Reproduce [ 5 ]