<< Back to previous view

[QB-2429] Quickbuild server massively consumes hardware resources after 10days
Created: 05/May/15  Updated: 17/Jan/18

Status: Closed
Project: QuickBuild
Component/s: None
Affects Version/s: 6.0.10
Fix Version/s: None

Type: Bug Priority: Blocker
Reporter: AlSt Assigned To: Robin Shen
Resolution: Cannot Reproduce Votes: 1
Remaining Estimate: Unknown Time Spent: Unknown
Original Estimate: Unknown
Environment: Ubuntu 12.04.5 LTS

File Attachments: JPEG File ruxit_buildmaster_cpu_usage.jpg     JPEG File testQuickbuildPerformance.jpg    

 Description   
After approx 10 days quickbuild server performance does down massively and server is constantly consuming hardware resources. Time for request processing goes up from below 1 sec to 40+ sec and a test configuration for that purpose runs takes up to 10 min, while it normally takes about 20 sec. That build took 3 min 46 sec with no build running and after restarting qb server it took 18 sec.



 Comments   
Comment by Robin Shen [ 05/May/15 11:20 PM ]
While this is happening, please check if heap memory of QB is used up, also please check the build queue page to see how many builds are queued there.
Comment by Maikel vd Hurk [ 06/May/15 05:48 AM ]
In fact we noticed similar problems with lower QB 6 version. The Cpu is exhausted and heap is also around 90% constantly in case the problem occurs. There is no other way out except restanten serve, even forcing garbage collects on the JVM process have no effect.
Comment by Robin Shen [ 06/May/15 06:31 AM ]
How many build requests exist in queue in this case?
Comment by Maikel vd Hurk [ 07/May/15 08:09 PM ]
Our queue was containing ~500 items at the point in time it happened. Cleaning whole queue didn't resolve the problem, only full restart helped. There were constant full garbage collects visible in the logging of the JVM.
Comment by AlSt [ 07/May/15 11:01 PM ]
The queue constantly contains between 150 and 250 build requests. The number of request at the time the problem started, was once approx. 250, the second time approx. 180.
Comment by Robin Shen [ 07/May/15 11:30 PM ]
Do you have any RESTful programs frequently pulling configuration data from QuickBuild? If yes, disable them temporarily or reduce frequency to see if it helps. Another reason can be that some custom script loading configuration to do some heavy jobs frequently to result in a lot of memory fragments to cause GC busy.
Comment by Robin Shen [ 07/May/15 11:30 PM ]
This is especially true when there are many configurations in the system.
Comment by AlSt [ 11/May/15 09:59 AM ]
We are doing all of the mentioned points:
* we are frequently accessing configuration data by rest calls
* we use custom groovy scripts accessing approx 600-700 configurations frequently
* we do have many configurations (approx 2200 configurations)

But the strange thing is, that it works fine for approx 10 days without any performance/memory issues, before quickbuild server process starts consuming resources (memory and cpu) exponentially.
Comment by Robin Shen [ 11/May/15 11:21 PM ]
Please take a memory dump with jmap to see what is consuming most of the memory when this happens.
Generated at Thu May 02 17:15:36 UTC 2024 using JIRA 189.