[#QB-2487] GC overhead limit exceeded on ArtifactCleanupTask

QuickBuild

GC overhead limit exceeded on ArtifactCleanupTask

Created: 15/Jul/15 07:52 AM Updated: 01/Aug/15 12:30 AM

Component/s:

None

Affects Version/s:

5.1.40

Fix Version/s:

6.0.24

Original Estimate:

Unknown

Remaining Estimate:

Unknown

Time Spent:

Unknown

Description

« Hide

The artifact cleanup task does so much work (allocating a lot of objects and maybe also run a lot in parallel).

During the execution of the task the request queue gets much slower and a lot of builds remain in the CHECKING_BUILD_CONDITION status.

Also we had a lot of configurations which had the red error field on top stating "GC overhead limit exceeded".

Can you please look into this.
Yesterday we had to do a emergency restart of QB server because of that.
Today it was again but luckily it recovered and works normal again.

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Robin Shen [16/Jul/15 12:35 AM]

How many configurations do you have? Currently QB has to load all configurations and iterate over them to do artifact cleanup for each configuration.

[ Permlink | « Hide ]

AlSt [16/Jul/15 06:53 AM]

Currently we have 16437 configurations.

[ Permlink | « Hide ]

Robin Shen [16/Jul/15 10:37 PM]

Oh, that is a lot of, will investigate to see if we can find a better approach to handle artifact cleanup of so many configurations with a resonable memory overhead.

[ Permlink | « Hide ]

AlSt [28/Jul/15 06:47 AM]

Currently we also get some "OutOfMemoryError: Java heap space" while checking build condition during the artifact cleanup task.
Did you already had a chance to look into this?

[ Permlink | « Hide ]

Robin Shen [28/Jul/15 11:46 PM]

We will try to get the improvement into next patch release which will be ready before next week.