History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-4069
Type: Improvement Improvement
Status: Open Open
Priority: Major Major
Assignee: Robin Shen
Reporter: Robin Shen
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Clean up live log entries to save memory for builds producing a lot of logs

Created: 24/Jan/24 12:08 AM   Updated: 24/Jan/24 12:09 AM
Component/s: None
Affects Version/s: None
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown


 Description  « Hide
We have some long running tasks in our Quickbuild environment (build steps running 6-7 hours straight). They produce between 300-500MB of standard output/error during that time. The agents assigned to run these long running tasks are only running a single build step during most of that 6-7 hour period.

We have been seeing, consistently, on these long running tasks is the agents assigned run out of heap memory (even after automatic JVM GC'ing attempts).

We setup a secondary test, running the same long running task, where the only modification was piping standard out/error to a file (instead of to the build step console). This secondary test ran w/o issue and the memory consumption of the JVM looked minimal.

We inspected the source code of com.pmease.quickbuild.log.BuildLogger (we are running version 10.0.39) and didn't see any code that would suggest build entry objects would remain in memory once they've been serialized to disk.

However, we saw that the entries were sent over to class com.pmease.quickbuild.log.DefaultLivelogManager . Once the build step entries enter this class, they are only ever able to be GC'ed once the build completes.

The "logEntries" map / list will continue to grow until the build / build step completes.

For long running build steps, this means the entire log of the build step remains in memory (which in our case, could equate to over 1GB of data). We ran the QB agent code in an IDE and confirmed that the "logEntries" map / list only ever grows until the build, build step completes.

We were wondering if you'd be willing to hotfix QB version 10 to fix this issue (we suspect other versions are impacted).

Our recommendation would be to only hold the last X entries in the map/list at any given time ( perhaps same value as Constants.MAX_LOG_ENTRIES ). This would allow build entries to be orphaned and GC'ed properly.

 All   Comments   Work Log   Change History      Sort Order:
Change by Robin Shen [24/Jan/24 12:09 AM]
Field Original Value New Value
Description We have some long running tasks in our Quickbuild environment (build steps running 6-7 hours straight). They produce between 300-500MB of standard output/error during that time. The agents assigned to run these long running tasks are only running a single build step during most of that 6-7 hour period.

We have been seeing, consistently, on these long running tasks is the agents assigned run out of heap memory (even after automatic JVM GC'ing attempts).

We setup a secondary test, running the same long running task, where the only modification was piping standard out/error to a file (instead of to the build step console). This secondary test ran w/o issue and the memory consumption of the JVM looked minimal.

We inspected the source code of com.pmease.quickbuild.log.BuildLogger (we are running version 10.0.39) and didn't see any code that would suggest build entry objects would remain in memory once they've been serialized to disk.

However, we saw that the entries were sent over to class com.pmease.quickbuild.log.DefaultLivelogManager . Once the build step entries enter this class, they are only ever able to be GC'ed once the build completes.

The "logEntries" map / list will continue to grow until the build / build step completes.

For long running build steps, this means the entire log of the build step remains in memory (which in our case, could equate to over 1GB of data). We ran the QB agent code in an IDE and confirmed that the "logEntries" map / list only ever grows until the build, build step completes.

We were wondering if you'd be willing to hotfix QB version 10 to fix this issue (we suspect other versions are impacted).

Our recommendation would be to only hold the last X entries in the map/list at any given time ( perhaps same value as Constants.MAX_LOG_ENTRIES ). This would allow build entries to be orphaned and GC'ed properly.