| << Back to previous view |
|
[QB-4069] Clean up live log entries to save memory for builds producing a lot of logs
|
|
| Status: | Open |
| Project: | QuickBuild |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major |
| Reporter: | Robin Shen | Assigned To: | Robin Shen |
| Resolution: | Unresolved | Votes: | 0 |
| Remaining Estimate: | Unknown | Time Spent: | Unknown |
| Original Estimate: | Unknown | ||
| Description |
|
We have some long running tasks in our Quickbuild environment (build steps running 6-7 hours straight). They produce between 300-500MB of standard output/error during that time. The agents assigned to run these long running tasks are only running a single build step during most of that 6-7 hour period.
We have been seeing, consistently, on these long running tasks is the agents assigned run out of heap memory (even after automatic JVM GC'ing attempts). We setup a secondary test, running the same long running task, where the only modification was piping standard out/error to a file (instead of to the build step console). This secondary test ran w/o issue and the memory consumption of the JVM looked minimal. We inspected the source code of com.pmease.quickbuild.log.BuildLogger (we are running version 10.0.39) and didn't see any code that would suggest build entry objects would remain in memory once they've been serialized to disk. However, we saw that the entries were sent over to class com.pmease.quickbuild.log.DefaultLivelogManager . Once the build step entries enter this class, they are only ever able to be GC'ed once the build completes. The "logEntries" map / list will continue to grow until the build / build step completes. For long running build steps, this means the entire log of the build step remains in memory (which in our case, could equate to over 1GB of data). We ran the QB agent code in an IDE and confirmed that the "logEntries" map / list only ever grows until the build, build step completes. We were wondering if you'd be willing to hotfix QB version 10 to fix this issue (we suspect other versions are impacted). Our recommendation would be to only hold the last X entries in the map/list at any given time ( perhaps same value as Constants.MAX_LOG_ENTRIES ). This would allow build entries to be orphaned and GC'ed properly. |