<< Back to previous view

[QB-4260] BuildResource.query() accumulates entities in StatefulPersistenceContext without eviction during permission-filtered pagination
Created: 20/Mar/26  Updated: 20/Mar/26

Status: Open
Project: QuickBuild
Component/s: None
Affects Version/s: 15.0.42
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Juan Fernando Alvarado Zamora Assigned To: Robin Shen
Resolution: Unresolved Votes: 0
Remaining Estimate: Unknown Time Spent: Unknown
Original Estimate: Unknown
Environment: QuickBuild version 15.0.26 (custom src build)
OS Linux amd64, kernel 6.8.0-87-generic
Java Oracle JDK 21.0.3+7-LTS-152
CPUs 32
Jetty max threads 512
Jetty port 8810
Connection pool HikariCP
DB PostgreSQL


 Description   
We are experiencing recurring OutOfMemoryError / heap exhaustion on our QuickBuild server (32 GB heap, 32 CPUs, PostgreSQL backend). Heap dump analysis with Eclipse MAT shows 5 Jetty request threads each
retaining 5-7 GB in org.hibernate.engine.internal.StatefulPersistenceContext, totaling ~28 GB (87% of heap). Each thread holds 50,000+ com.pmease.quickbuild.model.Build entities with ~8 million StepRuntime
objects.

 Comments   
Comment by Juan Fernando Alvarado Zamora [ 20/Mar/26 10:29 PM ]
  Root Cause Analysis:

  After decompiling BuildResource.query(), we identified a permission-filtered pagination loop that accumulates Hibernate entities without eviction:

  // BuildResource.query() — simplified
  builds = new ArrayList();
  while (true) {
      List result = BuildManager.instance.search(criteria, first, count);
      for (Build build : result) {
          if (!SecurityHelper.hasPermission(build.getConfiguration())) continue;
          builds.add(build);
      }
      if (builds.size() < count && result.size() >= count) {
          first += result.size();
          continue; // Next iteration loads more entities into same session
      }
      break;
  }

  The session lifecycle in DefaultServletContainer.service() opens the Hibernate session before the request and closes it only after Jersey serializes the response:

  // DefaultServletContainer.service()
  SessionManager.openSession();
  try {
      // ... authentication ...
      super.service(request, response); // BuildResource.query() + serialization
  } finally {
      SessionManager.closeSession(); // Only closed after full serialization
  }
Comment by Juan Fernando Alvarado Zamora [ 20/Mar/26 10:29 PM ]
  The problem: When a service account queries with count=50 but has restricted permissions across a large configuration tree, the while-loop iterates many times — each loading count more Build entities (with
  full StepRuntime graphs via lazy-load) into the StatefulPersistenceContext. There is:

  1. No session.clear() between loop iterations — rejected entities remain in the persistence context
  2. No upper bound on total iterations — maxReturnEntitiesViaRest (QB-3717) only validates the requested count, not the total entities loaded during permission filtering
  3. No eviction of permission-denied entities — builds that fail SecurityHelper.hasPermission() stay in session

  After the loop, accessing getStepRuntimes() and getRepositoryRuntimes() triggers lazy-load of PersistentBag collections, materializing the full entity graph while the session is still open.
Comment by Juan Fernando Alvarado Zamora [ 20/Mar/26 10:30 PM ]
  Reproduction scenario:

  - Service account with limited permissions polls /rest/builds?configuration_id=X&count=50&status=SUCCESSFUL across many configurations
  - Each response carries ~1 MB due to large StepRuntime payloads
  - High polling frequency (e.g. every 5 minutes across 40+ configurations) compounds the issue
  - If permission filtering rejects many results, a single request can load tens of thousands of Build entities before the session closes
Comment by Juan Fernando Alvarado Zamora [ 20/Mar/26 10:30 PM ]
  Suggested fix:

  Add session.clear() at the end of each while-loop iteration in BuildResource.query() after collecting permitted builds into a detached list, and add a hard cap on total entities loaded (not just the requested
   count). For example:

  Session session = SessionManager.getSession();
  int totalLoaded = 0;
  int maxTotalEntities = 10000; // configurable cap

  while (true) {
      List result = BuildManager.instance.search(criteria, first, count);
      for (Build build : result) {
          if (!SecurityHelper.hasPermission(build.getConfiguration())) continue;
          builds.add(build);
      }
      totalLoaded += result.size();
      session.clear(); // Evict loaded entities between iterations

      if (builds.size() < count && result.size() >= count && totalLoaded < maxTotalEntities) {
          first += result.size();
          continue;
      }
      break;
  }

  Note: session.clear() would require re-attaching or copying the permitted builds to a detached collection before clearing, since cleared entities become detached.
Comment by Juan Fernando Alvarado Zamora [ 20/Mar/26 10:31 PM ]
  Heap dump evidence:

  - Thread qtp55085102-38184087: retained 7.2 GB, accumulation point StatefulPersistenceContext, 51,995 Build objects, 227,253 PersistentBag, 8,359,018 StepRuntime
  - 4 additional threads with identical pattern (5.9 GB, 5.4 GB, 5.1 GB, 4.6 GB)
  - All 5 threads blocked at BuildResource.query() > AbstractEntityManager.searchEntities() > CriteriaImpl.list() > PostgreSQL socket read

  Related: QB-3717 (maxReturnEntitiesViaRest) partially addresses this but only caps the requested count, not total loaded entities during permission filtering.
Generated at Sun Mar 22 03:47:31 UTC 2026 using JIRA 189.