[#QB-2762] Activated cloud profile slows down QB main loop (in BuildEngine)

QuickBuild

Activated cloud profile slows down QB main loop (in BuildEngine)

Created: 20/Jul/16 06:42 AM Updated: 19/Jan/19 01:09 PM

Component/s:

None

Affects Version/s:

6.1.18

Fix Version/s:

None

Original Estimate:

Unknown

Remaining Estimate:

Unknown

Time Spent:

Unknown

Description

« Hide

When we add cloud profiles the build requests need a long time to get processed.

Usually there are around 300 requests in the queue. 133 build agents are running and we tried to use two cloud profiles.

We now removed the cloud profiles because it slows down the build request processing from about 10 seconds to more than 30 seconds.

We have one build which is triggered on every commit (via rest service directly from svn) and the build just starts other builds without waiting for them. The build itself takes 1 second.
Sometimes the configuration gets triggered 5 times in one minute, because there were 5 commits. If we assume that every minute there are 5 commits these will pile up because the previous builds are processed very slowly (only about 2 requests in 1 minute because of the loop taking over 30 seconds).

We had to remove the cloud profiles because of this.
I enabled the TRACE log for the BuildEngine to find that out. With cloud profiles it the cloud checking time is about 27 seconds, without it is max 5 milliseconds.

Please consider performance improvements there. We would like to have more dynamically created and removed build agents because we have very different workload at different times of the day.

Another question is why the credentials are not stored at the cloud profiles. We have different EC2 accounts (or better spoken EC2 and a local cloud solution with EC2 compatible API) and it is not possible to use these in parallel because of the credentials in a file. Also prioritizing cloud profiles would be nice. But first of all the negative performance impact needs to be solved before we can really use cloud profiles. Why not use multithreading configurationwise when processing build requests?

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Robin Shen [20/Jul/16 11:18 PM]

We can add credentials setting in profile, and use default credential in global EC2 setting if it is not defined. As to negative performance impact, it is possible that if the build engine has to launch cloud node frequently. However in peak hours, the node should not be launched frequently unless you've defined a quite short idle time in cloud profile to cause node being terminated/re-launched frequently.

Build request processing itself is multi-threading, but at least we need to have a single thread main loop to go through the build requests and allocate resources one by one.

[ Permlink | « Hide ]

AlSt [21/Jul/16 05:20 AM]

The thing is that we only have few builds which should run in amazon. The resource is pretty specific. So on every request processing run all currently queued requests are checked against the cloud profile and that takes a long time even when no resource (just for 1-2 configurations) will ever match on the cloud profiles.

[ Permlink | « Hide ]

Robin Shen [21/Jul/16 06:03 AM]

I checked the code and the only possible reason for the long loop is because that new EC2 nodes are launched in the loop. And this happens only when no existing nodes/available resources matches the build requirement, and some cloud profiles matches. Can you please check if this is the case at your side?