When we add cloud profiles the build requests need a long time to get processed.

Usually there are around 300 requests in the queue. 133 build agents are running and we tried to use two cloud profiles.

We now removed the cloud profiles because it slows down the build request processing from about 10 seconds to more than 30 seconds.

We have one build which is triggered on every commit (via rest service directly from svn) and the build just starts other builds without waiting for them. The build itself takes 1 second.
Sometimes the configuration gets triggered 5 times in one minute, because there were 5 commits. If we assume that every minute there are 5 commits these will pile up because the previous builds are processed very slowly (only about 2 requests in 1 minute because of the loop taking over 30 seconds).

We had to remove the cloud profiles because of this.
I enabled the TRACE log for the BuildEngine to find that out. With cloud profiles it the cloud checking time is about 27 seconds, without it is max 5 milliseconds.

Please consider performance improvements there. We would like to have more dynamically created and removed build agents because we have very different workload at different times of the day.

Another question is why the credentials are not stored at the cloud profiles. We have different EC2 accounts (or better spoken EC2 and a local cloud solution with EC2 compatible API) and it is not possible to use these in parallel because of the credentials in a file. Also prioritizing cloud profiles would be nice. But first of all the negative performance impact needs to be solved before we can really use cloud profiles. Why not use multithreading configurationwise when processing build requests?