After we deployed EC2 scaling in production, we noticed that when scaling of the cloud failed (due to the hitting the quota limit of running instances in EC2), the build request got removed, instead of staying in the queue. Because of this several builds which were scheduled for the last night were not executed.
This is the error that caused the problem:
There are not enough free addresses in subnet 'subnet-3862bc4f' to satisfy the requested number of instances. (Service: AmazonEC2; Status Code: 400; Error Code: InsufficientFreeAddressesInSubnet; Request ID: 33a674ac-4248-41a2-8c07-e0d3f04ee567)
So if scaling fails (for any reason), build request should return to the queue and wait until existing resources become available.
This is similar issue as reported earlier: QB-1904
|
