| << Back to previous view |
|
[QB-2283] AWS reporting InvalidInstanceID.NotFound when new agent is started
|
|
| Status: | Resolved |
| Project: | QuickBuild |
| Component/s: | None |
| Affects Version/s: | 5.1.36 |
| Fix Version/s: | 5.1.42, 6.0.4 |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Maikel vd Hurk | Assigned To: | Robin Shen |
| Resolution: | Fixed | Votes: | 0 |
| Remaining Estimate: | Unknown | Time Spent: | Unknown |
| Original Estimate: | Unknown | ||
| Environment: | QuickBuild EC2 plugin | ||
| Description |
|
AWS reporting InvalidInstanceID.NotFound when new instance is started:
"Error allocating node for step 'master': Status Code: 400, AWS Service: AmazonEC2, AWS Request ID: ecd3c270-6302-409b-8edc-a9b92489e856, AWS Error Code: InvalidInstanceID.NotFound, AWS Error Message: The instance ID 'i-0a5d92ed' does not exist" In this situation the instance id does actually exist on AWS side, but during setup there was most likely some eventual consistency of the API validation. This is well possible accordingly to the page of AWS API documentation: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/query-api-troubleshooting.html Checking the QuickBuild Application code it is visible that the following rules are not applied in there which means that the consistency is not guaranteed: * Confirm the state of the resource before you run a command to modify it. Run the appropriate Describe command using an exponential backoff algorithm to ensure that you allow enough time for the previous command to propagate through the system. To do this, run the Describe command repeatedly, starting with a couple of seconds of wait time, and increasing gradually up to five minutes of wait time. * Add wait time between subsequent commands, even if a Describe command returns an accurate response. Apply an exponential backoff algorithm starting with a couple of seconds of wait time, and increase gradually up to about five minutes of wait time. |
| Comments |
| Comment by Maikel vd Hurk [ 12/Jan/15 12:32 PM ] |
| Same code construct is used within 6.0.3 so issue applies to that version as well. |
| Comment by Robin Shen [ 13/Jan/15 12:28 AM ] |
| Is this an recently added requirement of AWS? Previously EC2 plugin works fine. |
| Comment by Maikel vd Hurk [ 13/Jan/15 07:43 AM ] |
| This is added based on the fact that people where facing a not consistent answer from EC2 API resulting in error messages like InvalidInstanceID.NotFound, based on these reports this page is created which is now mentioned several times on stack overflow! |
| Comment by Robin Shen [ 14/Jan/15 12:20 AM ] |
| Thanks for the info. So this is not happening all the time? And can you please attach the full stack trace of this error? |
| Comment by Maikel vd Hurk [ 14/Jan/15 07:01 AM ] |
| It doesn't happen all the time, but in case you are using EC2 frequently then you are seeing this on a larger scale actually. I can abstract later some stacktrace, but in fact this rules should be applied everywhere when you are modifying started instances as this is potentially like for example creating tags and add them to an instance. |