History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-2283
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Blocker Blocker
Assignee: Robin Shen
Reporter: Maikel vd Hurk
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

AWS reporting InvalidInstanceID.NotFound when new agent is started

Created: 12/Jan/15 11:49 AM   Updated: 20/Jan/15 11:51 AM
Component/s: None
Affects Version/s: 5.1.36
Fix Version/s: 5.1.42, 6.0.4

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown
Environment: QuickBuild EC2 plugin


 Description  « Hide
AWS reporting InvalidInstanceID.NotFound when new instance is started:
"Error allocating node for step 'master': Status Code: 400, AWS Service: AmazonEC2, AWS Request ID: ecd3c270-6302-409b-8edc-a9b92489e856, AWS Error Code: InvalidInstanceID.NotFound, AWS Error Message: The instance ID 'i-0a5d92ed' does not exist"

In this situation the instance id does actually exist on AWS side, but during setup there was most likely some eventual consistency of the API validation. This is well possible accordingly to the page of AWS API documentation: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/query-api-troubleshooting.html

Checking the QuickBuild Application code it is visible that the following rules are not applied in there which means that the consistency is not guaranteed:
* Confirm the state of the resource before you run a command to modify it. Run the appropriate Describe command using an exponential backoff algorithm to ensure that you allow enough time for the previous command to propagate through the system. To do this, run the Describe command repeatedly, starting with a couple of seconds of wait time, and increasing gradually up to five minutes of wait time.
* Add wait time between subsequent commands, even if a Describe command returns an accurate response. Apply an exponential backoff algorithm starting with a couple of seconds of wait time, and increase gradually up to about five minutes of wait time.

 All   Comments   Work Log   Change History      Sort Order:
Maikel vd Hurk [14/Jan/15 07:01 AM]
It doesn't happen all the time, but in case you are using EC2 frequently then you are seeing this on a larger scale actually. I can abstract later some stacktrace, but in fact this rules should be applied everywhere when you are modifying started instances as this is potentially like for example creating tags and add them to an instance.

Robin Shen [14/Jan/15 12:20 AM]
Thanks for the info. So this is not happening all the time? And can you please attach the full stack trace of this error?

Maikel vd Hurk [13/Jan/15 07:43 AM]
This is added based on the fact that people where facing a not consistent answer from EC2 API resulting in error messages like InvalidInstanceID.NotFound, based on these reports this page is created which is now mentioned several times on stack overflow!

Robin Shen [13/Jan/15 12:28 AM]
Is this an recently added requirement of AWS? Previously EC2 plugin works fine.

Maikel vd Hurk [12/Jan/15 12:32 PM]
Same code construct is used within 6.0.3 so issue applies to that version as well.