 |
History
| Log In
|
|
|
|
[
Permlink
| « Hide
]
Robin Shen [16/May/13 12:21 AM]
Which cloud environment are you using? For EC2, QB already has a plugin to launch node on demand and shutdown gracefully after using. For other environments, you may write your own plugin to launch/shutdown node using the cloud extension point in QB.
We use Cloudstack and an external tool which based on current load scales up and down our systems. The solution works nicely, and now we would like to apply it to Quickbuild build farm, without the need for writing custom plugins, because the scaling logic is already implemented.
The build agent can be shutdown gracefully by running "./agent.sh stop". As long as the command returns, agent will be stopped.
But this will break the current build, so it is not graceful. What is needed is a way to tell the agent to not accept new builds, so that it would still finish the current ones, and then shut down. Otherwise the build status would be red.
When request to stop, QB agent makes sure that current running steps assigned to it are finished. It can not wait for finish of build as build may contain many steps and it does not know which step will be assigned to it. So generally if you have agents start/stop frequently please make sure that steps are not tired to the agent, that is, a step should not rely on particular agent so that if one of them gets stopped, other agents in the pool can pick up the step. Another approach is to wrap all steps using particular agent with a container step and configure that container step to run on that specific agent, this way, agent will wait for the container step to stop.
I made a test and you are right: at step level everything works as it should i.e. "./agent.sh stop" makes agent exiting in a graceful way.
That's all I need. Thank you for your comments. Please close my request. | |||||||||||||||||||||||||||||||||||||||||||||||
 |
|
New Feature
Closed
Major
Graceful shutdown of agents in auto-scaling cloud environment