History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-142
Type: New Feature New Feature
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Robin Shen
Reporter: Robin Shen
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Add resource management functionality in Quickbuild

Created: 21/Oct/06 05:31 PM   Updated: 02/Mar/10 01:41 AM
Component/s: None
Affects Version/s: None
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown


 Description  « Hide
From Humberto,

To understand the basis for my issues, let me describe my company's build environment at least a little...
1) We use Subversion, Jira, and Ant. All heavily integrated together.

2) We have been doing distributed/automated builds (nightly, manual demand, manual promotion)
for awhile now using a bunch of ant scripts (using antserver to call between servers )
We trigger each build by simply filling in a set of parameters into a very simple ant file that is then run from a desktop
only to trigger a series of ant scripts residing on other servers (it's actually more flexible and maintainable than it sounds)

The cascading ant scripts check stuff out of subversion, build it (using Visual Studio), test it,
and then tag and stamp the build back into subversion, pushing the distribution into an ftp server
and notifying the user by SMTP if anything bad happened (or if a manually triggered build, that the build is complete).
One limitation of ant script remoting is that the ports we use are not exposable to outside the VPN,
so currently we are limited to inside the VPN only.

3) One of the parameters in our ant script is a "slot number" which is a directory that resides on a remote build computer
(the computer address is another parameter) where the build will occur.
The slot represents an actual directory where the build will occur (our builds are transparently relocatable by design)
There is a related slot on a database server (another parameter) where a dataset gets installed for testing the build.
This database server slot represents not just a directory, but a directory + database name + odbc datasource name.
Currently we reserve certain slots for night builds and we need to verbally negotiate for the other slots before triggering a demand build
(since currently we don't have any feedback as to which slot is active)

So basically, a slot is a reserved resource that gets temporarily filled/blocked by a build, but then becomes available afterwards.
Due to our slot based design, we have the capability of building the same configuration concurrently in two different slots
(perhaps because one build started a little bit after the other, after some new files were added - for testing with and without some feature)

---

The upshot of all this is that currently we are looking for a web UI (capable of triggering an Ant file and filling in the parameters)
to allow users (including those from outside the VPN) to manually trigger builds, and to manage the allocation of build slots.
As time goes on, we may even remove some of the cascading functionality back out of the Ant scripts into a builder style application,
and follow a more traditional builder process, if the builder was sufficiently flexible to handle our needs.

In any case, Quickbuild can certainly manage the ant triggering part (this is good)
and it can deal with filling in most of the parameters (this is also good)
but the concept and the implementation of build queues as currently exists in quickbuild,
leaves a bit to be desired in the handling of the concept of slots.

From what I have been able to see, Quickbuild queues only seem to manage a count of threads (which is not that useful for me),
and an aggregation policy for queuing (which is pretty neat for nightly and continuous builds).
Unfortunately, there are no other discernible properties to a build queue.
Furthermore, queues are directly attached to a build configuration, rather than attached at the time of launch..

What I would prefer to see is that queues manage a set of user defined parameters
(like the Variables section at the bottom of the basic settings panel of the Configuration editor)
and I would prefer to have the queue dynamically attach (when launched) to the build queue (based on availability of queues by loading)
The launched build should only be able to select from build queues available to the current user,
or, as defined in a list in the build configuration.

Once the build was "in flight", the parameters sent to ant could be sourced from the selected build queue as well as the build configuration.

The effect would be the equivalent of dynamically selecting a build slot from an allowed set, and queueing the build request into it.

If you were to integrate this concept into the current implementation of Queue, you would need to
1) modify Configurations so they can select a queue from a list at launch time (whenever manually triggered or scheduled)
The selection procedure should be some sort of availability based model.
2) Queues should carry variables, which would become available at runtime to any steps in the build configuration
3) You may also (for more regular build scenarios) want to specify an optional working directory override for the queue.

Personally, I don't see the value in the Queue threadcount (for me it will always be 1, to control 1 build at a time in the queue)
However, that's because of my perception of a Queue as a "slot".
Queue threadcount could potentially be good to limit the maximum number of concurrent builds.
But I see that limit more as a property of the machine as a whole,
and tend to control the total number of available slots in a given machine to manage this.
Perhaps it would be better to manage a machine's concurrent build limit differently.

Oh and another "queue-like" concept that might be useful
would be to have some type of queuing/blocking semaphore in a single step of the build.
We also have a case (for our older development) where one segment of the build
interacts with the registry and blocks the entire machine, but only for that build segment.
The rest of the build is able to proceed independently/concurrently with any other build.


 All   Comments   Work Log   Change History      Sort Order:
Robin Shen [02/Mar/10 01:41 AM]
Can now be achieved with QuickBuild's resource scripting API.