History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-142
Type: New Feature New Feature
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Robin Shen
Reporter: Robin Shen
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Add resource management functionality in Quickbuild

Created: 21/Oct/06 05:31 PM   Updated: 02/Mar/10 01:41 AM
Component/s: None
Affects Version/s: None
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown


 Description  « Hide
From Humberto,

To understand the basis for my issues, let me describe my company's build environment at least a little...
1) We use Subversion, Jira, and Ant. All heavily integrated together.

2) We have been doing distributed/automated builds (nightly, manual demand, manual promotion)
for awhile now using a bunch of ant scripts (using antserver to call between servers )
We trigger each build by simply filling in a set of parameters into a very simple ant file that is then run from a desktop
only to trigger a series of ant scripts residing on other servers (it's actually more flexible and maintainable than it sounds)

The cascading ant scripts check stuff out of subversion, build it (using Visual Studio), test it,
and then tag and stamp the build back into subversion, pushing the distribution into an ftp server
and notifying the user by SMTP if anything bad happened (or if a manually triggered build, that the build is complete).
One limitation of ant script remoting is that the ports we use are not exposable to outside the VPN,
so currently we are limited to inside the VPN only.

3) One of the parameters in our ant script is a "slot number" which is a directory that resides on a remote build computer
(the computer address is another parameter) where the build will occur.
The slot represents an actual directory where the build will occur (our builds are transparently relocatable by design)
There is a related slot on a database server (another parameter) where a dataset gets installed for testing the build.
This database server slot represents not just a directory, but a directory + database name + odbc datasource name.
Currently we reserve certain slots for night builds and we need to verbally negotiate for the other slots before triggering a demand build
(since currently we don't have any feedback as to which slot is active)

So basically, a slot is a reserved resource that gets temporarily filled/blocked by a build, but then becomes available afterwards.
Due to our slot based design, we have the capability of building the same configuration concurrently in two different slots
(perhaps because one build started a little bit after the other, after some new files were added - for testing with and without some feature)

---

The upshot of all this is that currently we are looking for a web UI (capable of triggering an Ant file and filling in the parameters)
to allow users (including those from outside the VPN) to manually trigger builds, and to manage the allocation of build slots.
As time goes on, we may even remove some of the cascading functionality back out of the Ant scripts into a builder style application,
and follow a more traditional builder process, if the builder was sufficiently flexible to handle our needs.

In any case, Quickbuild can certainly manage the ant triggering part (this is good)
and it can deal with filling in most of the parameters (this is also good)
but the concept and the implementation of build queues as currently exists in quickbuild,
leaves a bit to be desired in the handling of the concept of slots.

From what I have been able to see, Quickbuild queues only seem to manage a count of threads (which is not that useful for me),
and an aggregation policy for queuing (which is pretty neat for nightly and continuous builds).
Unfortunately, there are no other discernible properties to a build queue.
Furthermore, queues are directly attached to a build configuration, rather than attached at the time of launch..

What I would prefer to see is that queues manage a set of user defined parameters
(like the Variables section at the bottom of the basic settings panel of the Configuration editor)
and I would prefer to have the queue dynamically attach (when launched) to the build queue (based on availability of queues by loading)
The launched build should only be able to select from build queues available to the current user,
or, as defined in a list in the build configuration.

Once the build was "in flight", the parameters sent to ant could be sourced from the selected build queue as well as the build configuration.

The effect would be the equivalent of dynamically selecting a build slot from an allowed set, and queueing the build request into it.

If you were to integrate this concept into the current implementation of Queue, you would need to
1) modify Configurations so they can select a queue from a list at launch time (whenever manually triggered or scheduled)
The selection procedure should be some sort of availability based model.
2) Queues should carry variables, which would become available at runtime to any steps in the build configuration
3) You may also (for more regular build scenarios) want to specify an optional working directory override for the queue.

Personally, I don't see the value in the Queue threadcount (for me it will always be 1, to control 1 build at a time in the queue)
However, that's because of my perception of a Queue as a "slot".
Queue threadcount could potentially be good to limit the maximum number of concurrent builds.
But I see that limit more as a property of the machine as a whole,
and tend to control the total number of available slots in a given machine to manage this.
Perhaps it would be better to manage a machine's concurrent build limit differently.

Oh and another "queue-like" concept that might be useful
would be to have some type of queuing/blocking semaphore in a single step of the build.
We also have a case (for our older development) where one segment of the build
interacts with the registry and blocks the entire machine, but only for that build segment.
The rest of the build is able to proceed independently/concurrently with any other build.


 All   Comments   Work Log   Change History      Sort Order:
Change by Robin Shen [23/Oct/06 05:18 PM]
Field Original Value New Value
Fix Version/s 2.0 [ 10002 ]

Change by Robin Shen [06/Feb/07 05:04 PM]
Comment [ http://clipurl.com/?GCJ260
http://clipurl.com/?CID636
http://clipurl.com/?QJA842
http://clipurl.com/?QDK555
http://clipurl.com/?OJN754
http://clipurl.com/?LFA480
http://clipurl.com/?GQI323
http://clipurl.com/?GBI585
http://clipurl.com/?LFA314
http://clipurl.com/?VPF671
http://clipurl.com/?NRW144
http://clipurl.com/?VEP345
http://clipurl.com/?BXE976
http://clipurl.com/?FLG109
http://clipurl.com/?ADY800
http://clipurl.com/?FTQ923
http://clipurl.com/?WCE140
http://clipurl.com/?EMH991
http://clipurl.com/?TUB211
http://clipurl.com/?BGX661
http://clipurl.com/?OSX593
http://clipurl.com/?TES273
http://clipurl.com/?KEX928
http://clipurl.com/?XLK359
http://clipurl.com/?CAD504
http://clipurl.com/?QCD334
http://clipurl.com/?DUK562
http://clipurl.com/?VCV890
http://clipurl.com/?LPD559
http://clipurl.com/?ESS525
http://clipurl.com/?POG768
http://clipurl.com/?NQR107
http://clipurl.com/?ESD909
http://clipurl.com/?YXX462
http://clipurl.com/?ICA940
http://clipurl.com/?QJJ118
http://clipurl.com/?SCE452
http://clipurl.com/?LLP216
http://clipurl.com/?SGH401
http://clipurl.com/?DAS656
http://clipurl.com/?SJL646
http://clipurl.com/?SXU272
http://clipurl.com/?AVN392
http://clipurl.com/?UQD703
http://clipurl.com/?IGG924
http://clipurl.com/?XMB630
http://clipurl.com/?GPV901
http://clipurl.com/?KBK449
http://clipurl.com/?JRL153
http://clipurl.com/?KJD664
http://shurl.net/2R8
http://shurl.net/2R9
http://shurl.net/2Ra
http://shurl.net/2Rb
http://shurl.net/2Rc
http://shurl.net/2Rd
http://shurl.net/2Re
http://shurl.net/2Rf
http://shurl.net/2Rg
http://shurl.net/2Rh
http://shurl.net/2Ri
http://shurl.net/2Rj
http://shurl.net/2Rk
http://shurl.net/2Rl
http://shurl.net/2Rm
http://shurl.net/2Rn
http://shurl.net/2Ro
http://shurl.net/2Rp
http://shurl.net/2Rq
http://shurl.net/2Rr
http://shurl.net/2Rs
http://shurl.net/2Rt
http://shurl.net/2Ru
http://shurl.net/2Rv
http://shurl.net/2Rw
http://shurl.net/2Rx
http://shurl.net/2Ry
http://shurl.net/2Rz
http://shurl.net/2RA
http://shurl.net/2RB
http://shurl.net/2RC
http://shurl.net/2RD
http://shurl.net/2RE
http://shurl.net/2RF
http://shurl.net/2RG
http://shurl.net/2RH
http://shurl.net/2RI
http://shurl.net/2RJ
http://shurl.net/2RK
http://shurl.net/2RL
http://shurl.net/2RM
http://shurl.net/2RN
http://shurl.net/2RO
http://shurl.net/2RP
http://shurl.net/2RQ
http://shurl.net/2RR
http://shurl.net/2RS
http://shurl.net/2RT
http://shurl.net/2RU
http://shurl.net/2RV
http://linkook.com/115
http://linkook.com/116
http://linkook.com/117
http://linkook.com/118
http://linkook.com/119
http://linkook.com/11a
http://linkook.com/11b
http://linkook.com/11c
http://linkook.com/11d
http://linkook.com/11e
http://linkook.com/11f
http://linkook.com/11g
http://linkook.com/11h
http://linkook.com/11i
http://linkook.com/11j
http://linkook.com/11k
http://linkook.com/11l
http://linkook.com/11m
http://linkook.com/11n
http://linkook.com/11o
http://linkook.com/11p
http://linkook.com/11q
http://linkook.com/11r
http://linkook.com/11s
http://linkook.com/11t
http://linkook.com/11u
http://linkook.com/11v
http://linkook.com/11w
http://linkook.com/11x
http://linkook.com/11y
http://linkook.com/11z
http://linkook.com/11A
http://linkook.com/11B
http://linkook.com/11C
http://linkook.com/11D
http://linkook.com/11E
http://linkook.com/11F
http://linkook.com/11G
http://linkook.com/11H
http://linkook.com/11I
http://linkook.com/11J
http://linkook.com/11K
http://linkook.com/11L
http://linkook.com/11M
http://linkook.com/11N
http://linkook.com/11O
http://linkook.com/11P
http://linkook.com/11Q
http://linkook.com/11R
http://linkook.com/11S
http://win-win.main.jp/kb
http://win-win.main.jp/kc
http://win-win.main.jp/kd
http://win-win.main.jp/ke
http://win-win.main.jp/kf
http://win-win.main.jp/kg
http://win-win.main.jp/kh
http://win-win.main.jp/ki
http://win-win.main.jp/kj
http://win-win.main.jp/kk
http://win-win.main.jp/kl
http://win-win.main.jp/km
http://win-win.main.jp/kn
http://win-win.main.jp/ko
http://win-win.main.jp/kp
http://win-win.main.jp/kq
http://win-win.main.jp/kr ]

Change by Robin Shen [24/Sep/09 08:03 AM]
Fix Version/s trunk [ 10864 ]
Fix Version/s 2.0.0 [ 10002 ]

Robin Shen [02/Mar/10 01:41 AM]
Can now be achieved with QuickBuild's resource scripting API.

Change by Robin Shen [02/Mar/10 01:41 AM]
Status Open [ 1 ] Resolved [ 5 ]
Fix Version/s 2.1.x [ 11102 ]
Fix Version/s trunk [ 10864 ]
Resolution Fixed [ 1 ]