| << Back to previous view |
|
[QB-2644] Step Composition Wrapper
|
|
| Status: | Resolved |
| Project: | QuickBuild |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.0 |
| Type: | New Feature | Priority: | Major |
| Reporter: | J. Mash | Assigned To: | Unassigned |
| Resolution: | Fixed | Votes: | 1 |
| Remaining Estimate: | Unknown | Time Spent: | Unknown |
| Original Estimate: | Unknown | ||
| File Attachments: |
parallel-with-threadcount.png
|
| Description |
|
Greetings,
I have a couple different build steps that make heavy use of the 'repeat parameters' feature to execute a given build task dozens and, in some cases, hundreds of times, all of which currently occur in a sequential composition wrappers. The work these build tasks are doing isn't resource intensive, so it would be nice if there were a step composition type that could enable parallel processing of repeated parameters using some defined number of threads/workers. Please let me know if you need any further description on this. -J |
| Comments |
| Comment by Robin Shen [ 26/Feb/16 12:10 AM ] |
| If it is not resource intensive, I would suggest to write a custom step plugin to run the parallel logic inside the step itself. QB parallel composition is not designed to serve as a thread utility. |
| Comment by J. Mash [ 26/Feb/16 04:51 PM ] |
| Right, that's basically what this request is for; a new step type (a composition type seemed most apropos to me, but I defer to you guys on that) to handle running parallel logic based on settings defined in this new step type. As a concept, it seems generic and useful enough to be part of the QB core, rather than just some random one-off in our project. |
| Comment by Robin Shen [ 12/Apr/16 02:45 AM ] |
| How about to define a grid resource say "threads" with your desired count on desired node, and then set your build step to require that resource? You can then put that repeatable build step (the build itself can be a sequential container executing other child steps) into a parallel container. It is much flexible than a simple thread bounded step, as it can also be distributed on different nodes if you like. |
| Comment by J. Mash [ 12/Apr/16 05:00 AM ] |
|
Alright, this ticket started life out as a forum post, and I think I wasn't as clear as I could have been when making it, so allow me to do that now.
This request is not strictly about implementing a parallel / thread-count composition step, especially given that there are at least three different ways I can think of to emulate threads. Rather, this request is for a built-in composition construct that provides a job queue / worker pattern that can be used to dynamically parallelize tasks that aren't very resource intensive. This is something that we could use in half a dozen or so different build steps to great effect. Consider that I currently have a sequential composition with an embedded task that iterates over roughly 300 repeated parameters. Each iteration may take between 5 and 60 seconds to complete, so evenly collating them into groups won't have the desired effect because it's quite plausible that one "thread" could wind up with a disproportionately large number of longer-running jobs. What we need is the ability to define a complete body of work up front (job queue) that can be accessed by embedded steps to obtain a new job when they finish their previous job. To give you an idea of why I feel this is worthwhile, I've mocked this up in a test environment using groovy scripts and times were reduced by 85% for each of the places this would be used. The total time reduction overall in our process is around 25%. |
| Comment by J. Mash [ 12/Apr/16 05:26 AM ] |
|
Also, to add a bit more information as to why this is a request for a new composition step rather than just using the groovy script scenario:
1. This mockup simply collated the body of work into buckets stored in variables so child steps could access that data, ultimately resulting in scenarios where one specific worker would wind up with a disproportionate number of longer-running jobs. When the jobs stacked up this way, the build times were only reduced by 20-35%. Sure, the builds were faster, but not nearly as efficient as the could/should have been. 2. This particular implementation requires that we copy/paste or otherwise create custom groovy scripts for each location in the build hierarchy that we want to use it. That could easily become a maintenance nightmare. |
| Comment by Robin Shen [ 13/Apr/16 01:00 AM ] |
| This seems to me working much like a JDK fixed thread pool. However considering the complexity of introducing another type of container step (currently many QB logic only anticipates two type of container, sequential or parallel), we will only implement this if this ticket gets enough vote from different companies. Sorry about the decision. |