| << Back to previous view |
|
[QB-3503] Expose grape/ ivy settings
|
|
| Status: | Closed |
| Project: | QuickBuild |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major |
| Reporter: | Benoit Maury-Bouet | Assigned To: | Robin Shen |
| Resolution: | Fixed | Votes: | 0 |
| Remaining Estimate: | Unknown | Time Spent: | Unknown |
| Original Estimate: | Unknown | ||
| File Attachments: |
grapeConfig.xml
loop.png
plugins_ls.png
step_content.png
step_logs.png
|
| Description |
|
Dear all,
We would love to use IVY and grape to build groovy libs and apis we could easily reuse in scripts. so we could use @Grab to point to our gitlab instance. |
| Comments |
| Comment by Robin Shen [ 28/Feb/20 12:43 AM ] |
| I am not familiar with Ivy/grape things. How do you anticipate to use that in QB? Via a special step to call Ivy/grape command? |
| Comment by Benoit Maury-Bouet [ 02/Mar/20 08:03 AM ] |
|
Hi Robin,
Was I did a few months back as a proof of concept : 1. Added ivy-2.5.0-rc1.jar to plugins\com.pmease.quickbuild.libs 2. Create a .groovy folder in quickbuild's user home folder with a grapeConfig.xml 2. Build a groovy library 3. Publish the jar file to artifactory 4. Create a script step using groovy: ---------------------- groovy: @GrabResolver(name='artifactory', root='https://artifactory/artifactory/maven-playground-local/', m2Compatible='true') @Grab("org.ubisoft.bge2:bge2-qb-groovylib:0.4") import org.ubisoft.bge2.JiraTools def jt = new JiraTools(); ---------------------- The issue being this is only doable for scripts run on server right now because agents lacks the grapeconfig.xml file We can of course create our own specific agent setup to make sure this work. QB has it's own defaultGrapeConfig.xml in com.pmease.quickbuild_XXX.jar => groovy-all-1.8.9.jar and if i remeber correctly some paths doesn't exist anymore, so we have to wait for the timeout to fire before we can use our own config file I'm attaching our grapeConfig.xml as a sample |
| Comment by Robin Shen [ 02/Mar/20 11:56 PM ] |
| Looks to me that you want to use new groovy libraries. If so, how about putting the library (jira tool lib here I guess) directly into QB's "plugins/com.pmease.quickbuild.libs" directory? |
| Comment by Maciej M [ 26/Oct/20 09:53 PM ] |
|
Hello,
Is there any update? Actually I'm trying to implement the same case. BR, Maciej |
| Comment by Robin Shen [ 26/Oct/20 11:38 PM ] |
| I'd suggest to put desired groovy libraries into QB's "plugins/com.pmease.quickbuild.libs" directory as suggested above. You may also need to put the whole groovy (groovy-all.jar) into that directory if these libraries depends on new groovy versions. |
| Comment by Maciej M [ 27/Oct/20 01:26 AM ] |
|
Robin, we're aware of this possibility.
Although dynamic @Grab imports are more flexible, we don't have to manage dependencies manually. In our scenario @Grab and import statement makes some kind of infinite loop for script step - we're trying to grab artifact from official maven repository. |
| Comment by Robin Shen [ 27/Oct/20 01:33 AM ] |
| I am not familiar with Ivy/Grab/Grape things. Can you demonstrate how you want to enhance groovy support in QB via an example? |
| Comment by Maciej M [ 27/Oct/20 01:51 AM ] |
|
Sure,
Grape is groovy built-in package manager simillar to Maven. @Grape annotation are resolved during compilation Dependencies are downloaded and cached as normally as Maven would do Reference: http://docs.groovy-lang.org/latest/html/documentation/grape.html Here's sample: [Misc > Execute Script] ---------------------- groovy: @Grab(group='com.squareup.okhttp3', module='okhttp', version='4.9.0') import com.squareup.okhttp.OkHttpClient public final OkHttpClient client = new OkHttpClient(); // Do some fancy API calls ---------------------- ## First test case: 1. As Benoit mentioned I've added --> `1. Added ivy-2.5.0-rc1.jar to plugins\com.pmease.quickbuild.libs` 2. Run QB Script step >> Resulted: Running script above throws: java.lang.NoClassDefFoundError: org/apache/ivy/core/report/ResolveReport ## Second test case: 1. Added manually 'ivy-2.5.0-rc1.jar' to quickbuild-10.0.21.jar/libs/ivy-2.5.0-rc1.jar 2. Edited quickbuild-10.0.21.jar/.classpath adding: <classpathentry exported="true" kind="lib" path="lib/ivy-2.5.0-rc1.jar"/> 3. Run QB Script step >> Resulted: Script runs, but falls into infinite loop without any log to the console ## Third test case: 1. Get into QB machine shell 2. Run grape tool to download dependency manually --> `grape install com.squareup.okhttp3 okhttp` >> Resulted: Success! |
| Comment by Benoit Maury-Bouet [ 27/Oct/20 07:33 AM ] |
|
Hi Maciej,
Glad you tried too :) I have some success on my own, by modifying just the wrapper.conf for both server and agents using an envritonment variable to change the folder where the grapeConfig.xml file is read. Unfortunately there's an issue that bothers me, which is not related to QB but to grape/ivy itself. It works great for me as long as I'm still using java8 as soon as I move to java11 it crashes in grape. From what I see this as been reported as an issue for IVY but I failed to find any communication from the apache foundation on any plan to support this. I added this in both conf/wrapper.conf and agent/buildagent/wrapper.conf set.grape.config=../plugins/com.pmease.quickbuild.bootstrap/grapeConfig.xml then pushed my grapeConfig.xml to plugins/com.pmease.quickbuild.bootstrap so both server and agent get the configuration. for ivy lib I believe just having it in : plugins/com.pmease.quickbuild.libs would work (I didnt had to change the quickbuild-10.0.21.jar/.classpath on my end) |
| Comment by Robin Shen [ 27/Oct/20 08:07 AM ] |
| Thanks for the detailed instructions. I can run your script successfully by copying groovy-3.0.6.jar and ivy-2.5.0-rc1.jar into "<QB install dir>/plugins/com.pmease.quickbuild.libs". You may also need to copy other groovy 3.0.6 utility jars into that folder if necessary. |
| Comment by Maciej M [ 27/Oct/20 08:42 AM ] |
|
So I've reverted to clean QuickBuild 10.0.21, 2020-07-08.
Copied ivy and groovy to libs and still getting `NoClassDefFoundError` I've tried many combinations with groovy lib tho. Tried exact 1.8.9 as QB uses, then 2.4.9, then 3.0.6. |
| Comment by Maciej M [ 27/Oct/20 08:42 AM ] |
| ^Attached screenshots |
| Comment by Benoit Maury-Bouet [ 27/Oct/20 08:48 AM ] |
|
Sorry I realized I failed to mention I also added this in our wrapper.conf (again both server and buildagent)
wrapper.java.classpath.24=../plugins/com.pmease.quickbuild.bootstrap/lib/ivy-2.5.0-rc1.jar (of course .24 is for us , make just make sure no other line is already using that index) |
| Comment by Robin Shen [ 27/Oct/20 08:52 AM ] |
| Please remove groovy-2.4.9.jar from the folder. Otherwise it will conflict with groovy 3.0.6 |
| Comment by Maciej M [ 27/Oct/20 09:00 AM ] |
|
Thanks Benoit!
I'll try this, but currently we've got scheduled maintenance and I can't access the server :( |
| Comment by Maciej M [ 27/Oct/20 09:01 AM ] |
| Yes Robin, I'm keeping only one groovy lib at the same time when testing |
| Comment by Robin Shen [ 27/Oct/20 09:16 AM ] |
|
It is odd that it does not work at your side. I am testing with 10.0.25 on Linux with JDK8. And the step works on both server and agent. I have to modify your script a bit though (to remove public final) otherwise there is a compilation error:
@Grab(group='com.squareup.okhttp3', module='okhttp', version='4.9.0') import com.squareup.okhttp.OkHttpClient OkHttpClient client = new OkHttpClient(); logger.info(client.toString()); |
| Comment by Maciej M [ 27/Oct/20 09:33 AM ] |
|
Tested code without public final keywords resulted the same `NoClassDefFoundError`
Mine setup: QB server running in Docker container ( base Ubuntu 20 image ) running on VM (Ubuntu 20) VM --> Docker --> QB 10.0.21 Will try Benoit's approach with editting wrapper Although I'm a bit confused why Benoit located ivy-2.5.0-rc1.jar in `/com.pmease.quickbuild.bootstrap` location instead of `/com.pmease.quickbuild.libs` Isn't everything in com.pmease.quickbuild.bootstrap resolved by QuickBuild when server is starting? I'm not aware of differences between both dirs. |
| Comment by Robin Shen [ 27/Oct/20 09:39 AM ] |
| Is it possible that your docker container lacks some folder/environment required by Ivy? Please test out side of docker to see if it works at your side. Third party libs are suggested to put in com.pmease.quickbuild.libs as this folder will be synced with agent. Bootstrap jar is used to boot QB and it will not be synced with agent. |
| Comment by Maciej M [ 27/Oct/20 09:25 PM ] |
|
Hello,
Here's quick update: Tested outside of docker and still getting the same exception. Also tried Benoit approach with wrapper.conf and ivy jar in bootstrap dir. Any ideas? |
| Comment by Maciej M [ 27/Oct/20 11:22 PM ] |
|
Update:
I've rebuild docker container and copied ivy jar to "<QB install dir>/plugins/com.pmease.quickbuild.libs" Checked: - user has got ~/.groovy/grapes dir - user has got access to above - verified proxy settings Run example script as Execute Script with DEBUG log level and timeout for 60seconds. Resulted: >> Step is running, but cannot be stopped. Finally, after 8 minutes we've got runtime exception: General error during conversion: Error grabbing Grapes -- [unresolved dependency: com.squareup.okhttp3#okhttp;4.9.0: not found] java.lang.RuntimeException: Error grabbing Grapes -- [unresolved dependency: com.squareup.okhttp3#okhttp;4.9.0: not found] As far as you know is there any ivy/grape default config shaded in QB? Seems like Grab cannot pass through proxy. |
| Comment by Robin Shen [ 27/Oct/20 11:25 PM ] |
| There is no default ivy/grape config shipped with QB. To narrow down the issue, how about testing on a machine with direct access to external network? |
| Comment by Maciej M [ 27/Oct/20 11:28 PM ] |
|
Sorry, I've just realized, that Benoit answered that
Q: As far as you know is there any ivy/grape default config shaded in QB? Seems like Grab cannot pass through proxy. A:QB has it's own defaultGrapeConfig.xml in com.pmease.quickbuild_XXX.jar => groovy-all-1.8.9.jar In wrapper.conf set.grape.config=../plugins/com.pmease.quickbuild.bootstrap/grapeConfig.xml then pushed my grapeConfig.xml to plugins/com.pmease.quickbuild.bootstrap |
| Comment by Robin Shen [ 27/Oct/20 11:34 PM ] |
| I am not aware of grape config shipped in groovy-all-1.8.9.jar. However if groovy-3.0.6.jar is put in com.pmease.quickbuild.libs folder, QB will not be using groovy-all-1.8.9.jar. |
| Comment by Maciej M [ 27/Oct/20 11:44 PM ] |
|
Actually, how can I ensure, that QB uses groovy 3.0.6?
Done test case: groovy: logger.info(GroovySystem.version) >> Resulted: 23:43:18,708 INFO - 1.8.9 |
| Comment by Robin Shen [ 27/Oct/20 11:47 PM ] |
| At my side, it is printing 3.0.6. Since QB does not have official docker image. Please test it on bare metal machine first to narrow down issue. |
| Comment by Benoit Maury-Bouet [ 28/Oct/20 08:26 AM ] |
|
Yes in groovy-all there is a default grape config file in :
groovy\grape\defaultGrapeConfig.xml unfortunately it contains an url to a server no longer running. Maciej is your docker running a java 8 vm ? as I mentionned earlier there are issues with versions >= java 11 Also are you confident your grapeconfig is ok ? (did you check with a regular script outside of QB) In my groovy scripts I also add the grabresolver option : @GrabResolver(name='artifactory', root='https://artifactory/maven-playground', m2Compatible='true') @Grab("org.company.test:JiraTools:0.0.0.2") in case it helps |
| Comment by Maciej M [ 28/Oct/20 11:56 AM ] |
|
I'm pretty sure it's almost done... Probably proxy settings are missing.
Benoit: Checked grapeConfig, it's fine. We're running JDK 1.8 Robin: Is there any proxy related setting that QB reads? I've set proxy settings to both env. variables: JAVA_OPTS, ANT_OPTS To re-produce QB issue I've implemented clean groovy project in my IDEA. Run 3 tests: 1. IntelliJ - Build while connected with VPN: Error grabbing Grapes -- [unresolved dependency: com.google.code.google-collections#google-collect;snapshot-20080530: not found] 2. IntelliJ - Build while disconnected from VPN: Success! 3. Groovy shell - `groovy test.groovy` - while connected to VPN Success! |
| Comment by Maciej M [ 28/Oct/20 12:43 PM ] |
|
Hello!
I've found working solution :) I'm so thankful for your help! Last piece was adding proxy settings to wrapper.conf like so: #wrapper.java.additional.25=-Dhttp.proxySet=true #wrapper.java.additional.26=-Dhttp.proxyHost=<the proxy host> #wrapper.java.additional.27=-Dhttp.proxyPort=<the proxy port> |
| Comment by Robin Shen [ 28/Oct/20 11:13 PM ] |
| Glad to see it is working now. :) |