History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3929
Type: Task Task
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Robin Shen
Reporter: Nitin Turkar
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Need information on tables having variable data for quickbuild

Created: 04/Feb/23 06:57 PM   Updated: 03/Jul/23 11:29 PM
Component/s: None
Affects Version/s: None
Fix Version/s: None

Original Estimate: 1 hour Remaining Estimate: 1 hour Time Spent: Unknown
File Attachments: None
Image Attachments:

1. Node_selection_Master_Step.png
(6 kb)


 Description  « Hide
We do have thoursands of workflows setup in Quickbuild in FIS. We have setup variables under Settings/variables/Defined Inherited Variables. We need to look out for a specific value for one of the variables in all the workflows which is time consuming. Hence, we want to know the table details which actually captures the variable names & their values in Quickbuild database. Could you please advise on this?

 All   Comments   Work Log   Change History      Sort Order:
Robin Shen [03/Jul/23 11:29 PM]
Glad to be helpful. Let me know if you have any other questions.

Nitin Turkar [03/Jul/23 12:58 PM]
Hi Robin,

Many thanks for joining the call & helping us out.

We confirm that we can now run the workflow on master and able to achieve the desired results.
Thanks again for your time.

Regards,
Nitin

rani dharne [30/Jun/23 11:31 AM]
Hi Robin,

We have sent you the meeting invite.
please accept is you are available or let us know if need to reschedule.

Robin Shen [30/Jun/23 02:02 AM]
Please schedule a meeting and I will join. I am normally available from 8:00AM to 10:00PM at GMT+8.

rani dharne [29/Jun/23 12:08 PM]
Hi Robin,

Currently we cannot share any sample database as we don't have dummy data ready with us and test evrionment as well.
we are testing the build on our production env only.
is it possible to have screen share with us. we will reproduce the issue.

Let is know your availability. We will connect.


Robin Shen [30/May/23 02:19 PM]
This is odd. Works fine at my side. Can you please reproduce this with a sample database and send to [robin] AT [pmease] DOT [com]?

Nitin Turkar [30/May/23 02:04 PM]
Hi Robin,

Node Selection - On Server

Still it is throwing the same error:
2023-05-30 15:57:06 - Error allocating node for step 'master': Can not find any node matching specified criteria.

Robin Shen [30/May/23 12:07 PM]
Seems that the grid resource "Master_Server" does not have any node included. You may change the node selection to run on server directly without using resource.

Nitin Turkar [30/May/23 10:04 AM]
Node Selection - Master step

Nitin Turkar [30/May/23 10:00 AM]
Hi Robin, It is selected as follows:
Node Selection On node with specified resource
Resource Master_Server

Robin Shen [30/May/23 07:28 AM]
How is the node selection setting specified for the master step?

Nitin Turkar [30/May/23 07:00 AM]
Hi Robin, thanks for your help.

We tried out the above-mentioned suggestions but looks like it is not working for us.
It is throwing following error:

2023-05-30 08:57:27,038 [Thread-72] ERROR com.pmease.quickbuild.DefaultBuildEngine - Error allocating node (configuration: root/~qbtest/blackduck_workflows/update_blackduck_projects, build id: null, step: master)
    com.pmease.quickbuild.QuickbuildException: Can not find any node matching specified criteria.
        at com.pmease.quickbuild.DefaultBuildEngine.run(DefaultBuildEngine.java:1386)
        at java.lang.Thread.run(Thread.java:748)

quickbuild is not letting the master server to be used for the workflow.

could you please suggest if there any other alternative?

Robin Shen [13/Mar/23 11:17 PM]
Unfortunately there is no option to run this on agent. You may create a child step running on server specificially for this purpose.

rani dharne [13/Mar/23 05:21 PM]
 Hi Robin,

We might have some restrictions on running the build on master server. we searched for other option but didn't managed to get through.
Do you have any option instead of running them on master.

Robin Shen [11/Mar/23 12:20 AM]
Any script changing configuration setting including variables need to run at server side. Just configure the step to run on QB server.

To change node assignment rule of master step of a configuration, please use below script (should also run on QB server):

groovy:
import com.pmease.quickbuild.migration.VersionedDocument;

var confToEdit = system.configurationManager.get("path/to/some/configuration");
var stepToEdit = confToEdit.getStep("master");

stepToEdit.setNodeMatcher(...); // pass in appropriate node matcher here
confToEdit.stepDOMs[stepToEdit.name] = VersionedDocument.fromBean(stepToEdit);
system.configurationManager.save(confToEdit);

rani dharne [10/Mar/23 03:26 PM]
Hi Robin,

We have implemented the code to change variable value under specific workflow.
But getting error as "This operation does not allowed on agent side".
Do you have resolution on this.

Also we want to change the node assignment rule for master server configuration.
Is there any special configuration needs to be done from server side for this.

Regards,
Rani

Robin Shen [11/Feb/23 12:34 AM]
Hi Nitin,

You may check properties of different step types here:
https://build.pmease.com/download/5646/html/Javadoc/com/pmease/quickbuild/stepsupport/Step.html


Nitin Turkar [10/Feb/23 04:40 PM]
Hi Robin,

Thanks again for your help. Unfortunately, this didn't work for us this time.

Could you please let us know if you have any documentation somewhere on these methods, steps that we are trying to use in above Groovy script?
Would like to explore it more.

Thanks for your help.

Regards,
Nitin

Robin Shen [09/Feb/23 12:32 AM]
Hi Nitin,

I am vendor of QuickBuild, and do not know details of your setup. I guess your work can be achieved by inspecting variable value, change it as necessary and get it saved like below:

groovy:
import com.pmease.quickbuild.migration.VersionedDocument;

for (eachConf in system.configurationManager.getAll()) {
  logger.info("Variables defined in configuration: " + eachConf.pathName);
  for (eachVar in eachConf.getVariables()) {
    def confChanged = false;
    if (eachVar.value.contains("some special string")) {
      eachVar.setValue("new value");
      confChanged = true;
    }
  }
  if (confChanged)
    system.configurationManager.save(eachConf);
}

To inspect step and save it:

groovy:
for (eachConf in system.configurationManager.getAll()) {
  logger.info("Variables defined in configuration: " + eachConf.pathName);
  var confChanged = false;
  for (eachStep in eachConf.getSteps()) {
    if (eachStep.name.contains("some string")) { // you may also check other setting of the step depending on step type
      eachStep.description = "some new string"; // this is a demo, change other settings as necessary
      eachConf.stepDOMs.put(eachStep.name, VersionedDocument.fromBean(eachStep));
      confChanged = true;
    }
  }
  if (confChanged)
    system.configurationManager.save(eachConf);

}

Nitin Turkar [08/Feb/23 05:17 PM]
Hi Robin,

We have good news here. We are able to fetch all the required data from the variables using the Groovy script provided by you.

I have 2 more queries. Would be great if you can help us here:
1) Using the above groovy script, we are able to fetch the project details now from variables values. As some of the projects are become obsolete, we need to replace them with the new project details. By going to each and every workflow and update them manually would be tedious task and will take a lot of time. Is there any way, we can achieve this in easy manner?
2) We have one overridden step in all the workflows which can't be inherited from root. In this step, we are using one of the jar file in our command. The version of the jar file is changed, and we need to update all the workflows. Could you please advise how can we do it in best feasible way?

Thanks in advance

Regards,
Nitin

Nitin Turkar [06/Feb/23 06:17 PM]
Hi Robin,

Many thanks for sharing this groovy script. We will work on it and update you.

Regards,
Nitin

Robin Shen [05/Feb/23 11:03 PM]
Variables are saved in column qb_variables in table qb_configuration. The database name can be checked by examining "conf/hibernate.properties" under QB server directory. However you can not query the variables directly with SQL as all variables defined in a configuration are saved in a serialized binary form. However you can do that by creating a configuration, with a step running below groovy script:

groovy:
for (eachConf in system.configurationManager.getAll()) {
  logger.info("Variables defined in configuration: " + eachConf.pathName);
  for (eachVar in eachConf.getVariables()) {
    logger.info(eachVar.name + ":" + eachVar.value);
  }
}

You may modify the script to filter variable names to print those you wanted.

Nitin Turkar [05/Feb/23 12:34 PM]
I hope, I have explained the requirement why I am looking out for table & database name. Please let me know if you need more information.

Nitin Turkar [05/Feb/23 12:32 PM]
Hi Robin, there is one variable "farm_blackduck_project_name" which actually contains the blackduck project name for security scanning. Some of the blackduck projects are already crossed the limit of number of Scans but they are still configured in several workflows. If we keep using them in our workflows, our build will fail. Hence, we want to list down all the workflows with such projects. Could you please share more details about table name and database name for this? I want to run some queries and sort out this data for our quick action.

Robin Shen [05/Feb/23 12:34 AM]
Variables are saved together with configuration as binary data in database. However they can be printed into log via groovy scripts. Do you want to list all of them for all configurations?