History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3867
Type: Bug Bug
Status: Open Open
Priority: Major Major
Assignee: Steve Luo
Reporter: Marcin W
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
QuickBuild

[Restful][changes] Not getting SCM Changes for some builds via /rest/changes

Created: 21/Jun/22 01:33 PM   Updated: 31/Oct/22 04:21 PM
Component/s: None
Affects Version/s: 12.0.2
Fix Version/s: None

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown


 Description  « Hide
For some builds that have SCM Changes visible in web GUI I'm not getting any changes via rest API:
/rest/changes/123?build=456

But this request works fine and returns the expected changes:
/rest/changes/commits/build/456

These builds have finished a few weeks ago and I don't see any obvious differences between them and the ones that work fine.

 All   Comments   Work Log   Change History      Sort Order:
Steve Luo [31/Oct/22 04:21 PM]
Also, you may send me via email (steve at pmease) so you can attach some screen shots to me to let me know more detailed on your issue.

Steve Luo [31/Oct/22 04:19 PM]
Hi Jeremiasz,

It's hard for me to reproduce this issue. Have you tried rebuilding the SCM changes (see my previous post) to make sure the databases are synchronized?

Jeremiasz Macura [31/Oct/22 08:42 AM]
Hi Steve,

What to do in relation to the last message?

Jeremiasz Macura [03/Oct/22 09:43 AM]
Hi Steve,

This only occurred in some specified configurations. The rest of configurations works fine. In the configuration where the problem occurs, I found a dependency, which I mentioned earlier, that only builds that were cancelled or failed save the data in statistics page and the SCM Activities tab and also are seeable under the URL /rest/changes/123?build=456. Other builds that were successful cannot be reach from URL /rest/changes/123?build=456 and leaves no data in statistics page and the SCM Activities tab but they leaves data about SCM changes only under build page in SCM changes tab. So SCM changes in case of successful builds can be reach in build page under SCM changes tab but cannot be reach under statistic page and SCM Changes tab and under URL: /rest/changes/123?build=456.

Creating a test configuration for this case would be very difficult.

Steve Luo [29/Sep/22 11:24 AM]
Hi Jeremiasz,

Is this only occurred in some specified configurations or all of them? Is it possible to create a test setup for me? I can hardly reproduce the issue based on your comments. Thanks in advance.

Jeremiasz Macura [29/Sep/22 10:56 AM]
Hi Steve,

I checked SCM Activities in statistics page and last record is 2 months ago. This is despite the fact that there have been many repo changes and builds ran since Record SCM Changes in Advanced Settings is set to "yes".

Steve Luo [16/Sep/22 01:30 PM]
Hi Jeremiasz,

I have just tested with a new setup and can't find this issue. Please confirm the SCM changes are correct by visiting QuickBuild configuration statistics page and choose the SCM Activities tab.

@Marcin,

-- Maybe there is a method to synchronize these 2 databases somehow?

Yes. Please refer below page on how to rebuild SCM changes via a new configuration with a Rebuild Statistics step:

https://wiki.pmease.com/display/QB12/Rebuild+statistics+and+aggregation+data

before you rebuild statistics, please move the old one to another directory, the SCM changes for a configuration can be located at below directory:

${storarge-dir}/configurations/${id}/reports/scm_activity

Jeremiasz Macura [16/Sep/22 09:10 AM]
Hi,

Some new information about issue:

We have checked if new builds (currently SCM changes are switched on) also returns us no data while we are trying to hit URL /rest/changes/123?build=456.
Those builds which have passed successfully gives us empty dataset under that URL. Those builds which have been cancelled or failed returns us full dataset under that URL.

It looks like the only builds for which URL /rest/changes/123?build=45 is working are those which have failed or have been cancelled.

Marcin W [16/Sep/22 08:26 AM]
Hi,
The main problem for me was actually getting changesets from a range of builds, like this:
https://quickbuild.igk.intel.com/rest/changes/41553?from_build=9001692&to_build=9231692&limit=50
It is not possible with the other URL, or maybe I'm missing something.
I had to do a workaround for this, but it works much slower, so it would be really nice to have this API fixed.
I think that a patch that you've mentioned would not be sufficient for us.
Maybe there is a method to synchronize these 2 databases somehow?

Steve Luo [15/Sep/22 03:04 AM]
Hi Jeremiasz,

The two URLs were designed to do different work, one for fetching changes from build, and another for searching changes from configuration. We don't want to make them ambiguous, hope you can understand.

But, if you just want a temporary solution, I can build a patch release as a workaround for you. But please NOTE, this patch will not exist in our official release, that means after you upgrade your server, the workaround will not work anymore.


Jeremiasz Macura [14/Sep/22 10:02 AM]
Hi Steve,

For new builds, both URLs work fine. When we are trying to receive data from builds that we ran before SCM Changers tab was enabled we receive no data using URL: /rest/changes/123?build=456 but URL /rest/changes/commits/build/456 returns data for those builds.

In theory latter URL should work faster. I don't know how our clients exactly uses those URL but they would like to have previous URL: /rest/changes/123?build=456 to be working for those older builds that were ran when SCM Changers tab was disabled.

By disabled SCM changes tab I mean that probably some time ago in Advanced Settings Record SCM Changes option was disabled for a bit of time after we have migrated project to GitHub. It can be the case why for some builds that were ran at that time URL /rest/changes/123?build=456 is not working while URL /rest/changes/commits/build/456 is working.

if there is a chance that these databases will be synchronized (if of course that would help) that would be great. We would like to have URL /rest/changes/123?build=456 to be working for those builds for which is currently not working.

Also If it would be a problem to fix this URL for those builds then we will use the second URL. But if this is not a big deal then please fix the first URL for these builds.

Steve Luo [13/Sep/22 11:41 AM]
Hi Jeremiasz,

Both URLs /rest/changes/123?build=456 and /rest/changes/commits/build/456 can work. Could you please explain more on fixing for the former URL? Can't that work for you?

And for the latter URL, in theory, it should be a bit faster than the former one as it only read changes from a build database, while latter needs to read changes from a stats database which contains all changes of the specified configuration.

If SCM changes step (do you mean Record Changes Step or Checkout step?) is disabled, QuickBuild are not able to read the changes for those builds.

Jeremiasz Macura [13/Sep/22 10:34 AM]
Is there any chance that a broken API for this URL /rest/changes/123?build=456 will be fixed and usable for those builds before SCM Changers tab enabled? Meanwhile we use the second URL /rest/changes/commits/build/456 but this solution works slower for us and we would like to go back to the first URL.

Steve Luo [25/Aug/22 01:22 PM]
Yes, this can be a case. And also when the build is deleted, the changes can't be fetched either.

Jeremiasz Macura [24/Aug/22 07:37 AM]
After migrating project to GitHub we have diabled SCM Changers tab for some time but now it is abled again. Is it possible that this is the reason for the endpoint /rest/changes/123?build=456 not working for the selected builds?

Steve Luo [23/Jun/22 10:58 AM]
Yes, it's my fault, the URL /rest/changes/123?build=456 should also work. The difference between /rest/changes/123?build=456 and /rest/changes/commits/build/456 is that, the former reads changes from configuration stats database, while the latter reads changes from build database directly. So, maybe, sometimes the two databases are not synchronous correctly. But I don't know the root cause here. If you want to read changes for a specified build, the latter URL (/rest/changes/commits/build/456) may be a better choice.

I'm not very clear what kind of way you are looking for, besides "from_build" and "to_build", you can use "from_date" and "to_date" as filter.

For disabled tab, I also have no idea on this as we don't receive such report so far. Please let me know more details on this.

Marcin W [23/Jun/22 08:38 AM]
I understand, but why is this URL invalid?
I should be able to use this API with parameter 'build' or parameters 'from_build' and 'to_build':
/rest/changes/{configuration: .*}
And it works for most of the cases, but as I mentioned, I'm not getting only some of the changes via this API.
My main goal is getting changes in range of builds by specifying 'from_build' and 'to_build', but I cannot see other way to do this.
Also, I've heard that SCM Changes tab was disabled on our QB for some time, but I'm not sure if this is related.

Steve Luo [23/Jun/22 02:32 AM]
URL below is not valid:

/rest/changes/123?build=456

and, yes, below URL is the correct one for fetching changes from a specific build:
/rest/changes/commits/build/{buildId}

By default, /rest/changes/123 only returns the last 50 commits. You need paging search or specify the size of the result. See more details by visiting below URL:

http://your.quickbuild.host:8810/rest/changes/help

Marcin W [21/Jun/22 01:41 PM]
Also, I don't see these changes when I'm trying to get all of the changes from this configuration:
/rest/changes/123