History | Log In     View a printable version of the current page.  
Issue Details (XML | Word | Printable)

Key: QB-3406
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Unassigned
Reporter: Sebastian Potasiak
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
QuickBuild

Configuration fails when there's an additional non-existent remote repository added to git repository

Created: 12/Jun/19 10:57 AM   Updated: 17/Jun/19 06:32 AM
Component/s: None
Affects Version/s: 7.0.28
Fix Version/s: 9.0.12

Original Estimate: Unknown Remaining Estimate: Unknown Time Spent: Unknown


 Description  « Hide
For a git repository defined in QuickBuild we have to add an extra remote repository, but when the address of the repository is wrong or the remote repository doesn't exist anymore, we get following error when trying to run the configuration (it's thrown before checking build condition):


2019-06-12 10:19:07 - Failed to run command: git fetch --all --tags -q
Command return code: 1
Command error output: error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.

fatal: remote error: Repository not found
The requested repository does not exist, or you do not have permission to
access it.
error: Could not fetch external_repository
error: The last gc run reported the following. Please correct the root cause
and remove .git/gc.log.
Automatic cleanup will not be performed until the file is removed.

warning: There are too many unreachable loose objects; run 'git prune' to remove them.


In my opinion, since QuickBuild lets us define only one (origin) remote repository, it shouldn't try to fetch all remotes, but only the origin. Otherwise, we cannot do anything about this issue, as there's no way to handle errors thrown before checking build condition is triggered.

 All   Comments   Work Log   Change History      Sort Order:
Robin Shen [12/Jun/19 11:45 PM]
As you mentioned, QB only adds the "origin" remote repository, then how is the extra remote added? Also have you tried to clean the configuration when this happens?

Sebastian Potasiak [13/Jun/19 08:24 AM]
We add another remote repository with a command step. The working directory is set to the repository and we execute `git remote add ...` command there.

Cleaning it up indeed fixes the issue, but the URL to the remote repository is provided in a run dialog of the configuration by developers (multiple different repositories are added as a remote this way). There shouldn't be a need to clean the configuration at all. It's a bug, as `git fetch` command used internally in QB has a different scope (all remotes) than the defined repository (only one remote).

In my opinion this either should be fixed, so we could compensate for the lack of functionalities with workarounds, or git repositories should support multiple remotes.

Robin Shen [13/Jun/19 11:35 PM]
The "--all" option is added to fix issue QB-2320. QB manages its cloned repository internally and user should not hack to add additional remote to it in normal cases. Any reason you want to do that?

Sebastian Potasiak [14/Jun/19 08:10 AM]
The `--all` parameter is not needed to fix issue QB-2320, as the `--all` parameter is needed to fetch all remotes, not all branches (see: https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---all ). To fetch all branches from a remote, a `git fetch ` command should be used, e.g. `git fetch origin` or even `git fetch` in this case. There seems to be a lack of understanding of how git works. Note that when you are fetching changes from a remote those are not merged to the local branches, but are fetched to the remote refs. each remote has its own ref namespace, as well as your local branches, tags, and so on. The directory structure looks like this:

.git/refs
|-- heads
| \-- master
|-- remotes
| \-- origin
| |-- HEAD
| |-- bugfix
| |-- fix
| \-- master
\-- tags
  \-- 1.0.0

So, when you run `git fetch`, you get changes from the remote in the refs/remotes/origin directory. To have the changes on your local branches (refs/heads), you have to merge the changes from the remote branch to your local branch, e.g. `git merge origin/master master` OR use `git pull` that does both operations for you (fetch and merge). The `--all` parameter result is only that it fetches changes from all remotes if there is more than one remote added to local repository, and without adding the remotes like we do, using command step, the results of `git fetch` and `git fetch --all` are the same, as there's only one remote by default.

Example of running git fetch on a repository locally (I've obfuscated potentially sensitive data) WITHOUT `--all` parameter:

$ git fetch
remote: Counting objects: 21360, done.
remote: Compressing objects: 100% (7026/7026), done.
remote: Total 21360 (delta 14314), reused 19603 (delta 13070)
Receiving objects: 100% (21360/21360), 36.63 MiB | 1.94 MiB/s, done.
Resolving deltas: 100% (14314/14314), completed with 1865 local objects.
From ssh://url.of.our/repository
   c9b7cf781ff..786fa0992b2 master -> origin/master
 * [new branch] fix/fix-something -> origin/fix/fix-something
 * [new tag] 1.0.0 -> 1.0.0

Once the fetch is finished, you can calculate changes between all branches residing in the remote.

See also: https://git-scm.com/docs/git-fetch

And yes, we do need that because we have a huge project that requires interactions between multiple repositories that are used as remotes. Since there's no option to add more than one remote to the repository directly in the QuickBuild configuration, we have to "hack" it (to be honest, it's not a hack; we simply want to use git remotes like they are intended to be used and QuickBuild doesn't provide us this possibility, so we have to do it through command steps).

In summary:
1. You do not need `--all` parameter to fix QB-2320, so it can be removed. It would fix our issue.
2. It would be best if QuickBuild could handle multiple remotes natively, so we wouldn't have to manage them using command steps.

Robin Shen [14/Jun/19 11:34 PM]
This is a very detailed explanation. Thanks a lot!

Following your suggestion, I verified that "git fetch" does able to get all branches/tags of the origin remote, and has made the changes in QB 9.0.12:
https://build.pmease.com/build/4901

OTOH, we'd rather not to add additional options for multiple remotes to complicate the repository definition, as this seems like a rare case (at least not hearing about this requirement from other users till now)

Sebastian Potasiak [17/Jun/19 06:32 AM]
Great! Thank you for solving this issue. As for multiple remotes for the repositories, I think it's fine for us, we already have a way around it in place.