[#QB-2601] Git plugin should support "shallow" clones (using the "--depth" option to clone) to increase efficiency populating workspaces from large repositories

QuickBuild

Git plugin should support "shallow" clones (using the "--depth" option to clone) to increase efficiency populating workspaces from large repositories

Created: 11/Dec/15 12:53 AM Updated: 11/Jun/20 11:34 PM

Component/s:

None

Affects Version/s:

6.0.36

Fix Version/s:

6.1.0, 10.0.17

Original Estimate:	Unknown	Remaining Estimate:	Unknown	Time Spent:	Unknown
Environment:	Various variants of Linux and Windows

Description

« Hide

We are in the middle of a migration from subversion to git, and we have some very large repositories.
It would greatly decrease time to clone and lower disk usage if we could configure a QB git repo to use the '--depth' option to clone.

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Martin [11/Jun/20 08:29 AM]

We are experiencing some problems with the shallow clone option for git repositories. After setting it to for example 10 we can observe the following git command running:

{code}
ubuntu@ip-XX-XXX-XX-XX:~$ ps aux | grep intake
ubuntu 17186 0.0 0.0 13136 1000 pts/0 S+ 06:44 0:00 grep --color=auto intake
svc_qui+ 20851 0.0 0.0 19592 4260 ? S 06:38 0:00 git clone --no-checkout --depth 10 https://****:****@bitbucket.acme.org/scm/test/main.git /opt/buildagent/workspace/root/dev-pipeline/3-intake/parent_code -q
svc_qui+ 22541 0.8 0.0 35468 16504 ? S 06:39 0:02 git --shallow-file /opt/buildagent/workspace/root/dev-pipeline//3-intake/parent_code/.git/shallow.lock index-pack --stdin --fix-thin --keep=fetch-pack 7317 on quickbuild-agent-XX-XXX-XX-XX --pack_header=2,67859
{code}

and after some minutes a full fetch is invoked

{code}
ubuntu@ip-XX-XXX-XX-XX:/$ ps aux | grep git
svc_qui+ 12558 0.0 0.0 19036 3368 ? S 07:24 0:00 git fetch --all --tags -q
svc_qui+ 12559 0.0 0.0 22708 6084 ? S 07:24 0:00 git fetch --append --tags -q origin
svc_qui+ 12560 0.2 0.0 192036 30400 ? S 07:24 0:05 git-remote-https origin https://****:****@bitbucket.acme.org/scm/main.git
svc_qui+ 12831 0.0 0.0 747816 7100 ? Sl 07:24 0:01 git fetch-pack --stateless-rpc --stdin --lock-pack --thin --no-progress https://****:****@bitbucket.acme.org/scm/test/main.git/
svc_qui+ 17963 0.7 0.1 779548 43252 ? S 07:25 0:14 git index-pack --stdin --fix-thin --keep=fetch-pack 8875 on quickbuild-agent-XX-XXX-XX-XX --pack_header=2,1589619
ubuntu 22020 0.0 0.0 13136 968 pts/1 S+ 07:57 0:00 grep --color=auto git
{code}

This is actually unexpected behaviour as it means that the option is not useful. We need to use a shallow clone since we have huge repositories and we don't need the full history of the repository.

[ Show » ]

Martin [11/Jun/20 08:29 AM] We are experiencing some problems with the shallow clone option for git repositories. After setting it to for example 10 we can observe the following git command running: {code} ubuntu@ip-XX-XXX-XX-XX:~$ ps aux | grep intake ubuntu 17186 0.0 0.0 13136 1000 pts/0 S+ 06:44 0:00 grep --color=auto intake svc_qui+ 20851 0.0 0.0 19592 4260 ? S 06:38 0:00 git clone --no-checkout --depth 10 https://****:****@bitbucket.acme.org/scm/test/main.git /opt/buildagent/workspace/root/dev-pipeline/3-intake/parent_code -q svc_qui+ 22541 0.8 0.0 35468 16504 ? S 06:39 0:02 git --shallow-file /opt/buildagent/workspace/root/dev-pipeline//3-intake/parent_code/.git/shallow.lock index-pack --stdin --fix-thin --keep=fetch-pack 7317 on quickbuild-agent-XX-XXX-XX-XX --pack_header=2,67859 {code} and after some minutes a full fetch is invoked {code} ubuntu@ip-XX-XXX-XX-XX:/$ ps aux | grep git svc_qui+ 12558 0.0 0.0 19036 3368 ? S 07:24 0:00 git fetch --all --tags -q svc_qui+ 12559 0.0 0.0 22708 6084 ? S 07:24 0:00 git fetch --append --tags -q origin svc_qui+ 12560 0.2 0.0 192036 30400 ? S 07:24 0:05 git-remote-https origin https://****:****@bitbucket.acme.org/scm/main.git svc_qui+ 12831 0.0 0.0 747816 7100 ? Sl 07:24 0:01 git fetch-pack --stateless-rpc --stdin --lock-pack --thin --no-progress https://****:****@bitbucket.acme.org/scm/test/main.git/ svc_qui+ 17963 0.7 0.1 779548 43252 ? S 07:25 0:14 git index-pack --stdin --fix-thin --keep=fetch-pack 8875 on quickbuild-agent-XX-XXX-XX-XX --pack_header=2,1589619 ubuntu 22020 0.0 0.0 13136 968 pts/1 S+ 07:57 0:00 grep --color=auto git {code} This is actually unexpected behaviour as it means that the option is not useful. We need to use a shallow clone since we have huge repositories and we don't need the full history of the repository.

[ Permlink | « Hide ]

Robin Shen [11/Jun/20 11:34 PM]

Thanks for reporting. This has now been solved in 10.0.17:
https://build.pmease.com/build/5229