Since we require Python 3.6 now in the launcher, switch to NamedTuple
so we get better documentation & typing information.

Change-Id: Ic58fdc07db02fc49166eccbbc3e527f474973424
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/463721


Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>

We updated run_tests to use black-25, so update tox too.

Change-Id: I7ee6471fbc78825bd2dbc8c1f8dab9dc10460852
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/463601


Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>

Requires a little reformatting in the tree.

Change-Id: Iaa40fe0dfca372c49c04cc26edccb5f7b0c2a8ad
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462883


Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>

Move the test disable logic even closer to the exact test that's
disabled.  This way people updating tests have a better chance of
seeing they'll get reduced coverage in the CQ.

Change-Id: I57c1a073a844019798b27e14d742fd32925d9ae8
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462882


Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>

Our recipes have been disabling a bunch of tests.  To increase
visibility, and to make it easier to test changes, move that
logic to this script.

Change-Id: I3894f047715177c0f1d27a2fe4c3490972dab204
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462881


Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>

Using a generator w/all() causes the code to exit on the first error.
We really want to see all errors all the time, so use sum() instead.

Change-Id: Ib1adb8de199db9fe727d4b49c890b4d5061e9e6b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462901


Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>
Reviewed-by: Scott Lee <ddoman@google.com>

Since we require Python 3.6 now in the launcher, swap out our custom
RunResult class for the standard subprocess one.

Change-Id: Idd8598df37c0a952d3ef828df6e250cab03c6589
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/462341


Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>

Git allows to clone submodules as shallow clone [1]. On the other
hand, when repo synchronize a projcet with submodules inside, it
ignores the shallow parameter.

When a project contains submodules, project.py parses the .gitmodules
file for URL and path. This parsing does not consider the shallow
option. Consequently, this parameter is not propgated to newly
created Project instance for that submodule.

[1] https://git-scm.com/docs/gitmodules#Documentation/gitmodules.txt-submoduleltnamegtshallow

Change-Id: I54fc9c69ae1b8e3cda2801202e3f0c7693b718d2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/454261


Tested-by: Albert Akmukhametov <alb.02057@gmail.com>
Commit-Queue: Albert Akmukhametov <alb.02057@gmail.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Никита Сказкоподателев (Nask) <skazkopodatelev@gmail.com>

If the repo index is stale, reset --keep will refuse to reset workspace.
An index can be stale if there are any modifications to file node,
including mtime, atime, ownership changes, etc.

Bug: b/375423099
Change-Id: Ibef03d9d8d2babbb107041707281687342ab7a77
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/460022


Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>

If tags are omitted from the remote pack list, they must be present in
local pack. However, local packs don't have promisor objects, meaning
that all blobs must be available locally, and therefore all missing
blobs will be downloaded during rev-list phase. Git downloads those
sequentially, by invokving fetch operation (rev-list/fetch).

Instead of downloading tags' blobs, instruct Git to include all tags in
remote rev-list operation. This change was tested with `git fsck --all`.

R=yiwzhang@google.com

Bug: b/392732561
Change-Id: Id94a40aebbe4f084c952329583d559d296db1a11
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/451422


Reviewed-by: Yiwei Zhang <yiwzhang@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

This change moves further towards ensuring Git can understand repo's
submodules. 'submodule init' is used to make the submodules active[1].

[1] https://git-scm.com/docs/gitsubmodules#_active_submodules

Change-Id: I0c20ff1991101fc5be171e566d8fb644aab47200
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446182


Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>

This change takes another step towards ensuring Git can understand
repo's submodules to some extent. Replace the old '.git' symlink with
gitfile[1] pointing to the bare checkout of the submodule. This is
required for Git's 'recurse submodules' opts to work with repo's
submodules as '.git' is expected to be writable by Git when recursing
over submodules.

[1] https://git-scm.com/docs/gitrepository-layout#_description

Change-Id: I52d15451768ee7bd6db289f4d2b3be5907370d42
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446181


Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Nasser Grainawi <nasser.grainawi@oss.qualcomm.com>

Change the bare checkout directory for submodules from 'subprojects'
to 'modules'. Git expects bare submodule checkouts to be in the
'modules' directory. If old subproject directories are found, they
will be migrated to the new modules directory. This change is the
first step in ensuring Git can understand repo's submodules to some
extent.

Change-Id: I385029f1bb55d040616d970d6ffb4bb856692520
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444881


Tested-by: Kaushik Lingarkar <kaushikl@qti.qualcomm.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

This reverts commit 61224d01.

Reason for revert: the manifest will be updated during in the
post-upgrade process, and that can result in a missing object in
LocalHalf, since NetworkHalf is not skipped.

Bug: b/392979411
Change-Id: I8a46e5b54093ed78285c8b30f000bb08a8244179
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450181


Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>

The existing documentation on smart-sync behavior is a bit light on
details, and out of date wrt what the code actually does.  Start a
dedicated document and fill it out more.

Change-Id: I1a8a3ac6edf9291d72182ad55db865035d9b683e
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/450002


Commit-Queue: Mike Frysinger <vapier@google.com>
Tested-by: Mike Frysinger <vapier@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>

Change-Id: Iffe45d85a54dc380cdd37bbbbe64b058eacad0a9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/449901


Reviewed-by: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Mike Frysinger <vapier@google.com>

Currently, repo ignores the XDG path for the git config file, and
creates a new one in the user's home directory. This commit changes the
behavior to prefer the XDG path if it exists, which matches git behavior
and avoids littering the home directory.

Bug: 40012443
Change-Id: Icd3ec6db6b0832f47417bbe98ff9461306b51297
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/448385


Tested-by: lmaor xenix <25misha52@gmail.com>
Reviewed-by: Mike Frysinger <vapier@google.com>

When repo is initializing a git repository, it calls `git read-tree`.
During such operation, git is restoring workspace based on the current
index. However, some things can go wrong: a user can run out of disk
space, or, in case of partial clone, user may no longer reach the remote
host. That will leave affected repository in a bad state with partially
checked out workspace. The follow up repo sync won't try to fix such
state.

This change removes .git symlink, which will force the next `repo sync`
to redo Git repository setup.

Bug: b/363171216
Bug: b/390161127
Change-Id: I57db4b6cae0ef21826dc7cede4d3bf02cfc3d955
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447801


Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

When a repository is partially cloned, no longer needed blobs are never
removed. To reclaim some of disk space, allow user to pass --repack
which affects only repositories with filter=blob:none and if projects
are not shared.

Change-Id: I0608172c9eff82fb8a6b6ef703eb109fedb7a6cc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447722


Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Scott Lee <ddoman@google.com>

Change-Id: Icef4f28fbdb9658892611def7589f5eba43c952c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447721


Reviewed-by: Scott Lee <ddoman@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>

KeyboardInterrupt is handled during NetworkHalf. This patch handles
KeyboardInterrupt during LocalHalf.

Bug: b/372069163
Change-Id: I26847f7ca3cdf1fe57b265b4f6b18cc8102d2921
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/447401


Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>

When repo upgrades itself, it will restart itself and rerun sync
command. At that point, we know that network half is already done and we
can just proceed with local half.

R=ddoman@google.com

Bug: b/377567091
Change-Id: I77205b1f2df19891597347d55283a617de3c6634
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/446201


Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>

When projects are removed from manifest, they are only removed from
worktree and not from .repo/projects and .repo/project-objects. Keeping
data under .repo can be desired if user expects deleted projects to be
restored (e.g. checking out a release branch).

Android has ongoing effort to remove many stale projects and this change
allows users to easily free-up their disk space.

Bug: b/344018971
Bug: 40013312
Change-Id: Id23c7524a88082ee6db908f9fd69dcd5d0c4f681
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445921


Reviewed-by: Mike Frysinger <vapier@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@chromium.org>

If repo sync is invoked outside the repo root, and the latest manifest
removes symlinks, repo incorrectly tries to remove symlink - it starts
from `cwd` instead of the repo root.

Bug: b/113935847
Bug: 40010423
Change-Id: Ia50ea70a376e38c94389880f020c80da3c3f453c
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445901


Tested-by: Josip Sokcevic <sokcevic@chromium.org>
Reviewed-by: Gavin Mak <gavinmak@google.com>

After some refactoring earlier, the forall command was
broken briefly, returning after only one run instead
of after all projects.

This test, albeit simple in nature, would have caught that.

Due to the somewhat demanding nature of forall,
a lot more setup was needed than expected but seems
to do its job now so hopefully it catches similar stuff
in the future.

Change-Id: I51e161ff0e7e31a65401211c376f319bda504532
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/445461


Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Fredrik de Groot <fredrik.de.groot@haleytek.com>

gitc is no longer available.

Change-Id: I0cbfdf936832f2cdd4876104ae3cc5a6e26154e2
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/444841


Reviewed-by: Gavin Mak <gavinmak@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>

If a branch name contains a percent sign, it will be interpreted as a placeholder and color.py will fail to format it.

To avoid this, escape the percent signs prior to calling Coloring
method.

Bug: b/379090488
Change-Id: Id019c776bbf8cbed5c101f2773606f1d32c9e057
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443801


Reviewed-by: Scott Lee <ddoman@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>

Since commit 454fdaf1 (v2.48), syncing a
manifest without any projects would result in:

  Repo command failed: RepoUnhandledExceptionError
          Number of processes must be at least 1

Bug: 377546300
Change-Id: Iaa2f6a3ac64542ad65a19c0eef449f53c09cae67
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443442


Reviewed-by: Erik Elmeke <erik@haleytek.corp-partner.google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Tested-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>

command should be cmd_name, to match what git is emitting. This also
fixes arguments, so that only relevant arguments are passed instead
of the entire sys.args, which will contain wrapper information

Change-Id: Id436accfff511292ec2c56798fffb2306dda38fc
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/443741


Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>

Some tools like jj and cog will not have .git. This change
makes it possible to run all repo commands in such setups.

Change-Id: I7f3845dc970fbaa731c31e0aa48355a4b56ed3a6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/442821


Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>

With a large number of sync workers, the sync process may fail on
macOS due to connection errors. The root cause is that multiple
workers may attempt to connect to the multiprocessing manager server
at the same time when handling the first job. This can lead to
connection failures if there are too many pending connections, exceeding
the socket listening backlog.

Bug: 377538810
Change-Id: I1924d318d076ca3be61d75daa37bfa8d7dc23ed7
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441541


Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>

Incorrect tuple values were returned with http://go/grev/440221 -
instead of returning (Project, ReviewableBranch), _ProcessResults was
returning (int, ReviewableBranch).

R=jojwang@google.com

Bug: 376731172
Change-Id: I75205f42fd23f5ee6bd8d0c15b18066189b42bd9
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/441121


Reviewed-by: Sam Saccone <samccone@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>

--worktree was broken with incorrect paths in the .git files
whenever the local copy of git populated gitdir with relative paths
instead of absoulte paths.

Bug: 376251410
Change-Id: Id32dc1576315218967de2a9bfe43bf7a5a0e7aa6
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440801


Commit-Queue: Allen Webb <allenwebb@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Allen Webb <allenwebb@google.com>

rc should be returned only after all results are processed.

R=jojwang@google.com

Bug: b/376454189
Change-Id: I8200b9954240dd3e8e9f2ab82494779a3cb38627
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440901


Tested-by: Josip Sokcevic <sokcevic@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Joanna Wang <jojwang@google.com>

The manifest usually doesn't sepecify the revision with the full name
(e.g. refs/heads/REV).
However, when checking if the name of the merge branch, full name is
used on the merge branch.

The CL use full name of revision when comparing it with the merge
branch.

Bug: b/370919047
Test: repo upload on a project with `dest-branch` set
Change-Id: Ib6fa2f7246beb5bae0a26a70048a7ac03b6c5a2f
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/438401


Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Joe Hsu <joehsu@google.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>

The command _PostRepoFetch will try to self update
during repo sync. That is beneficial but adds
version uncertainty, fail potential and slow downs
in non-interactive scenarios.

Conditionally skip the update if env variable
REPO_SKIP_SELF_UPDATE is defined.

A call to selfupdate works as before, meaning even
with the variable set, it will run the update.

Change-Id: Iab0ef55dc3d3db3cbf1ba1f506c57fbb58a504c3
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/439967


Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>

This adds an optional, built-in checker for
guarding against patches hanging on wrong
base revisions, which is useful if a lower layer of
the manifest changes after a patch was done.

When adding a patch with a new revision using
extend-project or remove-project/project:

          C---D---E patches in project bla
         /
    A---B project bla in manifest state 1

<extend-project name="bla" revision="E" base-rev="B">

If project bla gets updated, in a new snap ID
or by a supplier or similar, to a new state:

          C---D---E patches in project bla
         /
    A---B---F---G project bla in manifest state 2

Parsing will fail because revision of bla is now G,
giving the choice to create a new patch branch
from G and updating base-rev, or keeping previous
branch for some reason and only updating base-rev.

Intended for use in a layered manifest with
hashed revisions. Named refs like branches and tags
also work fine when comparing, but will be misleading
if a branch is used as base-rev.

Change-Id: Ic6211550a7d3cc9656057f6a2087c505b40cad2b
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/436777


Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Fredrik de Groot <fredrik.de.groot@haleytek.com>
Commit-Queue: Josip Sokcevic <sokcevic@google.com>

With the global state and fork, we are left with uncleaned resources.
Isolate mulitprocessing.Value in a function so we stop the leak.

Bug: 353656374
Change-Id: If50bb544bda12b72f00c02bc1d2c0d19de000b88
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440261


Commit-Queue: Josip Sokcevic <sokcevic@google.com>
Reviewed-by: Gavin Mak <gavinmak@google.com>
Tested-by: Josip Sokcevic <sokcevic@google.com>

The done message was omitted if the task is shorter than 0.5s. This
might confuse users.

Bug: b/371638995
Change-Id: I3fdd2cd8daea16d34fba88457d09397fff71af15
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440222


Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>
Reviewed-by: Josip Sokcevic <sokcevic@google.com>

Follow the same approach as 39ffd997 to reduce serialization overhead.

Below benchmarks are tested with 2.7k projects on my workstation
(warm cache). git tracing is disabled for benchmark.

(seconds)              | v2.48 | v2.48 | this CL | this CL
	               |       |  -j32 |         |    -j32
-----------------------------------------------------------
with clean tree state:
branches (none)        |   5.6 |   5.9 |    1.0  |    0.9
status (clean)         |  21.3 |   9.4 |   19.4  |    4.7
diff (none)            |   7.6 |   7.2 |    5.7  |    2.2
prune (none)           |   5.7 |   6.1 |    1.3  |    1.2
abandon (none)         |  19.4 |  18.6 |    0.9  |    0.8
upload (none)          |  19.7 |  18.7 |    0.9  |    0.8
forall -c true         |   7.5 |   7.6 |    0.6  |    0.6
forall -c "git log -1" |  11.3 |  11.1 |    0.6  |    0.6

with branches:
start BRANCH --all     |  21.9 |  20.3 |   13.6  |    2.6
checkout BRANCH        |  29.1 |  27.8 |    1.1  |    1.0
branches (2)           |  28.0 |  28.6 |    1.5  |    1.3
abandon BRANCH         |  29.2 |  27.5 |    9.7  |    2.2

Bug: b/371638995
Change-Id: I53989a3d1e43063587b3f52f852b1c2c56b49412
Reviewed-on: https://gerrit-review.googlesource.com/c/git-repo/+/440221


Reviewed-by: Josip Sokcevic <sokcevic@google.com>
Tested-by: Kuang-che Wu <kcwu@google.com>
Commit-Queue: Kuang-che Wu <kcwu@google.com>