A compatibility wrapper for writev(3p) has been reintroduced,
including fixes for CMake build and 'MAX_IO_SIZE' limits on NonStop.
Calls to write(3p) in send_sideband() and cat_blob() have been
refactored to use writev(3p) wrappers to reduce syscall overhead.
* ps/writev:
fast-import: use writev(3p) to send cat-blob responses
sideband: use writev(3p) to send pktlines
wrapper: properly handle MAX_IO_SIZE in writev(3p)
wrapper: introduce writev(3p) wrappers
compat/posix: introduce writev(3p) wrapper
The 'pack-objects' and delta-encoding code paths have been updated to
use 'size_t' instead of 'unsigned long' for object sizes and offset
limits, avoiding potential truncation issues on 64-bit Windows.
* js/pack-objects-delta-size-t:
git-zlib: widen `git_deflate_bound()` to `size_t`
t/helper/test-pack-deltas: widen `do_compress()`'s maxsize local to `size_t`
http-push: widen `start_put()`'s size local from `ssize_t` to `size_t`
diff: widen `deflate_it()`'s bound local from int to `size_t`
archive-zip: widen `zlib_deflate_raw()`'s maxsize local to `size_t`
packfile, git-zlib: widen `use_pack()` and zstream avail fields to `size_t`
delta: widen `create_delta()` and `diff_delta()` to `size_t`
pack-objects: widen `mem_usage` and `try_delta()`'s out-param to `size_t`
pack-objects: widen `free_unpacked()` return to `size_t`
pack-objects: widen delta-cache accounting to `size_t`
delta: widen `create_delta_index()` parameter to `size_t`
diff-delta: widen `struct delta_index`' size fields to `size_t`
'git add' has been taught a new '--resolved' option to stage
conflict-resolved paths, while leaving unrelated local changes
unstaged. It scans the unmerged paths for leftover conflict
markers and aborts if any are found.
* jc/add-resolved:
add: introduce '--resolved' option
read-cache: add remove_file_from_index_with_flags()
merge-ll: consolidate conflict marker scanning logic
read-cache: reindent
'git worktree' add did not prevent DWIM behavior when '-b' or '-B'
was specified, which has been corrected.
* yn/worktree-add-no-dwim-with-b:
worktree add: shouldn't dwim if -b or -B is given
The known limitations of the ref format migration in 'git refs' have
been moved to be displayed as a warning admonition directly under the
description of the 'migrate' subcommand, improving visibility. A
reference to 'git-maintenance' has also been corrected to use the
'linkgit' macro.
* kh/doc-refs-migrate-limitations:
doc: refs: linkgit to git-maintenance(1)
doc: refs: put ref migration warning under the command
The 'git branch' command has been taught the '--delete-merged' option
to remove local branches that are already merged into their tracked
remote-tracking branches.
* hn/branch-delete-merged:
branch: add --dry-run for --delete-merged
branch: add branch.<name>.deleteMerged opt-out
branch: add --delete-merged <pattern>
branch: prepare delete_branches for a bulk caller
branch: let delete_branches skip unmerged branches on bulk refusal
branch: convert delete_branches() to a flags argument
branch: add --forked filter for --list mode
'git -C <dir> diff fi<TAB>' did not complete 'dir/file', which has
been corrected.
* jc/complete-diff-tracked-paths:
completion: 'git diff' completes untracked paths as a last resort
completion: complete tracked paths for 'git diff'
completion: no-op refactoring of diff completion
The 'ssh-agent' tests in 't7528' have been fixed to work when the
user's login shell is csh-like, by explicitly passing '-s' to
'ssh-agent' to force Bourne shell syntax.
* kl/t7528-ssh-agent-for-csh-users:
t7528: fix failure under csh
Concurrent downloads of packfiles via packfile URIs and dumb HTTP are
safer by avoiding concurrent appends to the staging file. Opening in
read-write mode with separate file offsets prevents corruption and
preserves resumability. 'fetch-pack' now tolerates pre-existing
'.keep' files.
* tn/packfile-uri-concurrency:
fetch-pack: accept "pack" output for packfile URIs
http: permit unlinking partial packs on Windows
http: avoid concurrent appends to partial packs
http: accept HTTP 416 for complete partial packs
http: avoid closing index-pack input twice
http-fetch: correct --index-pack-arg documentation
The 'TRACE2_ANCESTRY' prerequisite in the 't0213' test script has been
refined to avoid failures under user-mode emulation by verifying that
the ancestry collector reports the expected process names rather than
the emulator binary name.
* jm/t0213-skip-emulated-ancestry-tests:
t0213: skip ancestry tests under user-mode emulation
Tests for 'git merge-base --is-ancestor' have been added to cover
exit codes (0 for success, 1 for non-ancestor, 128 for errors) and
to ensure it cannot be combined with '--all'.
* ns/merge-base-is-ancestor-tests:
merge-base: add tests for --is-ancestor
'git mv' has been updated to check for a missing destination
leading directory during the checking phase, allowing 'git mv -n'
to report the failure. The error message when the rename(2)
syscall fails has also been improved to name both the source and
the destination.
* lo/mv-missing-dest-dir-check:
mv: reject a destination whose leading path is missing or a symlink
mv: name both source and destination when rename fails
Documentation for 'git replay' has been updated to refer to its
configuration variables.
* kh/doc-replay-config:
doc: replay: move “default” to the right-hand side
doc: replay: use a nested description list
doc: replay: improve config description
doc: link to config for git-replay(1)
A new test helper commit_body() has been introduced to print the
message body of a commit, and various tests have been updated to use
it instead of spelling out the command pipeline manually and losing
the exit status of the 'git cat-file' command on the upstream of the
pipe.
* sk/test-commit-body-helper:
t: use commit_body to extract commit message bodies
test-lib-functions: add commit_body helper
Synopsis and options in the documentation for 'git format-patch',
'git imap-send', 'git send-email', and 'git request-pull' have been
updated to the modern style.
* ja/doc-synopsis-style-yet-more:
doc: convert git-request-pull synopsis and options to new style
doc: convert git-send-email synopsis and options to new style
doc: convert git-format-patch synopsis and options to new style
doc: convert git-imap-send synopsis and options to new style
A crash in the 'sparse-index' collapse code when encountering an
invalidated cache-tree node (due to an intent-to-add path) has been
fixed by avoiding collapsing such subtrees.
* ds/sparse-index-ita-crash:
sparse-index: avoid crash on intent-to-add entry outside the cone
A boundary case check in reachability bitmap traversal has been
corrected to properly handle the object at position zero, which was
previously skipped, leading to redundant bitmap loading.
* dl/pack-bitmap-position-zero:
pack-bitmap: handle objects at bitmap position zero
A memory leak in 'git merge' when run without arguments (which
triggers the default-to-upstream path) has been fixed. A test has
been added to cover this case.
* tc/merge-default-to-upstream-leakfix:
merge: fix leak with merge.defaultToUpstream
'git cat-file --batch-command' that asked for 'contents' without
'type' segfaults, which has been corrected.
* jk/cat-file-batch-wo-type-fix:
cat-file: handle content request for --batch-command without type
The get_commit_action() function has been refactored to be a pure
predicate by moving the side-effecting line-level log range folding to
simplify_commit(). This ensures that evaluating a commit's action
before the walk reaches it does not prematurely mutate its tracked
line ranges, making it safer for potential lookahead evaluations.
* mm/revision-pure-get-commit-action:
revision: make get_commit_action() a pure predicate
The code path that deals with relative paths in the 'diff-lib' has
been cleaned up.
* jk/diff-relative-cached-unmerged-more:
diff-lib: skip paths outside prefix in oneway_diff()
diff-lib: drop stale comment about advancing o->pos
We taught 'git diff' to first try to complete revisions (unless '--'
is present on the command line) and, failing that, to complete
tracked paths. If this yields nothing, it lets the Bash default,
which offers paths in $PWD, kick in.
Teach it to complete untracked paths before giving up and letting
the Bash default kick in. With this change,
$ git -C another-directory diff un<TAB>
finds the 'untracked' file in another-directory and offers it as a
completion candidate.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When completing arguments for 'git diff', _git_diff() delegates to
__git_complete_revlist_file(), which only completes revision
references. This is good [*], as mixing both revisions and paths in a
single list for the user to pick from is simply too confusing.
If no reference matches, or if '--' is given, however, _git_diff()
leaves COMPREPLY empty. Bash then falls back to default filename
completion in $PWD. This fails when 'git -C <path>' is used because
$PWD is not the target repository.
Update _git_diff() to use __git_complete_index_file() when '--' is
present, or when revision reference completion yields no matching
candidates, so that tracked paths are offered as candidates.
This changes behavior even in the case where '-C <there>' is not
used. The new behavior omits untracked paths from suggestions when
no revs match the prefix but matching tracked paths exist, which is
more useful in the context of 'git diff'.
When run outside the working tree of a repository, or when nothing
matches from revisions or tracked paths, Bash still falls back to
default filename completion in $PWD, so such a use case would be
just like completing paths for any 'diff' command, rather than for
'git diff'.
[Footnote]
* In https://lore.kernel.org/git/al%2Fw2qgBfhe9qMg6@szeder.dev/
SZEDER made the same argument for "git send-email 0<TAB>".
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "git diff" completion function punts very early when it sees
"--" on the command line, since it is a sign that options or
revisions can appear and the current completion does not need to do
anything "git diff" specific. By returning, it lets Bash default
action that completes the names of the files in $PWD to kick in.
In preparation for the next step to change what happens when we
"punt", arrange the code flow to avoid this early return. The
behaviour at this step is unchanged, but the control flow just
falls straight to the end.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When answering a `cat-blob` command, `cat_blob()` issues three separate
calls to write(3p) on the cat-blob fd: one for the header line, one for
the full blob payload, and one for the trailing newline. Frontends like
git-filter-repo issue these commands in bulk, once per rewritten blob,
so the syscall overhead adds up.
Use `writev_in_full()` to send all three parts with a single syscall.
This can be benchmarked with the following setup:
$ git cat-file --unordered --filter=object:type=blob
--batch-check='cat-blob %(objectname)' --batch-all-objects >request
$ git fast-import --cat-blob-fd=3 <request
Executing this with 100,000 objects in linux.git:
Benchmark 1: HEAD~
Time (mean ± σ): 1.320 s ± 0.003 s [User: 1.154 s, System: 0.161 s]
Range (min … max): 1.314 s … 1.324 s 10 runs
Benchmark 2: HEAD
Time (mean ± σ): 1.270 s ± 0.022 s [User: 1.133 s, System: 0.132 s]
Range (min … max): 1.209 s … 1.282 s 10 runs
Summary
HEAD ran
1.04 ± 0.02 times faster than HEAD~
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Every pktline that we send out via `send_sideband()` currently requires
two syscalls: one to write the pktline's length, and one to send its
data. This typically isn't all that much of a problem, but under extreme
load the syscalls may cause contention in the kernel.
Refactor the code to instead use the newly introduced writev(3p) infra
so that we can send out the data with a single syscall. This reduces the
number of syscalls from around 133,000 calls to write(3p) to around
67,000 calls to writev(3p).
This change leads to a performance improvement for git-upload-pack(1),
but we have to cheat a bit to really make it measurable. Usually, the
time is strongly dominated by generating the packfile itself. But if we
precompute the pack and serve it via the pack-objects hook then we can
essentially eliminate that overhead. The following setup is executed in
the Git repository:
$ cat >request <<-EOF
0048want 5ce91c059e side-band no-progress
00000009done
EOF
$ echo 5ce91c059e | git pack-objects --revs --stdout >pack
$ cat >hook <<-EOF
#!/bin/sh
cat >/dev/null
cat "$(pwd)"/pack
EOF
$ chmod u+x hook
$ git -c uploadpack.packObjectsHook="$(pwd)"/hook upload-pack . <request
Benchmarking the last command leads to the following results:
Benchmark 1: HEAD~
Time (mean ± σ): 192.9 ms ± 0.6 ms [User: 106.5 ms, System: 95.3 ms]
Range (min … max): 191.7 ms … 194.1 ms 50 runs
Benchmark 2: HEAD
Time (mean ± σ): 141.1 ms ± 0.7 ms [User: 63.2 ms, System: 86.6 ms]
Range (min … max): 139.8 ms … 142.7 ms 50 runs
Summary
HEAD ran
1.37 ± 0.01 times faster than HEAD~
This might not be impressive in absolute numbers when you also take into
account the time it takes to generate the packfile itself. But GitLab
(and supposedly other forges) have caching mechanisms in place that work
exactly like the above setup, where repeated incoming requests can be
served from the same cached packfile. And in those cases, the impact is
sizeable.
More importantly though, as hinted at above, GitLab has observed in the
past that with enough cache hits we eventually start to saturate a
semaphore in the Linux kernel itself in the pipe write path. This
bottleneck is being moved a bit by having to do less syscalls.
Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some systems like NonStop set a comparatively small `MAX_IO_SIZE`, which
limits the maximum number of bytes we're allowed to write in a single
call. We already handle this limit properly in `xwrite()`, but we have
recently introduced wrappers for writev(3p) where we don't. This will
cause the syscall to return EINVAL in case somebody passes an iovec
entry to writev(3p) that is larger than `MAX_IO_SIZE`.
Introduce a new function `xwritev()` that is similar to `xwrite()` in
that it handles such platform-specific nuances:
- We only pass the leading iovec entries to writev(3p) that fit into
`MAX_IO_SIZE`, pretending that the underlying syscall performed a
short write. This mirrors how `xwrite()` chomps overly large
requests before handing them to write(3p). As a consequence, callers
will never see writev(3p)'s EINVAL error for requests whose summed
length would overflow an ssize_t, but observe a short write instead.
- If already the first iovec entry exceeds the limit we instead punt
to `xwrite()`, which knows to handle this case for us.
- We restart the underlying syscall on EINTR and EAGAIN, just like
`xwrite()` does for write(3p).
Adapt `writev_in_full()` to use this new wrapper. With the retry logic
now living in `xwritev()`, the calling loop becomes the exact mirror
image of `write_in_full()`, which also retains the responsibility of
translating a zero-length write into ENOSPC.
Reported-by: Randall Becker <randall.becker@nexbridge.ca>
Helped-by: Jeff King <peff@peff.net>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the preceding commit we have added a compatibility wrapper for the
writev(3p) syscall. Introduce some generic wrappers for this function
that we nowadays take for granted in the Git codebase.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
I have to scroll down at least three screens in man(1) from the
`migrate` description in order to see the “known limitations” for
it. This is important information since the text says that concurrent
writes can lead to an inconsistent migrated state. Let’s move that text
up to the command description and put it inside a Warning admonition.
This section made sense when it was added in 25a0023f (builtin/refs:
new command to migrate ref storage formats, 2024-06-06); `migrate` was
the only subcommand, and this section was visible from the command
description. A one-page man page. But that is not the case anymore
now that the command has nine subcommands to describe.
Acked-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Kristoffer Haugsbakk <code@khaugsbakk.name>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git for Windows has been updated to avoid auto-detecting the symlink
type if the target path starts with a slash, preventing NTLM
credential leaks when checking out repositories with crafted
symbolic links pointing to network shares.
* js/mingw-symlink-net-share-leak:
mingw: skip symlink type auto-detection for network share targets
The alias tests in 't/t0014-alias.sh' have been updated to dynamically
query the list of deprecated commands using 'git
--list-cmds=deprecated' to avoid test failures when running with
'WITH_BREAKING_CHANGES' in a build directory that contains stale
executables of formerly deprecated commands.
* jk/t0014-dynamic-deprecated-cmds:
t0014: generate deprecated command names dynamically
t0014: factor out choice of deprecated commands
The image version used by the static-analysis CI job has been bumped
to ubuntu-latest (Ubuntu 24.04), which brings in a newer Coccinelle
version that resolves a severe performance regression. A false
positive warning from the 'CHECK_ASSERTION_SIDE_EFFECTS' build with
GCC 15 in the Bloom filter code has also been silenced to facilitate
the image upgrade.
* jk/ci-static-analysis-image-bump:
ci: bump ubuntu image version for static-analysis job
bloom: silence CHECK_ASSERTION_SIDE_EFFECTS false positive
'git branch -d' has been taught to report when a branch cannot be
deleted because it is being used in an active bisect run.
* rs/branch-delete-bisect-warning:
branch: report active bisect run when rejecting delete
Object database housekeeping in 'git gc' and 'git maintenance' has
been refactored to be pluggable. The files-backend-specific logic,
including incremental and geometric repacking as well as object
pruning, has been moved out of the command implementation and into the
files object database source, enabling future alternative object
database backends to implement their own housekeeping services.
* ps/odb-pluggable-housekeeping:
odb: make optimizations pluggable
builtin/gc: fix signedness issues in ODB-related functionality
builtin/gc: refactor ODB optimizations to operate on "files" source
builtin/gc: introduce `odb_optimize_required()`
builtin/gc: move geometric repacking into `odb_optimize()`
builtin/gc: introduce object database optimization options
builtin/gc: inline config values specific to the "files" backend
builtin/gc: make repack arguments self-contained
builtin/gc: extract object database optimizations into separate function
builtin/gc: move worktree and rerere tasks before object optimizations
odb: run "pre-auto-gc" hook for all maintenance tasks
t7900: simplify how we check for maintenance tasks
Two bugs in how 'git rebase' handles skipped 'fixup' and 'squash'
commands have been fixed. One bug caused an incorrect commit count to
be shown in the template message when multiple commands were skipped,
and another prevented the editor from opening when the final command
in a chain containing 'fixup -c' was skipped.
* pw/rebase-fixup-fixes:
rebase: remember fixup -c after skipping fixup/squash
rebase -i: fix counting of fixups after rebase --skip
'git diff --relative' running with '--cached' has been corrected to
avoid a segfault when encountering unmerged paths outside the
prefix.
* jk/diff-relative-cached-unmerged:
diff-lib: add idx/tree sanity check to oneway_diff
diff: ignore unmerged paths outside prefix with --relative --cached
"git branch --dry-run --delete-merged ..." prints one line per ref that
would be deleted without modifying refs or branch configuration.
--dry-run is only meaningful together with --delete-merged and is
rejected otherwise.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Setting branch.<name>.deleteMerged=false exempts that branch from
"git branch --delete-merged", which is useful for a topic you want
to keep developing after an early round of it has been merged
upstream. Unless --quiet is given, each skip is reported so the
user knows why their topic was kept.
Explicit deletion with "git branch -d" still uses the normal merge
check and ignores this setting.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git branch (--delete-merged <pattern>)... [<branch-pattern>...]
deletes local branches matching the optional branch patterns when their
configured upstream matches one of the --delete-merged arguments and
their tip is reachable from that upstream. The work has already landed
on the upstream they track, so the local copy is no longer needed.
Each <pattern> may name a ref, a remote, or a shell glob. The option can
be repeated to widen the upstream match. Keeping the candidate patterns
as positional arguments lets users bound the set of local branches that
may be deleted independently of the upstream selection.
A branch is not deleted when:
* it is checked out in any worktree
* its configured upstream ref no longer exists, since a missing
upstream is not by itself a sign of integration
* pushing it to the remote configured by branch.<name>.remote would
update its upstream, as determined by that remote's configured push
and fetch refspecs. For example, a local "main" that tracks
"origin/main" is kept even when remote.pushDefault names a fork.
Right after a pull it merely looks fully merged.
* it is the local upstream of a branch that is not being deleted, so
no branch is deleted out from under stacked work.
A branch whose work is not yet merged into its upstream is silently
skipped, so one unmerged topic does not abort the whole sweep.
Collect protected local upstreams without changing the candidate set
during ref iteration, then remove them after iteration. This makes the
result independent of ref iteration order. If a protected branch's own
upstream is deleted by the same sweep, clear its upstream configuration.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Teach delete_branches() a new mode for the upcoming --delete-merged
caller that checks whether a branch is merged into its upstream without
falling back to HEAD when there is no upstream. Existing callers keep
their current behavior.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a skip-unmerged mode to delete_branches() and check_branch_commit()
so a bulk caller can silently skip branches that are not fully merged
and carry on, rather than erroring with the "use 'git branch -D'"
advice that the plain "git branch -d" path emits. Existing callers are
unaffected.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
delete_branches() takes separate force and quiet parameters, while
check_branch_commit() takes force. The next commits would grow this
collection further. Replace them with a single unsigned flags argument
and an enum.
Test the FORCE and QUIET bits directly from flags at each use site so
that mutating or forwarding flags cannot leave cached values stale.
No change in behavior.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a --forked option to "git branch" list mode that lists only
branches whose configured upstream matches <branch>. The argument
can be a ref (e.g. "origin/main", "master"), a remote name like
"origin" for the branch its origin/HEAD points at, or a shell glob
(e.g. "origin/*"), and may be repeated to widen the filter.
It is an ordinary list filter, so it combines with the others:
git branch --merged origin/main --forked 'origin/*'
lists branches forked from origin that are already merged into
origin/main, and --no-merged inverts the question.
This is the building block for --delete-merged, which deletes the
listed branches once they have landed on their upstream.
Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'git worktree add <path> <branch>' DWIMs <branch> to a
remote-tracking branch when neither -b, -B, nor --detach
is given.
However, 'git worktree add -b <new-branch> <path> <branch>' can
still DWIM <branch>, causing <new-branch> to be ignored. This is a
regression introduced by 128e5496b3 (worktree add: extend DWIM to
infer --orphan, 2023-05-17), which appeared in Git 2.42.
Signed-off-by: Yoichi NAKAYAMA <yoichi.nakayama@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
All four `unsigned long`/`int`/`ssize_t` receivers across archive-zip,
diff, http-push and t/helper/test-pack-deltas were widened to `size_t`
in the prior commits, and remote-curl and fast-import were already
there. With every caller prepared, both the parameter and the return
type can now move without introducing any silent narrowing.
For inputs above zlib's `uLong` range (i.e. >4 GiB on platforms where
`uLong` is 32-bit, notably 64-bit Windows), defer to zlib's stored-block
formula (the same fallback it would itself use, see
https://github.com/madler/zlib/blob/v1.3.2/deflate.c#L832-L928 keeping
in mind that for large sizes, the `storelen` would be relevant, also
compare with https://github.com/madler/zlib/issues/549 for a fuller
story) plus the worst-case wrapper overhead. The existing path through
`deflateBound()` is unchanged for inputs that fit.
Assisted-by: Opus 4.7
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
