Commit Graph

80862 Commits

Author SHA1 Message Date
Taylor Blau
dcccd99746 pack-bitmap: sort bitmaps before XORing
Reachability bitmaps may be stored as XORs against nearby bitmaps, up to
10 away. However, when callers provide selected commits in an arbitrary
order, the writer may miss good ancestor/descendant pairs and produce
much larger bitmap files without changing query coverage.

Sort the selected bitmaps in date order (from oldest to newest) before
computing XOR offsets, leaving pseudo-merge bitmaps alone (which we will
deal with separately in following commits).

On our same testing repository from previous commits, this change shrunk
our selection of 1,261 bitmaps from ~635.46 MiB to 176.4 MiB for a
~72.24% reduction in the on-disk size of our *.bitmap file. The time to
generate the smaller bitmap file decreased by ~3.69 seconds, though this
is likely mostly noise.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-28 05:23:01 +09:00
Taylor Blau
c720bbcc53 pack-bitmap: cache object positions during fill
The previous commits removed some redundant work from bitmap generation
by avoiding unnecessary tree recursion and by reusing selected bitmaps
that have already been computed.

Even with those changes in place, there is still an extremely hot path
from `fill_bitmap_commit()` and `fill_bitmap_tree()` to translate object
IDs into their corresponding bit positions in order to generate their
bitmaps.

In a small repository, this overhead is not significant. However, in a
very large repository (e.g., the one that we have been using as a
benchmark over the past several commits with ~57M total objects), the
overhead of locating object bit positions (often repeatedly) adds up
significantly.

Combat this by adding a small, direct-mapped cache to the bitmap writer
which maps object IDs to their corresponding bit positions. Size the
cache according to the number of objects being written, with fixed lower
and upper bounds so small repositories do not pay for a large table and
large repositories can avoid most repeated packlist and MIDX lookups.

On my machine with (a somewhat outdated) GCC 15.2.0, each entry in the
cache is 40 bytes wide:

    $ pahole -C bitmap_pos_cache_entry pack-bitmap-write.o
    struct bitmap_pos_cache_entry {
            struct object_id           oid;                  /*     0    36 */
            uint32_t                   pos;                  /*    36     4 */

            /* size: 40, cachelines: 1, members: 2 */
            /* last cacheline: 40 bytes */
    };

, and we will allocate up to 2^21 entries for a maximum total of 80 MiB
of cache overhead.

In our example repository from above and in earlier commits, this
results in a ~9.4% reduction in runtime relative to the previous commit:

    +------------------+-------------+-------------+---------------------+
    |                  | HEAD^       | HEAD        | Delta               |
    +------------------+-------------+-------------+---------------------+
    | elapsed          |   324.8 s   |   294.1 s   |    -30.7 s  (-9.4%) |
    | cycles           | 1,508.6 B   | 1,365.5 B   |   -143.0 B  (-9.5%) |
    | instructions     | 1,436.6 B   | 1,389.8 B   |    -46.9 B  (-3.3%) |
    | CPI              |     1.050   |     0.983   |   -0.068    (-6.4%) |
    +------------------+-------------+-------------+---------------------+

When generating bitmaps on this repository (to produce the above
timings), the cache grew to its maximum size of 80 MiB, and resulted in
1.024B cache hits and 59.957M cache misses.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-28 05:23:01 +09:00
Taylor Blau
ece3465d44 pack-bitmap: consolidate find_object_pos() success path
Both sides of `find_object_pos()` report success in the same way by
setting the optional `found` out-parameter and return the resolved
bitmap position.

Prepare for adding more bookkeeping around object-position lookups by
storing the result in a local `pos` variable and sharing the success
return path between the packlist and MIDX cases.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-28 05:23:00 +09:00
Taylor Blau
3ea5fe8482 pack-bitmap: reuse stored selected bitmaps
When `fill_bitmap_commit()` reaches an ancestor that was selected for
its own bitmap and processed earlier, its object closure is already
stored in `writer->bitmaps` as an EWAH bitmap. As a result, walking
through that commit's tree and parents again is redundant.

Teach `fill_bitmap_commit()` to notice that case. For non-root commits in
the walk, look for a stored selected bitmap and OR it into the bitmap
being built. If one exists, skip the commit, its tree, and its parents.

Building bitmaps from scratch on the same test repository from the
previous commits yields a significant speed-up:

    +------------------+-------------+-------------+---------------------+
    |                  | HEAD^       | HEAD        | Delta               |
    +------------------+-------------+-------------+---------------------+
    | elapsed          |   562.8 s   |   324.8 s   |   -237.9 s (-42.3%) |
    | cycles           | 2,621.3 B   | 1,508.6 B   | -1,112.7 B (-42.4%) |
    | instructions     | 2,348.9 B   | 1,436.6 B   |   -912.3 B (-38.8%) |
    | CPI              |     1.116   |     1.050   |   -0.066    (-5.9%) |
    +------------------+-------------+-------------+---------------------+

In our testing repository, there are 1,261 commits selected for bitmap
coverage, and 1,382 maximal commits induced as a result of that. Of the
1,382 calls made to `fill_bitmap_commit()` (one per maximal commit), 131
of them can be short-circuited at some point during their traversal as a
consequence of this change.

In large repositories where the cost of filling the bitmap for any
individual commit is large, being able to short-circuit even ~9.5% of
the calls to `fill_bitmap_commit()` results in a significant savings.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-28 05:23:00 +09:00
Taylor Blau
1760c37258 pack-bitmap: check subtree bits before recursing
In the previous commit, we adjusted the callers of `fill_bitmap_tree()`
to pass in the bit position of the tree they wish to fill.

This commit makes use of that information at the call site to avoid
setting up a stack frame for fill_bitmap_tree() entirely whenever a
tree's bit position is already set.

Since this is such a hot path, the avoided cost of setting up and
tearing down stack frames for each noop'd call to `fill_bitmap_tree()`
is significant:

    +--------------+-------------+-------------+-------------------+
    |              | HEAD^       | HEAD        | Delta             |
    +--------------+-------------+-------------+-------------------+
    | elapsed      |   582.4 s   |   562.8 s   |  -19.6 s  (-3.4%) |
    | cycles       | 2,713.3 B   | 2,621.3 B   |  -92.0 B  (-3.4%) |
    | instructions | 2,415.5 B   | 2,348.9 B   |  -66.6 B  (-2.8%) |
    | CPI          |     1.123   |     1.116   |  -0.007   (-0.7%) |
    +--------------+-------------+-------------+-------------------+

In the same repository as in the previous commit, our timings dropped
from ~582.4 seconds down to ~562.77 seconds.

While the cycles-per-instruction ratio is basically unchanged, we
execute significantly fewer instructions, and correspondingly fewer
cycles.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-28 05:23:00 +09:00
Taylor Blau
e3959cc78c pack-bitmap: pass object position to fill_bitmap_tree()
In the following commit, callers of `fill_bitmap_tree()` will be
required to check the bit corresponding to their tree before calling
that function. That change will reduce the overhead of setting up and
tearing down stack frames for trees whose bits are already set.

To prepare for that change, have callers pass in the tree's bit position
in `fill_bitmap_tree()`, which will make the next commit easier to read.

In the meantime, this change has a surprising and measurable benefit
during bitmap generation, particularly on very large repositories.

When processing sub-trees within `fill_bitmap_tree()`, the preimage of
this patch did the following:

    while (tree_entry(&desc, entry)) {
        switch (object_type(entry.mode)) {
        case OBJ_TREE:
            if (fill_bitmap_tree(writer, bitmap,
                                 lookup_tree(writer->repo,
                                             &entry.oid)) < 0) {
                /* ... */
            }
            /* ... */
        }
    }

, first performing the object lookup via `lookup_tree()`, and then
locating its bit position within the recursive call. This patch
effectively reorders those two calls so that we first discover the
sub-tree's bit position, *then* load its tree.

By reordering these two operations, we spend fewer CPU cycles per
instruction, likely due to improved CPU dependency/cache/pipeline
behavior. Comparing the results of: running `perf stat` before and after
this commit, we have:

    +--------------+-------------+-------------+-------------------+
    |              | HEAD^       | HEAD        | Delta             |
    +--------------+-------------+-------------+-------------------+
    | elapsed      |   612.5 s   |   582.4 s   |  -30.1 s  (-4.9%) |
    | cycles       | 2,857.3 B   | 2,713.3 B   | -144.0 B  (-5.0%) |
    | instructions | 2,413.2 B   | 2,415.5 B   |   +2.3 B  (+0.1%) |
    | CPI          |     1.184   |     1.123   |  -0.061   (-5.1%) |
    +--------------+-------------+-------------+-------------------+

In a large repository with ~4.8M commit, and ~37.1M tree objects this
change improves timing from ~612.5 seconds down to ~582.4 seconds, or a
~4.9% improvement. More importantly, the number of CPU cycles spent
dropped off significantly as a result of this commit, lowering our
cycles-per-instruction ratio by about ~5.1%.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-28 05:23:00 +09:00
Junio C Hamano
d9e49a6abe Merge branch 'tb/pseudo-merge-bugfixes' into tb/bitmap-build-performance
* tb/pseudo-merge-bugfixes:
  pack-bitmap: prevent pattern leak on pseudo-merge re-assignment
  Documentation: fix broken `sampleRate` in gitpacking(7)
  pack-bitmap: reject pseudo-merge "sampleRate" of 0
  pack-bitmap: parse commits in `find_pseudo_merge_group_for_ref()`
  pack-bitmap: fix pseudo-merge lookup for shared commits
  pack-bitmap: fix inverted binary search in `pseudo_merge_at()`
  pack-bitmap-write: sort pseudo-merge commit lookup table in pack order
  t5333: demonstrate various pseudo-merge bugs
  t/helper: add 'test-tool bitmap write' subcommand
2026-05-20 11:32:39 +09:00
Junio C Hamano
1c00d2d839 The 5th batch
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-20 10:30:58 +09:00
Junio C Hamano
f5fc0f53de Merge branch 'sb/unpack-index-pack-buffer-resize'
Use a larger buffer size in the code paths to ingest pack stream.

* sb/unpack-index-pack-buffer-resize:
  index-pack, unpack-objects: increase input buffer from 4 KiB to 128 KiB
2026-05-20 10:30:58 +09:00
Junio C Hamano
ca7d7d6424 Merge branch 'ps/history-fixup'
"git history" learned "fixup" command.

* ps/history-fixup:
  builtin/history: introduce "fixup" subcommand
  builtin/history: generalize function to commit trees
  replay: allow callers to control what happens with empty commits
2026-05-20 10:30:57 +09:00
Junio C Hamano
345bc60828 Merge branch 'jh/alias-i18n-fixes'
Further update to the i18n alias support to avoid regressions.

* jh/alias-i18n-fixes:
  alias: restore support for simple dotted aliases
2026-05-20 10:30:57 +09:00
Junio C Hamano
cf7151110d Merge branch 'bc/sign-commit-with-custom-encoding'
Signing commit with custom encoding was passing the data to be
signed at a wrong stage in the pipeline, which has been corrected.

* bc/sign-commit-with-custom-encoding:
  commit: sign commit after mutating buffer
  commit: name UTF-8 function appropriately
2026-05-20 10:30:57 +09:00
Junio C Hamano
18581f836f Merge branch 'js/adjust-tests-to-explicitly-access-bare-repo'
Some tests assume that bare repository accesses are by default
allowed; rewrite some of them to avoid the assumption, rewrite
others to explicitly set safe.bareRepository to allow them.

* js/adjust-tests-to-explicitly-access-bare-repo:
  safe.bareRepository: default to "explicit" with WITH_BREAKING_CHANGES
  status tests: filter `.gitconfig` from status output
  ls-files tests: filter `.gitconfig` from `--others` output
  t5601: restore `.gitconfig` after includeIf test
  t1305: use `--git-dir=.` for bare repo in include cycle test
  t1300: remove global config settings injected by test-lib.sh
  t7900: do not let `$HOME/.gitconfig` interfere with XDG tests
  test-lib: allow bare repository access when breaking changes are enabled
2026-05-20 10:30:57 +09:00
Junio C Hamano
3b3a3ef066 Merge branch 'en/diffstat-utf8-truncation-fix'
The computation to shorten the filenames shown in diffstat measured
width of individual UTF-8 characters to add up, but forgot to take
into account error cases (e.g., an invalid UTF-8 sequence, or a
control character).

* en/diffstat-utf8-truncation-fix:
  diff: fix out-of-bounds reads and NULL deref in diffstat UTF-8 truncation
2026-05-20 10:30:56 +09:00
Junio C Hamano
91ddfe3d5c Merge branch 'js/mingw-no-nedmalloc'
Stop using unmaintained custom allocator in Windows build which was
the last user of the code.

* js/mingw-no-nedmalloc:
  mingw: remove the vendored compat/nedmalloc/ subtree
  mingw: drop the build-system plumbing for nedmalloc
  mingw: stop using nedmalloc
2026-05-20 10:30:56 +09:00
Junio C Hamano
a6876b2068 Merge branch 'js/objects-larger-than-4gb-on-windows'
Update code paths that assumed "unsigned long" was long enough for
"size_t".

* js/objects-larger-than-4gb-on-windows:
  ci: run expensive tests on push builds to integration branches
  t5608: mark >4GB tests as EXPENSIVE
  test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1
  test-tool synthesize: precompute pack for 4 GiB + 1
  test-tool synthesize: use the unsafe hash for speed
  t5608: add regression test for >4GB object clone
  test-tool: add a helper to synthesize large packfiles
  delta, packfile: use size_t for delta header sizes
  odb, packfile: use size_t for streaming object sizes
  git-zlib: handle data streams larger than 4GB
  index-pack, unpack-objects: use size_t for object size
2026-05-20 10:30:56 +09:00
Junio C Hamano
7bcaabddcf The 4th batch
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-19 09:57:46 +09:00
Junio C Hamano
bad1648bf0 Merge branch 'sj/submodule-update-clone-config-fix'
The configuration variable submodule.fetchJobs was not read correctly,
which has been corrected.

* sj/submodule-update-clone-config-fix:
  submodule-config: fix reading submodule.fetchJobs
2026-05-19 09:57:46 +09:00
Junio C Hamano
ded7b8cf55 Merge branch 'rs/sideband-clear-line-before-print'
Tweak the way how sideband messages from remote are printed while
we talk with a remote repository to avoid tickling terminal
emulator glitches.

* rs/sideband-clear-line-before-print:
  sideband: clear full line when printing remote messages
2026-05-19 09:57:46 +09:00
Junio C Hamano
1cde0a68d8 Merge branch 'ag/rebase-update-refs-limit-to-branches'
"git rebase --update-refs", when used with an rebase.instructionFormat
with "%d" (describe) in it, tried to update local branch HEAD by
mistake, which has been corrected.

* ag/rebase-update-refs-limit-to-branches:
  rebase: ignore non-branch update-refs
2026-05-19 09:57:46 +09:00
Junio C Hamano
c297fdf18c Merge branch 'kh/doc-restore-double-underscores-fix'
Doc update.

* kh/doc-restore-double-underscores-fix:
  doc: restore: remove double underscore
2026-05-19 09:57:45 +09:00
Junio C Hamano
55c4feae1c Merge branch 'kh/doc-commit-graph'
Ramifications of turning off commit-graph has been documented a bit
more clearly.

* kh/doc-commit-graph:
  doc: add caveat about turning off commit-graph
2026-05-19 09:57:44 +09:00
Junio C Hamano
056472b82d Merge branch 'kh/name-rev-custom-format'
A new builtin "git format-rev" is introduced for pretty formatting
one revision expression per line or commit object names found in
running text.

* kh/name-rev-custom-format:
  format-rev: introduce builtin for on-demand pretty formatting
  name-rev: make dedicated --annotate-stdin --name-only test
  name-rev: factor code for sharing with a new command
  name-rev: run clang-format before factoring code
  name-rev: wrap both blocks in braces
2026-05-19 09:57:44 +09:00
Junio C Hamano
6bfdc87e99 Merge branch 'sg/t6112-unwanted-tilde-expansion-fix'
Test fix.

* sg/t6112-unwanted-tilde-expansion-fix:
  t6112: avoid tilde expansion
2026-05-19 09:57:44 +09:00
Junio C Hamano
0330568152 Merge branch 'en/xdiff-cleanup-3'
Preparation of the xdiff/ codebase to work with Rust.

* en/xdiff-cleanup-3:
  xdiff/xdl_cleanup_records: make execution of action easier to follow
  xdiff/xdl_cleanup_records: make setting action easier to follow
  xdiff/xdl_cleanup_records: make limits more clear
  xdiff/xdl_cleanup_records: use unambiguous types
  xdiff: use unambiguous types in xdl_bogo_sqrt()
  xdiff/xdl_cleanup_records: delete local recs pointer
2026-05-19 09:57:44 +09:00
Junio C Hamano
60afbae98d Merge branch 'mc/http-emptyauth-negotiate-fix'
The 'http.emptyAuth=auto' configuration now correctly attempts
Negotiate authentication before falling back to manual credentials.
This allows seamless Kerberos ticket-based authentication without
requiring users to explicitly set 'http.emptyAuth=true'.

* mc/http-emptyauth-negotiate-fix:
  doc: clarify http.emptyAuth values
  t5563: add tests for http.emptyAuth with Negotiate
  http: attempt Negotiate auth in http.emptyAuth=auto mode
  http: extract http_reauth_prepare() from retry paths
2026-05-19 09:57:43 +09:00
Junio C Hamano
68aca6b912 The 3rd batch
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-17 22:58:31 +09:00
Junio C Hamano
6d7492cf85 Merge branch 'rs/grep-column-only-match-fix'
"git grep" update.

* rs/grep-column-only-match-fix:
  grep: fix --column --only-match for 2nd and later matches
2026-05-17 22:58:31 +09:00
Junio C Hamano
d17a7b8191 Merge branch 'hn/git-checkout-m-with-stash'
"git checkout -m another-branch" was invented to deal with local
changes to paths that are different between the current and the new
branch, but it gave only one chance to resolve conflicts.  The command
was taught to create a stash to save the local changes.

* hn/git-checkout-m-with-stash:
  checkout -m: autostash when switching branches
  checkout: rollback lock on early returns in merge_working_tree
  sequencer: teach autostash apply to take optional conflict marker labels
  sequencer: allow create_autostash to run silently
  stash: add --label-ours, --label-theirs, --label-base for apply
2026-05-17 22:58:31 +09:00
Junio C Hamano
068c10c741 Merge branch 'pw/rename-to-get-current-worktree'
Code clean-up.

* pw/rename-to-get-current-worktree:
  worktree: rename get_worktree_from_repository()
2026-05-17 22:58:30 +09:00
Junio C Hamano
c26c58de0c Merge branch 'mf/format-patch-cover-letter-format-docfix'
Docfix.

* mf/format-patch-cover-letter-format-docfix:
  Fix docs for format.commitListFormat
2026-05-17 22:58:30 +09:00
Junio C Hamano
3cfbb7b3f7 Merge branch 'en/ort-cached-rename-with-trivial-resolution'
"ort" merge backend improvements.

* en/ort-cached-rename-with-trivial-resolution:
  merge-ort: handle cached rename & trivial resolution interaction better
2026-05-17 22:58:30 +09:00
Junio C Hamano
f7ae594a85 Merge branch 'ss/t7004-unhide-git-failures'
Test clean-up.

* ss/t7004-unhide-git-failures:
  t7004: avoid subshells to capture git exit codes
  t7004: dynamically grab expected state in tests
  t7004: drop hardcoded tag count for state verification
2026-05-17 22:58:30 +09:00
Junio C Hamano
27caa6b4f7 Merge branch 'en/backfill-fixes-and-edges'
The 'git backfill' command now rejects revision-limiting options that
are incompatible with its operation, uses standard documentation for
revision ranges, and includes blobs from boundary commits by default
to improve performance of subsequent operations.

* en/backfill-fixes-and-edges:
  backfill: default to grabbing edge blobs too
  backfill: document acceptance of revision-range in more standard manner
  backfill: reject rev-list arguments that do not make sense
2026-05-17 22:58:29 +09:00
Junio C Hamano
59ff4886a5 Merge branch 'ps/clang-w-glibc-2.43-and-_Generic'
Headers from glibc 2.43 when used with clang does not allow
disabling C11 language features, causing build failures..

* ps/clang-w-glibc-2.43-and-_Generic:
  build: tolerate use of _Generic from glibc 2.43 with Clang
2026-05-13 10:57:55 +09:00
Junio C Hamano
29bd7ed512 The second batch
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 11:04:46 +09:00
Junio C Hamano
d14a9794dd Merge branch 'js/maintenance-fix-deadlock-on-win10'
To help Windows 10 installations, avoid removing files whose
contents are still mmap()'ed.

* js/maintenance-fix-deadlock-on-win10:
  maintenance(geometric): do release the `.idx` files before repacking
  mingw: optionally use legacy (non-POSIX) delete semantics
2026-05-12 11:04:46 +09:00
Junio C Hamano
8fa3c82fcc Merge branch 'jc/t5551-fix-expensive'
Test fix.

* jc/t5551-fix-expensive:
  t5551: "GIT_TEST_LONG=Yes make test" is broken
2026-05-12 11:04:45 +09:00
Junio C Hamano
1b6e85c7cd Merge branch 'js/t5564-socks-use-short-path'
Avoid hitting the pathname limit for socks proxy socket during the
test..

* js/t5564-socks-use-short-path:
  t5564: use a short path for the SOCKS proxy socket
2026-05-12 11:04:45 +09:00
Junio C Hamano
14e08ec60c Merge branch 'js/ci-github-actions-update'
Update various GitHub Actions versions.

* js/ci-github-actions-update:
  l10n: bump mshick/add-pr-comment from v2 to v3
  ci: bump git-for-windows/setup-git-for-windows-sdk from v1 to v2
  ci: bump actions/checkout from v5 to v6
  ci: bump actions/github-script from v8 to v9
  ci: bump actions/{upload,download}-artifact to v7 and v8
  ci: bump microsoft/setup-msbuild from v2 to v3
2026-05-12 11:04:44 +09:00
Junio C Hamano
a3325b06a3 Merge branch 'jk/revert-aa-reap-transport-child-processes'
Revert a recent change that introduced a regression to help mksh users.

* jk/revert-aa-reap-transport-child-processes:
  Revert "transport-helper, connect: use clean_on_exit to reap children on abnormal exit"
2026-05-12 11:04:44 +09:00
Taylor Blau
5e6e8dc786 pack-bitmap: prevent pattern leak on pseudo-merge re-assignment
When "bitmapPseudoMerge.*.pattern" appears more than once for the same
group, `pseudo_merge_config()` frees the old `regex_t *` pointer
but does not call `regfree()` on it first. This leaks whatever internal
state `regcomp()` allocated.

The final cleanup path in `pseudo_merge_group_release()` does call
`regfree()` before `free()`, so only the intermediate replacement is
affected.

Fix this by guarding the replacement with a NULL check and calling
`regfree()` before `free()` when the pointer is non-NULL.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
84780db636 Documentation: fix broken sampleRate in gitpacking(7)
The documentation explaining some sample configurations for bitmap
pseudo-merges incorrectly uses a sample rate outside of the allowed
(0,1] range.

This dates back to faf558b23e (pseudo-merge: implement support for
selecting pseudo-merge commits, 2024-05-23), and was likely written when
the allowable range for this configuration was the integral values
between (0,100].

Fix this to conform to the actual allowable range for this
configuration.

Noticed-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
03c7a30cee pack-bitmap: reject pseudo-merge "sampleRate" of 0
The "bitmapPseudoMerge.*.sampleRate" configuration controls what
fraction of unstable commits are included in each pseudo-merge group.
The config validation accepts values in the range `[0, 1]`, but a value
of exactly 0 causes a division by zero in `select_pseudo_merges_1()`:

    if (j % (uint32_t)(1.0 / group->sample_rate))

When `sample_rate` is 0, `1.0 / 0.0` produces `+inf`, and casting
infinity to `uint32_t` is undefined behavior in C. On most platforms
this yields 0, making the subsequent modulo operation (`j % 0`) a
fatal arithmetic trap.

This path was not previously reachable because an earlier bug caused
all pseudo-merge candidates to be classified as "stable" (where the
sampling rate is not used), regardless of their actual commit date. Now
that the date classification is fixed, the unstable path is exercised
and the division by zero can fire.

Fix this by changing the validation to require a strict lower bound and
thus reject 0.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
78e85e05f3 pack-bitmap: parse commits in find_pseudo_merge_group_for_ref()
`find_pseudo_merge_group_for_ref()` uses the commit's date to classify
it as either "stable" (older than the stable threshold) or "unstable"
(otherwise).

However, to find the relevant commit from a given OID, the function
`find_pseudo_merge_group_for_ref()` uses `lookup_commit()` which does
not parse commits.

Because an unparsed commit has its "date" set to zero, every candidate
is placed in the "stable" bucket regardless of its actual committer
timestamp. This means the `bitmapPseudoMerge.*.threshold` and
`stableThreshold` configuration options have no effect: the
stable/unstable split is always determined by comparing against zero
rather than the real commit date.

The net result is that pseudo-merge groups are partitioned by
`stableSize` instead of the intended decay-based sizing, and the
`sampleRate` knob (which only applies to the unstable path) is never
exercised.

Fix this by calling `repo_parse_commit()` after `lookup_commit()`,
bailing out of the callback if parsing fails.

The corresponding test configures two pseudo-merge groups that both
match all tags. The "stable" group uses `threshold=1.month.ago`, and the
"all" group uses `threshold=now`. The test use our custom
"GIT_TEST_DATE_NOW" environment variable by setting it to the value of
"$test_tick" to align Git's notion of "now" (and therefore
"1.month.ago") with the `test_tick` timestamps, so the commits appear to
be younger than one month: only the "all" group matches them, producing
exactly one pseudo-merge.

Without the fix every commit has `date == 0`, which satisfies `date <=
threshold` for both groups (since 0 is older than one month ago), and
the "stable" group erroneously matches as well.

Now that commits are correctly classified as "unstable", the bug
described in the test exercising the "sampleRate=0" test is reachable,
and the test is marked as failing. It will be fixed in a following
commit.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
8b5f199f30 pack-bitmap: fix pseudo-merge lookup for shared commits
When a commit appears in more than one pseudo-merge group, its entry in
the commit lookup table has the high bit set in its offset field,
indicating that the offset points to an "extended" table containing the
set of pseudo-merges for that commit.

There are three bugs in this path:

 * The `next_ext` offset in `write_pseudo_merges()` undercounts the
   per-entry size of the lookup table (8 vs. 12 bytes).

 * `nth_pseudo_merge_ext()` calls `read_pseudo_merge_commit_at()` on a
   pseudo-merge bitmap offset, misinterpreting it as a 12-byte commit
   table entry.

 * The error check after `pseudo_merge_ext_at()` in
   `apply_pseudo_merges_for_commit()` tests `< -1` instead of `< 0`,
   silently swallowing errors from `error()`.

The first bug is on the write side: each commit lookup entry contains a
4- and 8-byte unsigned value for a total of 12 bytes, but the
calculation assumes that the entry only contains 8 bytes of data. This
makes `next_ext` too small, so the extended-table offsets that get
written point into the middle of the non-extended lookup table rather
than past it. The reader then interprets non-extended lookup data as
extended entries, producing garbage.

The second bug is on the read side and is independently fatal: even with
a correctly positioned extended table, `nth_pseudo_merge_ext()` feeds
the offset it reads (which points at pseudo-merge bitmap data) to
`read_pseudo_merge_commit_at()`. That function tries to parse 12 bytes
as a `pseudo_merge_commit` struct, clobbering `merge->pseudo_merge_ofs`
with whatever happens to be at that location. The caller only needs
`pseudo_merge_ofs`, so the fix is to store the offset directly rather
than re-parsing a commit table entry. The `commit_pos` field is left
untouched, retaining the value that `find_pseudo_merge()` set earlier.

The third bug is latent. With the first two fixes applied, the extended
table is correctly written and read, so `pseudo_merge_ext_at()` does not
fail during normal operation. The `< -1` vs `< 0` distinction only
matters when the bitmap file is corrupt or truncated, in which case the
error would be silently ignored and the code would proceed with
uninitialized data.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
b1e3fcdb9b pack-bitmap: fix inverted binary search in pseudo_merge_at()
The binary search in `pseudo_merge_at()` has its "lo" and "hi" updates
swapped: when the midpoint's offset is less than the target, it sets `hi
= mi` (searching left) instead of `lo = mi + 1` (searching right), and
vice versa.

This means that lookups for pseudo-merges whose offset is not near the
midpoint of the pseudo-merge table are likely to fail.

In practice, with a single pseudo-merge group this is masked because the
lone entry is always at the midpoint. With multiple groups, the inverted
comparisons cause lookups to search in the wrong direction, potentially
missing entries.

Swap the "lo" and "hi" assignments to search in the correct direction,
making it possible to apply pseudo-merges during fill-in when more than
one pseudo-merge exists in a group.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
ff21343a59 pack-bitmap-write: sort pseudo-merge commit lookup table in pack order
The pseudo-merge commit lookup table stores each commit's position in
the pack- or pseudo-pack order, and is used to perform a binary search
in order to determine which pseudo-merge(s) a given commit belongs to.

However, the table was previously sorted in lexical order (via
`oid_array_sort()`), causing the binary search to fail.

While this causes pseudo-merge bitmaps to be de-facto broken for fill-in
traversal, there are a couple of important points to keep in mind:

 * Pseudo-merge application during the initial phases of a bitmap-based
   traversal are applied via `cascade_pseudo_merges_1()`. This function
   enumerates the known pseudo-merges and determines if its parents are
   a subset of the traversal roots.

   This is a different path than the fill-in traversal, where we are
   looking for any pseudo-merges which may be satisfied after visiting
   some commit along an object walk, which involves the aforementioned
   (broken) binary search.

   As a consequence, any pseudo-merges we apply at this stage are done
   so correctly.

 * While this bug makes applying pseudo-merges during fill-in traversal
   effectively broken, it does not produce wrong results. Instead of
   applying the *wrong* pseudo-merge, we will simply fail to find
   satisfied pseudo-merges, leaving the traversal to use the existing
   fill-in routines.

Fix this by sorting the table by bit position before writing, matching
the order that the reader's binary search expects.

This does produce a change the on-disk format insofar as the actual code
now complies with the documented format (for more details, refer to:
Documentation/technical/bitmap-format.adoc). Given that this never
worked in the first place, such a change should be OK to perform.

If an out-of-tree implementation of pseudo-merges happened to generate
bitmaps that comply with the documented format, they will continue to be
read and interpreted as normal.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
49369d8290 t5333: demonstrate various pseudo-merge bugs
Using the test helper introduced via the previous commit, add various
failing tests demonstrating bugs in the pseudo-merge implementation.

These are all marked as failing with one exception. The "sampleRate=0"
test describes a latent bug, which is only reachable through a code path
that is itself masked by a separate bug. A future commit will fix that
bug, and, in turn, cause the aforementioned test to fail. Accordingly,
that commit will mark the test as failing, and it will be re-marked as
passing in a separate commit which fixes the once-latent bug.

For the rest: the following commits will explain and fix the underlying
bugs in detail.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:18 +09:00
Taylor Blau
bc2f611569 t/helper: add 'test-tool bitmap write' subcommand
In f16eb1c091 (pseudo-merge: fix disk reads from find_pseudo_merge(),
2026-03-31), we noted that `apply_pseudo_merges_for_commit()` is never
triggered by the existing test suite, and that this bears further
investigation.

This patch is the first one to begin that investigation. The following
patches will expose and fix a variety of bugs in the implementation of
pseudo-merge bitmaps.

In order to do so, however, many of these tests require very precise
selection of which commits receive bitmaps and which do not. To date,
there isn't a standard approach to easily facilitate this. Address this
by introducing a `test-tool bitmap write` subcommand that writes a
bitmap for a given packfile, reading the set of commits which should
receive individual bitmaps from stdin like so:

    test-tool bitmap write <pack-basename> </path/to/commits.list

, where "<pack-basename>" is the filename for a specific packfile (e.g.,
"pack-abc123.pack"), and "/path/to/commits.list" is a list of commit
OIDs which will receive bitmaps.

The helper respects `bitmapPseudoMerge.*` configuration for creating
pseudo-merge bitmaps alongside the regular commit bitmaps.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-05-12 10:36:17 +09:00