dilim/git
1
0
Fork 0
mirror of https://github.com/git/git.git synced 2026-08-05 15:41:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

diff: consult oid-only hunk providers via diff.<driver>.process

Browse Source 
The provider chain so far holds the diff-hunks store in front of the
terminal builtin computation.  Open it to external processes: a pair on
a path whose driver configures diff.<driver>.process is answered by a
long-running process speaking a pkt-line protocol (following the filter
process protocol), registered at the head of the chain and consulted
before the store and before any blob is loaded.

The protocol starts with the smallest request that can carry an answer:
object names alone.  A request is the pathname and the pair's
old-oid/new-oid, with no content.  The process answers with hunk lines,
with a zero-hunk success that asserts the blobs equivalent (trailing
newlines included), or with status=need-content, on which the pair
falls through to the builtin answer.  This serves the two shapes that
need no content pushed to them: a cache keyed on the blob pair, and a
process that fetches the blobs itself (for example over "git cat-file
--batch").  A pair whose side is not a stored blob carries a NULL id;
the provider sends no request and passes it.  Because Git holds no
content for the exchange, the answer is used as sent: hunks are
validated for order, overlap, lockstep alignment, and magnitude, then
replayed without the normalization xdiff applies to diffs it computes
itself.  The magnitude bound is the blobs' sizes, read from the object
database without loading content: a blob of N bytes holds at most N
lines.

Because the process's answer is authoritative, it outranks the store,
and its head-of-chain position says so.  A pair the process answers
never reaches the store and is never recorded, so nothing it produces
enters the store, which holds the builtin answer only.  A request it
does not answer, whether need-content, a missing capability, or a
missing id, passes down the chain to the builtin answer, which is what
the store serves, so the store may serve such a pair and a warming run
may record it.  Entries recorded before a process was configured are
not purged; a pair the process answers ignores them, and "git diff-hunks
clear" discards them.

The provider gates itself per request.  The driver is looked up by the
old-side path, so a renamed file resolves to the same driver, and by
the repository-relative path, so a diff.relative run from a
subdirectory names the pair the same way.  Options the process is never
told about select no process: the whitespace-ignoring options, -I,
--anchored, and an algorithm forced by option or configuration (blame
routes its algorithm through xdl_opts, so --histogram is covered).  The
request gains its last field, the path; the consumers change only by
filling it, and neither names the process.

The provider's state is its repository's pool of running processes,
keyed by the configured command, so drivers sharing a command share a
process, a submodule speaks to its own, and releasing the provider
(from repo_clear()) stops them.  The pool owns a copy of each command
string, so an entry outlives a config re-read.  A command that fails
stays as an entry that is not retried: its request and every later one
pass, so the store may serve the path for the rest of the command.

A protocol error in a response never kills the command.  The response
is read through a packet reader gentle about framing, so an error takes
one path: a single warning, the process stopped and marked failed, and
the builtin diff for the rest of the command.  That covers garbage
bytes, a truncated response, an empty packet, a bare status, and an
unrecognized status.  Semantically invalid coordinates cost only their
pair: the response is drained, the pair is computed, and the process
stays alive.  A path the protocol cannot carry (an embedded newline, or
one too long for a packet) falls back per path rather than costing the
command its process.  The handshake keeps one fatal check: a process
that announces a capability Git did not request aborts the command, as
the long-running filter protocol does.

Consulting is allowed per command, following the allow_textconv
precedent.  "git diff", "git log" and "git show", and "git blame" set
allow_diff_process; the plumbing diff commands and the interactive-patch
machinery never set it, so scripted and staging output stays builtin.
The options adjust the flag:

- --no-ext-diff clears it and --ext-diff sets it;
- --diff-process and --no-diff-process set and clear it alone, leaving
  external diff drivers as they were;
- format-patch clears it unconditionally, so a generated patch applies
  for recipients without the process;
- range-diff passes --no-ext-diff to the "git log" it compares.

git blame and the summary formats consult the process.  For blame, a
pair reported equivalent emits no hunks, so the whole commit passes to
its parent.  In the stat formats such a pair sums to a zero-count entry,
which the "nothing changed" rule omits, as under -w.  The subprocess is
long-running: one startup cost across a traversal, one round-trip per
consulted pair.  Answers travel in struct xdl_hunk, new in
xdiff-interface.h, holding xdiff's 1-based coordinates; nothing feeds
them back to xdiff, since only coordinate consumers consult.

A content-carrying request is the natural extension: it would serve
sides that are not stored blobs and processes that want content pushed
to them, and bring patch output and log -L's range tracking to the same
answer.  As it stands, a process's answers show in blame and the summary
formats while patch output stays builtin.

t4080 exercises the protocol, the per-command gate, and the error paths:

- each adversarial response shape warns and falls back to builtin, the
  request log proving which failures disable the process and which keep
  it alive (a malformed hunk line, coordinates past the blob size, a
  count overflowing strtol(), overlapping or misaligned hunks, an
  unrecognized status, a bare status, an empty packet, a mid-response
  crash, and raw garbage);
- a capability-less process and status=abort degrade without noise, and
  a failed start warns once and returns the path to the store;
- a trailing token on a hunk line is ignored, pinning field
  appendability;
- positive consults for git diff, git show, and diff-tree under
  --ext-diff and --diff-process; textconv output and gitlink sides are
  never identified; a diff.relative run consults by the repo-relative
  path;
- the equivalence answer is pinned from both consumers, and a warming
  run past a deferring process records the pair for a later read.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This commit is contained in:
Michael Montalbo
2026-08-01 10:41:53 -07:00
committed by Junio C Hamano
parent ffd18fb3ae
commit 91c8989d4e
24 changed files with 1916 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
Documentation/gitattributes.adoc
View File

@@ -832,6 +832,166 @@ NOTE: If `diff.<name>.command` is defined for path with the
(see above), and adding `diff.<name>.algorithm` has no effect, as the
algorithm is not passed to the external diff driver.
Answering diffs from a long-running process
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Unlike `diff.<name>.command`, which replaces the textual patch, the
process configured in `diff.<name>.process` feeds hunks back into
Git's own machinery:
it answers "which line ranges changed between these two blobs", and
Git's output is produced from that answer. A process is started
lazily, once per configured command string and per repository, and
consulted over a pkt-line protocol (following the long-running filter
process protocol; see the "Long Running Filter Process" section above
for the filter analogue).
The process is asked by object names alone: a request carries the
pathname and the `old-oid`/`new-oid` of the blob pair, and no content.
The pathname is the repository-relative old-side (preimage) path, not
the shortened display path a `--relative` diff shows, so a driver
scoped to a directory matches whatever directory the command runs from.
This suits a process that keeps a persistent cache keyed on the pair, and
a process that fetches the blobs itself (for example via
`git cat-file --batch`) to compute its own notion of the changed
lines. Pairs with a side that is not a stored blob (a working-tree
file, textconv output) are not sent; Git computes those itself.
The exchange opens with a handshake: Git announces its role and
version, the process replies in kind, and then Git lists the
capabilities it supports and the process replies with the ones it
implements. A process that announces a capability Git did not list
aborts the command.
-----------------------
packet: git> git-diff-client
packet: git> version=1
packet: git> 0000
packet: git< git-diff-server
packet: git< version=1
packet: git< 0000
packet: git> capability=hunks-by-oid
packet: git> 0000
packet: git< capability=hunks-by-oid
packet: git< 0000
-----------------------
After the handshake, each request and response looks like:
-----------------------
packet: git> command=hunks-by-oid
packet: git> pathname=path/file.c
packet: git> old-oid=<hex>
packet: git> new-oid=<hex>
packet: git> 0000
packet: git< hunk <old_start> <old_count> <new_start> <new_count>
packet: git< 0000
packet: git< status=success
packet: git< 0000
-----------------------
Start values are 1-based and counts are non-negative; a count of 0
describes a pure insertion or deletion at the 1-based line the change
sits before (a start of 0 is accepted for an empty file side). Hunks
must be listed in order, must not overlap, and must keep the unchanged
runs between them the same length on both sides; Git validates this
and, with a warning, falls back to its builtin diff on a response
that violates these rules.
A `status=success` response with zero hunks asserts that the blobs are
equivalent, including their trailing newlines. A process that cannot
answer a pair from its object names (or cannot rule out a
trailing-newline-only difference) responds `status=need-content`, and
Git produces that pair's diff itself. An asserted equivalence makes
the pair vanish from the summary formats, but the pair still counts
as changed for `--exit-code`, the same way a whitespace-only pair
does under `-w`.
The status names the disposition of the whole request. Git
understands three: `success` (the hunk lines are the answer),
`need-content` (Git produces this pair's diff itself), and `abort`, which
withdraws the capability the request used: Git stops sending
`hunks-by-oid` requests to that process for the rest of the command,
while the process stays alive for request forms negotiated under
other capabilities. Any other status is a protocol error: Git warns,
stops the process, and uses the builtin diff for the remainder of the
command.
Every response has the same shape whatever its status: zero or more
hunk lines, a flush packet, and a status packet terminated with a
flush packet. A response that carries no hunks, `need-content`
included, still begins with the empty hunk section's flush packet; a
bare status packet is a protocol error:
-----------------------
packet: git< 0000
packet: git< status=need-content
packet: git< 0000
-----------------------
The process must read the entire request before it responds; Git
writes the whole request before it reads the response.
The protocol extends without breaking deployed processes: a process
must ignore request keys it does not recognize, and Git ignores
trailing space-separated tokens after the last field of a hunk line,
so a later protocol version can append request keys and hunk fields.
New request forms arrive as capabilities, which a process may decline
to announce; announcing a capability Git did not request aborts the
command, as it does under the long-running filter process protocol.
There is no shutdown handshake: Git's side of the pipes closes when
the command exits, and the process should exit when it reads EOF. No
flush point is guaranteed, so a process that maintains persistent
state (such as a cache) should persist as it answers rather than at
exit. Git applies no timeout to a response; a process that hangs
hangs the command, as with the long-running filter processes.
`git blame` and the `--stat`, `--numstat`, and `--shortstat` formats
consult the process; the textual patch and `git log -L` range
tracking are produced by the builtin machinery, so a process whose
answers deliberately differ from the builtin diff shows that
difference only in blame and those formats. `--dirstat=lines` routes
through the diffstat path and consults; the other `--dirstat` modes do
not. A merge's `--stat` (including under `--cc`) is computed against
the first parent, so it consults like any other stat; the combined
patch itself compares one merge result against all of its parents at
once, which the pairwise request above does not express, so that patch
uses the builtin diff, and extending the protocol to combined diffs is
left for future work. A content-carrying extension of this
protocol would bring patch output and `git log -L` range tracking to
the same answer.
Consulting is allowed per command, as with textconv: `git diff`,
`git log` (`git whatchanged` included) and `git show`, and
`git blame` consult a configured process; the plumbing diff commands
do not unless `--ext-diff` or `--diff-process` is given explicitly,
and the interactive-patch commands (`git add -p` and friends), which
build the hunks they present from plumbing output, always stage from
the builtin diff. `git range-diff` generates the patches it
compares with `--no-ext-diff`. `--diff-process` and
`--no-diff-process` allow or forbid only the consulting;
`--no-ext-diff` disables all external diff mechanisms, this one
included. Options the process is never told about never select it:
with the whitespace-ignoring options, `--ignore-matching-lines`, and
`--anchored`, the pair is answered as when no process is configured.
`--diff-algorithm` (or a configured `diff.algorithm`) forces a builtin
algorithm and bypasses the process the same way. A per-path
++diff.++__<driver>__++.algorithm++ does so for `git diff` and the stat
formats, which build their diff parameters from it; `git blame` builds
its parameters from its own diff options, so a per-driver algorithm
does not by itself keep blame from consulting the process.
`git format-patch` never
consults the process, so generated patches are always based on the
builtin diff and apply for recipients without the process. On a
path whose driver has a process, the process is consulted before the
diff-hunks store (see linkgit:git-diff-hunks[1]): a pair the process
answers is never served from the store and never recorded into it.
A pair the process does not answer, for example with
`status=need-content`, gets the builtin diff, so the store may serve
it and a warming run may record it: the store holds builtin results,
and for such a pair the builtin result is what would be computed
anyway.
Defining a custom hunk-header
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^