The 'struct odb_read_stream' and 'struct odb_write_stream'
structures have been consolidated into a single unified 'struct
odb_stream' structure, simplifying object database streaming APIs
and enabling streaming of arbitrary object types.
* ps/odb-streams:
odb/streaming: unify function names to create new streams
odb/streaming: rename `struct input_zstream_data`
odb/streaming: rename `struct read_object_fd_data`
odb/streaming: consolidate read and write streams
odb/streaming: rename `struct odb_read_stream`
odb/streaming: support streaming arbitrary object types
odb/streaming: drop `is_finished` field
odb/streaming: track write stream size in the structure
A new 'diff.<driver>.process' configuration has been introduced to
allow a long-running external process to act as a hunk provider,
enabling external tools to control which lines Git considers changed
while leaving all output formatting (word diff, color, blame, etc.) to
Git's standard pipeline.
* mm/diff-process-hunks:
diff: consult oid-only hunk providers via diff.<driver>.process
userdiff: add diff.<driver>.process config
sub-process: add a gentle status read
sub-process: separate process lifecycle from hashmap management
blame: read precomputed hunks
diff: read precomputed hunks for stat output
diff: record precomputed hunks during stat output
diff-hunks: add the store format, library, and command
diff: introduce a hunk provider interface
gitattributes: document how external diff drivers relate to diff features
The creation of the on-disk data structures for the object database
has been made pluggable, allowing future backends to customize their
setup. As part of this, the initialization of the object database
has been deferred, and the loading of the loose-object map has been
detangled from repository initialization.
* ps/odb-make-creation-pluggable:
odb: make creation of on-disk structures pluggable
odb/source: introduce function to map source type to name
setup: defer object database creation
setup: handle ODB-related environment variables in `odb_new()`
setup: detangle loading of loose object maps
loose: load loose object map for the correct source
When initializing a repository's object database we have to respect the
GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment
variables, which can be set by the user to override the default location
of where we write objects to and read objects from.
This is handled in `apply_repository_format()`, which is fine. But in a
subsequent commit we'll have to defer constructing the object database
to a later point in some cases, and that will require a second site
where we call `odb_new()`. And of course, that second site would have to
handle those environment variables, as well.
It would be somewhat awkward to duplicate the logic though. But there's
a better alternative: instead of handling this logic in "setup.c", we
can easily handle environment variables in `odb_new()` itself. This
ensures that object database creation is neatly self-contained, and we
don't have to duplicate any of the logic.
Another benefit is that in a future patch series we plan to move
handling of alternates into the backends themselves [1], and that will
require us to also handle those environment variables in the "files"
backend itself. So moving the logic into the ODB level already gets us
one step closer to that goal.
Refactor the logic accordingly.
[1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Object database housekeeping in 'git gc' and 'git maintenance' has
been refactored to be pluggable. The files-backend-specific logic,
including incremental and geometric repacking as well as object
pruning, has been moved out of the command implementation and into the
files object database source, enabling future alternative object
database backends to implement their own housekeeping services.
* ps/odb-pluggable-housekeeping:
odb: make optimizations pluggable
builtin/gc: fix signedness issues in ODB-related functionality
builtin/gc: refactor ODB optimizations to operate on "files" source
builtin/gc: introduce `odb_optimize_required()`
builtin/gc: move geometric repacking into `odb_optimize()`
builtin/gc: introduce object database optimization options
builtin/gc: inline config values specific to the "files" backend
builtin/gc: make repack arguments self-contained
builtin/gc: extract object database optimizations into separate function
builtin/gc: move worktree and rerere tasks before object optimizations
odb: run "pre-auto-gc" hook for all maintenance tasks
t7900: simplify how we check for maintenance tasks
The `struct odb_read_stream` and `struct odb_write_stream` both provide
the same functionality: they allow a caller to read object data from an
arbitrary source. Historically, the only difference was that the read
stream was used to read data out of the object database, whereas the
write stream was used to write data into the object database, but the
interfaces were mostly the same.
Over the preceding commits we have refactored the write stream to have
almost exactly the same interface as the read stream. With these
refactorings we can now easily merge those two streams into a single
interface that's used for both use cases.
While most of the changes are mechanical, there are two sites that need
special mention:
- "builtin/unpack-objects.c" creates a write stream from compressed
object data.
- "odb/streaming.c" creates a write stream from a file descriptor.
Adapting these sites to yield the new stream type requires a couple more
changes. Most importantly, instead of embedding the pointer to the data
in `struct odb_write_stream`, we now allocate a structure that wraps the
new `struct odb_stream` base. Other than that though, the changes are
rather straight forward.
Some of the structures and functions are now somewhat misnamed. These
will be fixed in subsequent commits.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When passing around a `struct odb_write_stream` we typically also have
to pass the number of bytes that the stream will yield. This is required
because the object header itself contains that size, and consequently we
cannot write the header without that information.
Move this information into the stream itself so that it becomes self-
describing. In addition to that, this also brings the `struct
odb_write_stream` a bit closer to the `struct odb_read_stream` so that
we can eventually merge both stream types.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Blame and "git log --stat" recover hunk coordinates by diffing blob
pairs, and recompute them on every run. Add a cache of those
coordinates at $GIT_DIR/objects/info/diff-hunks, beside the
commit-graph, so a later run can look them up instead of decompressing
the blobs and running xdiff again.
The store is a single chunk-format file (see gitformat-chunk(5)): an
8-byte header, a DHIX index of fixed-size entries sorted by key, a DHDT
segment of hunk records, and a trailing hash checksum. An entry is
keyed by the two blob object ids and the xdl_opts the pair was diffed
under, so a stored result is served only where that exact key recurs,
independent of path. A zero-context diff trims unchanged lines from
hunk edges and can pick a different but equally valid set of hunks than
an untrimmed diff, so a recording caller stores a pair only when its
trimmed and untrimmed diffs are identical; such an entry answers any
consumer at any context, and the rare divergent pair is always
computed. Identical hunk blocks are interned once and shared across
keys.
The library provides a reader (repo_diff_hunks_store and _replay, gated
by core.diffHunks), loaded once and cached on the object database as the
commit-graph is, and a writer that accumulates entries and flushes them
in one atomic pass. An absent, corrupt, or disabled store reads as all
misses. A record with no hunks is invalid too: replaying it would claim
the pair equivalent, which the store never asserts, so it reads as a
miss.
Ordinary reads are diagnostic-free. Loading parses the chunk table
through read_table_of_contents_quiet(), new in chunk-format, which
prints nothing on a malformed table and takes the repository's hash
algorithm rather than the_hash_algo, so the file is bounds-checked under
the algorithm it is keyed by.
The flush closes the repository's mmapped store and forgets that loading
was attempted before committing the lockfile. A warming run that also
reads may hold the file it is replacing mapped, and the rename must not
land on a live mapping, which Windows refuses; a read after the flush
then observes the committed file. commit-graph closes its graph before
committing for the same reason.
Writing is off by default, enabled per run by GIT_DIFF_HUNKS_WRITE or
persistently by diffHunks.write, the environment winning. A writer
seeds from the existing store, so a flush merges rather than replaces.
The seed's checksum is verified first: a corrupt store is discarded, not
rewritten with a fresh checksum verify could no longer catch. An entry
that fails the shared diff_provider_check_hunk() or names no blob is
dropped with a warning, since it would only ever read as a miss. A seed
that discarded or dropped anything forces the flush even when the
warming run computed nothing new. The writer fsyncs through a new
diff-hunks core.fsync component.
"git diff-hunks" inspects and manages the file: "verify" checks the
checksum, chunk table, sort order, entry bounds, and every entry's hunk
sequence against that shared check, so a store whose entries could only
read as misses fails verify; "clear" removes the file. Later patches
wire the readers and the writer into the diff and blame paths.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The function `force_object_loose()` is used to loosen packed objects
before repacking. It passes the pack's mtime along so that the newly
written loose object inherits the same timestamp. This matters for
object pruning, which uses the mtime to determine whether an object is
old enough to be pruned.
In a subsequent commit, `force_object_loose()` will be converted to use
the generic `odb_source_write_object()` interface instead of calling
`write_loose_object()` directly. But the generic interface doesn't yet
support setting a specific mtime, which makes it impossible to implement
the logic as of now.
Prepare for the change by introducing a new `mtime` parameter to this
function that we plumb through the stack. If set, the backends are
instructed to set the object's mtime accordingly. If unset, the backends
are expected to use the current time instead.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Before writing a new loose object we first check whether the object
already exists in any of the sources attached to the object database.
This results in a couple of issues:
- We have a layering violation, where the source needs to be aware of
objects stored in any of the other sources.
- Every backend would have to reimplement this check, which feels
somewhat pointless.
- It is not possible to easily write an object into a source in case
the same object already exists in another source.
Refactor the code and lift up the object existence check from the
"loose" backend into the generic ODB layer. No callers need adjustment
as none of them write via a specific source, but via the ODB layer.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Same as in a preceding commit, compute the object hash in
`odb_write_object_ext()` so that we can unify this logic.
Besides unification, this change also allows us to lift the object
existence check out of the "loose" backend into the generic layer, which
will happen in the next commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Repositories can have a compatibility hash configured, which means that
such a repository is expected to maintain a mapping between canonical
and compatibility object hashes. Maintaining this mapping is the
responsibility of the object database sources, where we either store
them as part of the loose objects map or in packfile indices v3 (once we
gain support for this feature).
But besides storing these compatibility hashes, the sources are also
responsible for generating the compatibility hash in the first place.
This is somewhat unnecessary though, as the compatibility hash should be
computed the same no matter which source is being used. The consequence
is that we need to duplicate this functionality across the different
backends, which does not make a lot of sense.
Refactor the code so that we instead compute the compatibility hash in
`odb_write_object_ext()` and then pass the computed value to the
sources. No callers need adjustment as there are none that write objects
via the source interfaces directly.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The 'whence' field in 'struct object_info' has been removed. The
backend-specific object information retrieval has been refactored into
an opt-in 'struct object_info_source' structure.
* ps/odb-drop-whence:
odb: document object info fields
odb: drop `whence` field from object info
treewide: convert users of `whence` to the new source field
odb: add `source` field to struct object_info_source
odb: make backend-specific fields optional
packfile: thread odb_source_packed through packed_object_info()
The 'reprepare()' callback for object database sources has been
generalized into a 'prepare()' callback with an optional flush cache
flag, and a new 'odb_prepare()' wrapper has been introduced to allow
pre-opening object database sources.
* ps/odb-generalize-prepare:
odb: introduce `odb_prepare()`
odb/source: generalize `reprepare()` callback
Move `odb_optimize()` and `odb_optimize_required()` from "builtin/gc.c"
into the "files" source and wire them up via newly introduced vtable
pointers for the object database sources. This makes the logic pluggable
and thus allows other backends to have their own, custom implementation.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In the preceding commits we have migrated all callers to derive their
information of how a specific object is stored to use the new object
info source instead, and hence the field is now unused. Drop it.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The previous commit introduced `struct object_info_source` as an opt-in
container for backend-specific information, but for now we only moved
preexisting data into this structure. Most importantly, the caller has
no way yet to learn about which source an object was actually looked up
from. Instead, callers have to rely on the `whence` enum to distinguish
the object type, but cannot use that enum to tell the object source.
Add a `struct odb_source *source` field to the structure and populate it
from each backend's lookup path.
The `whence` enum is still set and used by callers; it will be removed
in a subsequent commit now that `sourcep->source` can identify the
backend on its own.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `struct object_info` carries two pieces of information
about how an object was looked up:
- The `whence` enum identifying the backend.
- The backend-tagged union `u` exposing backend-specific details
(currently only the packed-source case, which records the owning
pack, offset and packed object type).
The union is populated unconditionally, even though most callers don't
care about provenance at all.
Split the backend-specific union out into a new public type, `struct
object_info_source`, and make the object info structure carry it via
just another opt-in request pointer. As with all the other requestable
information, callers that need source info allocate a `struct
object_info_source` on the stack and point `sourcep` at it; callers that
don't care about it simply leave the field as a `NULL` pointer. Adapt
callers accordingly.
Note that the `whence` enum is strictly-speaking also backend-specific
information, so it would be another good candidate to be moved into the
`struct object_info_source`. For now though it is left alone, as it will
be replaced by a `struct odb_source` pointer in a subsequent commit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce `odb_prepare()` as a simple wrapper to prepare alternates and
then prepare each individual source. Adapt git-grep(1) to use it.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `reprepare()` callback function can be used to flush caches of a
given object source and then prepare it anew. This is for example used
when a concurrent process may have written new objects. Ultimately, this
can be seen as doing two separate steps:
1. We drop any caches.
2. We prepare the source.
We have one callsite in git-grep(1) though that really only want to do
(2). This is done by reaching into the "files" backend directly and then
calling `odb_source_packed_prepare()`, which of course may not work with
alternate backends.
We could in theory just call `reprepare()` here, and that would likely
not have any significant downside. But this would certainly feel like a
code smell.
Instead, generalize the `reprepare()` callback to `prepare()` with a
flag that optionally instructs the backend to also flush the caches,
which allows us to drop the external `odb_source_packed_prepare()`
declaration.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When `js/objects-larger-than-4gb-on-windows` widened the streaming,
index-pack and unpack-objects code paths, in the interest of keeping the
patches somewhat reasonably-sized, it left the public ODB API still
typed in `unsigned long`. In particular `struct object_info::sizep` and
the four wrappers built on top of it (`odb_read_object`,
`odb_read_object_peeled`, `odb_read_object_info`, `odb_pretend_object`)
still return the unpacked size through `unsigned long *`, so on Windows
`cat-file -s` and the `git add` / `git status` paths for a >4 GiB blob
silently cap at 4 GiB.
Widen the field and the four wrappers. The previous commits already
widened the `unpack_entry()` cascade and pack-objects' in-core size
accessors, so most of the cascade arrives here with no further work: the
temporary shims in `packed_object_info_with_index_pos()` and in
`unpack_entry()`'s delta-base recovery path go away, the two
`SET_SIZE(entry, cast_size_t_to_ulong(canonical_size))` calls in
`check_object()` and the matching one in `drop_reused_delta()` collapse
to plain `SET_SIZE`, and `oe_get_size_slow()`'s tail
`cast_size_t_to_ulong()` is gone too.
What remains narrow are the boundaries this series does not
intend to touch: the diff, blame, textconv and fast-import machinery.
Even so, this patch is unfortunately quite large.
Assisted-by: Opus 4.7
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Implement the `write_object()` callback function for the in-memory
source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Implement the `read_object_info()` callback function for the in-memory
source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The function `odb_pretend_object()` writes an object into the in-memory
object database source. The effect of this is that the object will now
become readable, but it won't ever be persisted to disk.
Before storing the object, we first verify whether the object already
exists. This is done by calling `odb_has_object()` to check all sources,
followed by `find_cached_object()` to check whether we have already
stored the object in our in-memory source.
This is unnecessary though, as `odb_has_object()` already checks the
in-memory source transitively via:
- `odb_has_object()`
- `odb_read_object_info_extended()`
- `do_oid_object_info_extended()`
- `find_cached_object()`
Drop the explicit call to `find_cached_object()`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Implement the `free()` callback function for the "in-memory" source.
Note that this requires us to define `struct cached_object_entry` in
"odb/source-inmemory.h", as it is accessed in both "odb.c" and
"odb/source-inmemory.c" now. This will be fixed in subsequent commits
though.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Next to our typical object database sources, each object database also
has an implicit source of "cached" objects. These cached objects only
exist in memory and some use cases:
- They contain evergreen objects that we expect to always exist, like
for example the empty tree.
- They can be used to store temporary objects that we don't want to
persist to disk, which is used by git-blame(1) to create a fake
worktree commit.
Overall, their use is somewhat restricted though. For example, we don't
provide the ability to use it as a temporary object database source that
allows the user to write objects, but discard them after Git exists. So
while these cached objects behave almost like a source, they aren't used
as one.
This is about to change over the following commits, where we will turn
cached objects into a new "in-memory" source. This will allow us to use
it exactly the same as any other source by providing the same common
interface as the "files" source.
For now, the in-memory source only hosts the cached objects and doesn't
provide any logic yet. This will change with subsequent commits, where
we move respective functionality into the source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The current ODB transaction interface is colocated with other ODB
interfaces in "odb.{c,h}". Subsequent commits will expand `struct
odb_transaction` to support write operations on the transaction
directly. To keep things organized and prevent "odb.{c,h}" from becoming
more unwieldy, split out `struct odb_transaction` into a separate
header.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Rename `odb_has_object()` flags to be properly prefixed with the
function name.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We've got a couple of functions that accept `odb_write_object()` flags,
but all of them accept the flags as an `unsigned` integer. In fact, we
don't even have an `enum` for the flags field.
Introduce this `enum` and adapt functions accordingly according to our
coding style.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We've got a couple of callsites where we pass `odb_for_each_object()`
flags, but accept an `unsigned` flags field instead of the corresponding
enum. Adapt these to accept the enum type instead.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new generic `odb_find_abbrev_len()` function as well as
source-specific callback functions. This makes the logic to compute the
required prefix length to make a given object unique fully pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `odb_for_each_object()` function only accepts a bitset of flags. In
a subsequent commit we'll want to change object iteration to also
support iterating over only those objects that have a specific prefix.
While we could of course add the prefix to the function signature, or
alternatively introduce a new function, both of these options don't
really seem to be that sensible.
Instead, introduce a new `struct odb_for_each_object_options` that can
be passed to a new `odb_for_each_object_ext()` function. Splice through
the options structure into the respective object database sources.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Similar to the preceding commit, introduce counting of objects on the
object database level, replacing the logic that we have in
`repo_approximate_object_count()`.
Note that the function knows to cache the object count. It's unclear
whether this cache is really required as we shouldn't have that many
cases where we count objects repeatedly. But to be on the safe side the
caching mechanism is retained, with the only excepting being that we
also have to use the passed flags as caching key.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Note that this function is a bit less straight-forward to convert
compared to the other functions. The reason here is that the logic to
read an object is:
1. We try to read the object. If it exists we return it.
2. If the object does not exist we reprepare the object database
source.
3. We then try reading the object info a second time in case the
reprepare caused it to appear.
The second read is only supposed to happen for the packfile store
though, as reading loose objects is not impacted by repreparing the
object database.
Ideally, we'd just move this whole logic into the ODB source. But that's
not easily possible because we try to avoid the reprepare unless really
required, which is after we have found out that no other ODB source
contains the object, either. So the logic spans across multiple ODB
sources, and consequently we cannot move it into an individual source.
Instead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the
backend that we already tried to look up the object once, and that this
time around the ODB source should try to find any new objects that may
have surfaced due to an on-disk change.
With this flag, the "files" backend can trivially skip trying to re-read
the object as a loose object. Furthermore, as we know that we only try
the second read via the packfile store, we can skip repreparing loose
objects and only reprepare the packfile store.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new callback function in `struct odb_source` to make the
function pluggable.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The primary object database source may be initialized with a relative
path. When the process changes its current working directory we thus
have to update this path and have it point to the same path, but
relative to the new working directory.
This logic is handled in the object database layer. It consists of three
steps:
1. We undo any potential temporary object directory, which are used
for transactions. This is done so that we don't end up modifying
the temporary object database source that got applied for the
transaction.
2. We then iterate through the non-transactional sources and reparent
their respective paths.
3. We reapply the temporary object directory, but update its path.
All of this logic is heavily tied to how the object database source
handles paths in the first place. It's an internal implementation
detail, and as sources may not even use an on-disk path at all it is not
a mechanism that applies to all potential sources.
Refactor the code so that the logic to reparent the sources is hosted by
the "files" source and the temporary object directory subsystems,
respectively. This logic is easier to reason about, but it also ensures
that this logic is handled at the correct level.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The "files" backend is implemented as a pointer in the `struct
odb_source`. This contradicts our typical pattern for pluggable backends
like we use it for example in the ref store or for object database
streams, where we typically embed the generic base structure in the
specialized implementation. This pattern has a couple of small benefits:
- We avoid an extra allocation.
- We hide implementation details in the generic structure.
- We can easily downcast from a generic backend to the specialized
structure and vice versa because the offsets are known at compile
time.
- It becomes trivial to identify locations where we depend on backend
specific logic because the cast needs to be explicit.
Refactor our "files" object database source to do the same and embed the
`struct odb_source` in the `struct odb_source_files`.
There are still a bunch of sites in our code base where we do have to
access internals of the "files" backend. The intent is that those will
go away over time, but this will certainly take a while. Meanwhile,
provide a `odb_source_files_downcast()` function that can convert a
generic source into a "files" source.
As we only have a single source the downcast succeeds unconditionally
for now. Eventually though the intent is to make the cast `BUG()` in
case the caller requests to downcast a non-"files" backend to a "files"
backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce a new "files" object database source. This source encapsulates
access to both loose object files and the packfile store, similar to how
the "files" backend for refs encapsulates access to loose refs and the
packed-refs file.
Note that for now the "files" source is still a direct member of a
`struct odb_source`. This architecture will be reversed in the next
commit so that the files source contains a `struct odb_source`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Subsequent commits will expand the `struct odb_source` to become a
generic interface for accessing an object database source. As part of
these refactorings we'll add a set of function pointers that will
significantly expand the structure overall.
Prepare for this by splitting out the `struct odb_source` into a
separate header. This keeps the high-level object database interface
detached from the low-level object database sources.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
* ps/object-info-bits-cleanup:
odb: convert `odb_has_object()` flags into an enum
odb: convert object info flags into an enum
odb: drop gaps in object info flag values
builtin/fsck: fix flags passed to `odb_has_object()`
builtin/backfill: fix flags passed to `odb_has_object()`
* ps/odb-for-each-object:
odb: drop unused `for_each_{loose,packed}_object()` functions
reachable: convert to use `odb_for_each_object()`
builtin/pack-objects: use `packfile_store_for_each_object()`
odb: introduce mtime fields for object info requests
treewide: drop uses of `for_each_{loose,packed}_object()`
treewide: enumerate promisor objects via `odb_for_each_object()`
builtin/fsck: refactor to use `odb_for_each_object()`
odb: introduce `odb_for_each_object()`
packfile: introduce function to iterate through objects
packfile: extract function to iterate through objects of a store
object-file: introduce function to iterate through objects
object-file: extract function to read object info from path
odb: fix flags parameter to be unsigned
odb: rename `FOR_EACH_OBJECT_*` flags
