Files
git/repository.c
T
Patrick Steinhardt cbfd4cb86f odb/source-files: move alternates into the backend
Originally, when designing pluggable object databases the goal was that
the object database can have multiple sources, and every source attached
to it could use a different backend. This would have allowed for quite a
lot of flexibility, as you could trivially mix and match different kinds
of object storages in whatever way you like.

But while well-intentioned, this design led to a bunch of conceptual
problems:

  - We're now trying to read objects in source order, whereas we
    previously tried to read objects via packfiles before trying to read
    them via loose objects. This led to a performance regression when
    using alternates or when using a quarantine directory.

  - Some data structures are supposed to only ever exist once, like for
    example bitmaps and commit graphs. At the same time, those data
    structures also span across the union of all objects, so they may
    cross sources.

  - It is unclear how we can extend GIT_OBJECT_DIRECTORY or
    GIT_ALTERNATE_OBJECT_DIRECTORIES to become backend-agnostic in a
    backwards-compatible way. In general, introducing an object storage
    extension into the current status quo where alternates may have to
    be extended to become generic was proving to be painful.

  - Some mechanisms of alternates assume way too much about how exactly
    their backends work. Alternate refs for example assume that the
    alternate is backed by a filesystem path, and that this filesystem
    path may also allow us to read references. This is not a given
    though, as backends may not even have local data at all.

In short, there are a bunch of conceptual mismatches when we have
alternates and pluggable object databases coexist. So while the original
idea was nice, it does not result in a system that is easy to reason
about.

Correct course by moving alternates into the "files" source itself so
that it becomes an implementation detail thereof so that we can avoid
all of these shortcomings. While it's unfortunate that we cannot easily
mix and match sources now, that ability doesn't go away. It's still very
much feasible to introduce a new backend that allows for exactly that
use case, and such a backend may also be a lot more flexible as we can
now add new logic to determine which objects should be stored where. So
the original motivation for having per-source backends can still be
realized with the new architecture.

Note that as part of this move, we also handle the GIT_OBJECT_DIRECTORY
and GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables in the
"files" backend. This may be surprising at first, but object directories
are very much a concept of that backend, too. So these variables would
have bad interactions with other backends, and they create a bit of a
mismatch with the eventual object storage extension that we plan to
introduce.

Note that this commit is way larger than I'd like it to be. I'm sorry, I
couldn't find a way to split it up further. That being said, most of the
changes are straight-forward conversions that go from iterating over
sources to iterating over object directories. The more involved changes
are moving the infrastructure to track individual sources from "odb.c"
and moving them into "odb/source-files.c".

This also serves to show a bit of a who's-who of commands that don't
work properly with pluggable object databases. Almost all of these
commands are related to housekeeping though -- some of them will be
converted eventually, like for example commit graphs or MIDXs. But many
of them will stay incompatible going forward as they are simply too
specific to the "files" backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2026-10-08 07:31:19 -07:00

480 lines
12 KiB
C

#include "git-compat-util.h"
#include "abspath.h"
#include "repository.h"
#include "hook.h"
#include "odb.h"
#include "odb/source.h"
#include "config.h"
#include "gettext.h"
#include "object.h"
#include "lockfile.h"
#include "path.h"
#include "read-cache-ll.h"
#include "remote.h"
#include "setup.h"
#include "loose.h"
#include "submodule-config.h"
#include "sparse-index.h"
#include "trace2.h"
#include "promisor-remote.h"
#include "refs.h"
/*
* We do not define `USE_THE_REPOSITORY_VARIABLE` in this file because we do
* not want to rely on functions that implicitly use `the_repository`. This
* means that the `extern` declaration of `the_repository` isn't visible here,
* which makes sparse unhappy. We thus declare it here.
*/
extern struct repository *the_repository;
/* The main repository */
static struct repository the_repo;
struct repository *the_repository = &the_repo;
/*
* An escape hatch: if we hit a bug in the production code that fails
* to set an appropriate hash algorithm (most likely to happen when
* running outside a repository), we can tell the user who reported
* the crash to set the environment variable to "sha1" (all lowercase)
* to revert to the historical behaviour of defaulting to SHA-1.
*/
static void set_default_hash_algo(struct repository *repo)
{
const char *hash_name;
uint32_t algo;
hash_name = getenv("GIT_TEST_DEFAULT_HASH_ALGO");
if (!hash_name)
return;
algo = hash_algo_by_name(hash_name);
if (algo == GIT_HASH_UNKNOWN)
return;
repo_set_hash_algo(repo, algo);
}
struct repo_config_values *repo_config_values(struct repository *repo)
{
if (repo != the_repository)
BUG("trying to read config from wrong repository instance");
if (!repo->initialized)
BUG("config values from uninitialized repository");
return &repo->config_values_private_;
}
void initialize_repository(struct repository *repo)
{
if (repo->initialized)
BUG("repository initialized already");
repo->initialized = true;
repo->remote_state = remote_state_new();
repo->parsed_objects = parsed_object_pool_new(repo);
ALLOC_ARRAY(repo->index, 1);
index_state_init(repo->index, repo);
repo->check_deprecated_config = true;
repo->bare_cfg = -1;
repo->fetch_if_missing = 1;
repo_config_values_init(&repo->config_values_private_);
/*
* When a command runs inside a repository, it learns what
* hash algorithm is in use from the repository, but some
* commands are designed to work outside a repository, yet
* they want to access the_hash_algo, if only for the length
* of the hashed value to see if their input looks like a
* plausible hash value.
*
* We are in the process of identifying such code paths and
* giving them an appropriate default individually; any
* unconverted code paths that try to access the_hash_algo
* will thus fail. The end-users however have an escape hatch
* to set GIT_TEST_DEFAULT_HASH_ALGO environment variable to
* "sha1" to get back the old behaviour of defaulting to SHA-1.
*
* This escape hatch is deliberately kept unadvertised, so
* that they see crashes and we can get a report before
* telling them about it.
*/
if (repo == the_repository)
set_default_hash_algo(repo);
}
static void expand_base_dir(char **out, const char *in,
const char *base_dir, const char *def_in)
{
free(*out);
if (in)
*out = xstrdup(in);
else
*out = xstrfmt("%s/%s", base_dir, def_in);
}
const char *repo_get_git_dir(struct repository *repo)
{
if (!repo->gitdir)
BUG("repository hasn't been set up");
return repo->gitdir;
}
const char *repo_get_common_dir(struct repository *repo)
{
if (!repo->commondir)
BUG("repository hasn't been set up");
return repo->commondir;
}
const char *repo_get_object_directory(struct repository *repo)
{
if (!repo->objects->source)
BUG("repository hasn't been set up");
return repo->objects->source->path;
}
const char *repo_get_index_file(struct repository *repo)
{
if (!repo->index_file)
BUG("repository hasn't been set up");
return repo->index_file;
}
const char *repo_get_graft_file(struct repository *repo)
{
if (!repo->graft_file)
BUG("repository hasn't been set up");
return repo->graft_file;
}
const char *repo_get_work_tree(struct repository *repo)
{
return repo->worktree;
}
static void repo_set_commondir(struct repository *repo,
const char *commondir)
{
struct strbuf sb = STRBUF_INIT;
free(repo->commondir);
if (commondir) {
repo->different_commondir = 1;
repo->commondir = xstrdup(commondir);
return;
}
repo->different_commondir = get_common_dir_noenv(&sb, repo->gitdir);
repo->commondir = strbuf_detach(&sb, NULL);
}
void repo_set_gitdir(struct repository *repo,
const char *root,
const struct set_gitdir_args *o)
{
const char *gitfile = read_gitfile(root);
/*
* repo->gitdir is saved because the caller could pass "root"
* that also points to repo->gitdir. We want to keep it alive
* until after xstrdup(root). Then we can free it.
*/
char *old_gitdir = repo->gitdir;
repo->gitdir = xstrdup(gitfile ? gitfile : root);
free(old_gitdir);
repo_set_commondir(repo, o->commondir);
repo->disable_ref_updates = o->disable_ref_updates;
expand_base_dir(&repo->graft_file, o->graft_file,
repo->commondir, "info/grafts");
expand_base_dir(&repo->index_file, o->index_file,
repo->gitdir, "index");
}
void repo_set_hash_algo(struct repository *repo, uint32_t hash_algo)
{
repo->hash_algo = &hash_algos[hash_algo];
}
void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t algo)
{
#ifdef WITH_RUST
if (hash_algo_by_ptr(repo->hash_algo) == algo)
BUG("hash_algo and compat_hash_algo match");
repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
#else
if (algo)
die(_("compatibility hash algorithm support requires Rust"));
#endif
}
void repo_set_ref_storage_format(struct repository *repo,
enum ref_storage_format format,
const char *payload)
{
repo->ref_storage_format = format;
free(repo->ref_storage_payload);
repo->ref_storage_payload = xstrdup_or_null(payload);
}
/*
* Attempt to resolve and set the provided 'gitdir' for repository 'repo'.
* Return 0 upon success and a non-zero value upon failure.
*/
static int repo_init_gitdir(struct repository *repo, const char *gitdir)
{
int ret = 0;
int error = 0;
char *abspath = NULL;
const char *resolved_gitdir;
struct set_gitdir_args args = { NULL };
abspath = real_pathdup(gitdir, 0);
if (!abspath) {
ret = -1;
goto out;
}
/* 'gitdir' must reference the gitdir directly */
resolved_gitdir = resolve_gitdir_gently(abspath, &error);
if (!resolved_gitdir) {
ret = -1;
goto out;
}
repo_set_gitdir(repo, resolved_gitdir, &args);
out:
free(abspath);
return ret;
}
void repo_set_worktree(struct repository *repo, const char *path)
{
repo->worktree = real_pathdup(path, 1);
trace2_def_repo(repo);
}
static int read_repository_format_from_commondir(struct repository_format *format,
const char *commondir)
{
int ret = 0;
struct strbuf sb = STRBUF_INIT;
strbuf_addf(&sb, "%s/config", commondir);
read_repository_format(format, sb.buf);
strbuf_reset(&sb);
strbuf_release(&sb);
return ret;
}
/*
* Initialize 'repo' based on the provided 'gitdir'.
* Return 0 upon success and a non-zero value upon failure.
*/
int repo_init(struct repository *repo,
const char *gitdir,
const char *worktree)
{
struct repository_format format = REPOSITORY_FORMAT_INIT;
struct strbuf err = STRBUF_INIT;
memset(repo, 0, sizeof(*repo));
initialize_repository(repo);
if (repo_init_gitdir(repo, gitdir))
goto error;
if (read_repository_format_from_commondir(&format, repo->commondir))
goto error;
if (apply_repository_format(repo, &format, 0, &err) < 0) {
warning("%s", err.buf);
goto error;
}
repo->objects = odb_new(repo, 0);
if (worktree)
repo_set_worktree(repo, worktree);
clear_repository_format(&format);
strbuf_release(&err);
return 0;
error:
clear_repository_format(&format);
strbuf_release(&err);
repo_clear(repo);
return -1;
}
int repo_submodule_init(struct repository *subrepo,
struct repository *superproject,
const char *path,
const struct object_id *treeish_name)
{
struct strbuf gitdir = STRBUF_INIT;
struct strbuf worktree = STRBUF_INIT;
int ret = 0;
repo_worktree_path_append(superproject, &gitdir, "%s/.git", path);
repo_worktree_path_append(superproject, &worktree, "%s", path);
if (repo_init(subrepo, gitdir.buf, worktree.buf)) {
/*
* If initialization fails then it may be due to the submodule
* not being populated in the superproject's worktree. Instead
* we can try to initialize the submodule by finding it's gitdir
* in the superproject's 'modules' directory. In this case the
* submodule would not have a worktree.
*/
const struct submodule *sub =
submodule_from_path(superproject, treeish_name, path);
if (!sub) {
ret = -1;
goto out;
}
strbuf_reset(&gitdir);
submodule_name_to_gitdir(&gitdir, superproject, sub->name);
if (repo_init(subrepo, gitdir.buf, NULL)) {
ret = -1;
goto out;
}
}
subrepo->submodule_prefix = xstrfmt("%s%s/",
superproject->submodule_prefix ?
superproject->submodule_prefix :
"", path);
out:
strbuf_release(&gitdir);
strbuf_release(&worktree);
return ret;
}
static void repo_clear_path_cache(struct repo_path_cache *cache)
{
FREE_AND_NULL(cache->squash_msg);
FREE_AND_NULL(cache->merge_msg);
FREE_AND_NULL(cache->merge_rr);
FREE_AND_NULL(cache->merge_mode);
FREE_AND_NULL(cache->merge_head);
FREE_AND_NULL(cache->fetch_head);
FREE_AND_NULL(cache->shallow);
}
void repo_clear(struct repository *repo)
{
struct hashmap_iter iter;
struct strmap_entry *e;
FREE_AND_NULL(repo->gitdir);
FREE_AND_NULL(repo->commondir);
FREE_AND_NULL(repo->prefix);
FREE_AND_NULL(repo->graft_file);
FREE_AND_NULL(repo->index_file);
FREE_AND_NULL(repo->worktree);
FREE_AND_NULL(repo->submodule_prefix);
FREE_AND_NULL(repo->ref_storage_payload);
odb_free(repo->objects);
repo->objects = NULL;
if (repo->parsed_objects)
parsed_object_pool_clear(repo->parsed_objects);
FREE_AND_NULL(repo->parsed_objects);
repo_settings_clear(repo);
repo_config_values_clear(&repo->config_values_private_);
if (repo->config) {
git_configset_clear(repo->config);
FREE_AND_NULL(repo->config);
}
if (repo->submodule_cache) {
submodule_cache_free(repo->submodule_cache);
repo->submodule_cache = NULL;
}
if (repo->index) {
discard_index(repo->index);
FREE_AND_NULL(repo->index);
}
if (repo->hook_config_cache) {
hook_cache_clear(repo->hook_config_cache);
FREE_AND_NULL(repo->hook_config_cache);
}
strmap_clear(&repo->event_jobs, 0); /* values are uintptr_t, not heap ptrs */
string_list_clear(&repo->disabled_events, 0);
if (repo->promisor_remote_config) {
promisor_remote_clear(repo->promisor_remote_config);
FREE_AND_NULL(repo->promisor_remote_config);
}
if (repo->remote_state) {
remote_state_clear(repo->remote_state);
FREE_AND_NULL(repo->remote_state);
}
if (repo->refs_private) {
ref_store_release(repo->refs_private);
FREE_AND_NULL(repo->refs_private);
}
strmap_for_each_entry(&repo->submodule_ref_stores, &iter, e)
ref_store_release(e->value);
strmap_clear(&repo->submodule_ref_stores, 1);
strmap_for_each_entry(&repo->worktree_ref_stores, &iter, e)
ref_store_release(e->value);
strmap_clear(&repo->worktree_ref_stores, 1);
repo_clear_path_cache(&repo->cached_paths);
}
int repo_read_index(struct repository *repo)
{
int res;
/* Complete the double-reference */
if (!repo->index) {
ALLOC_ARRAY(repo->index, 1);
index_state_init(repo->index, repo);
} else if (repo->index->repo != repo) {
BUG("repo's index should point back at itself");
}
res = read_index_from(repo->index, repo->index_file, repo->gitdir);
prepare_repo_settings(repo);
if (repo->settings.command_requires_full_index)
ensure_full_index(repo->index);
/*
* If sparse checkouts are in use, check whether paths with the
* SKIP_WORKTREE attribute are missing from the worktree; if not,
* clear that attribute for that path.
*/
clear_skip_worktree_from_present_files(repo->index);
return res;
}
int repo_hold_locked_index(struct repository *repo,
struct lock_file *lf,
int flags)
{
if (!repo->index_file)
BUG("the repo hasn't been setup");
return repo_hold_lock_file_for_update(repo, lf, repo->index_file, flags);
}