Patrick Steinhardt
cbfd4cb86f
Originally, when designing pluggable object databases the goal was that
the object database can have multiple sources, and every source attached
to it could use a different backend. This would have allowed for quite a
lot of flexibility, as you could trivially mix and match different kinds
of object storages in whatever way you like.
But while well-intentioned, this design led to a bunch of conceptual
problems:
- We're now trying to read objects in source order, whereas we
previously tried to read objects via packfiles before trying to read
them via loose objects. This led to a performance regression when
using alternates or when using a quarantine directory.
- Some data structures are supposed to only ever exist once, like for
example bitmaps and commit graphs. At the same time, those data
structures also span across the union of all objects, so they may
cross sources.
- It is unclear how we can extend GIT_OBJECT_DIRECTORY or
GIT_ALTERNATE_OBJECT_DIRECTORIES to become backend-agnostic in a
backwards-compatible way. In general, introducing an object storage
extension into the current status quo where alternates may have to
be extended to become generic was proving to be painful.
- Some mechanisms of alternates assume way too much about how exactly
their backends work. Alternate refs for example assume that the
alternate is backed by a filesystem path, and that this filesystem
path may also allow us to read references. This is not a given
though, as backends may not even have local data at all.
In short, there are a bunch of conceptual mismatches when we have
alternates and pluggable object databases coexist. So while the original
idea was nice, it does not result in a system that is easy to reason
about.
Correct course by moving alternates into the "files" source itself so
that it becomes an implementation detail thereof so that we can avoid
all of these shortcomings. While it's unfortunate that we cannot easily
mix and match sources now, that ability doesn't go away. It's still very
much feasible to introduce a new backend that allows for exactly that
use case, and such a backend may also be a lot more flexible as we can
now add new logic to determine which objects should be stored where. So
the original motivation for having per-source backends can still be
realized with the new architecture.
Note that as part of this move, we also handle the GIT_OBJECT_DIRECTORY
and GIT_ALTERNATE_OBJECT_DIRECTORIES environment variables in the
"files" backend. This may be surprising at first, but object directories
are very much a concept of that backend, too. So these variables would
have bad interactions with other backends, and they create a bit of a
mismatch with the eventual object storage extension that we plan to
introduce.
Note that this commit is way larger than I'd like it to be. I'm sorry, I
couldn't find a way to split it up further. That being said, most of the
changes are straight-forward conversions that go from iterating over
sources to iterating over object directories. The more involved changes
are moving the infrastructure to track individual sources from "odb.c"
and moving them into "odb/source-files.c".
This also serves to show a bit of a who's-who of commands that don't
work properly with pluggable object databases. Almost all of these
commands are related to housekeeping though -- some of them will be
converted eventually, like for example commit graphs or MIDXs. But many
of them will stay incompatible going forward as they are simply too
specific to the "files" backend.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
104 lines
2.5 KiB
C
104 lines
2.5 KiB
C
#define USE_THE_REPOSITORY_VARIABLE
|
|
|
|
#include "test-tool.h"
|
|
#include "commit-graph.h"
|
|
#include "repository.h"
|
|
#include "odb.h"
|
|
#include "odb/source.h"
|
|
#include "bloom.h"
|
|
#include "setup.h"
|
|
|
|
static void dump_graph_info(struct commit_graph *graph)
|
|
{
|
|
printf("header: %08x %d %d %d %d\n",
|
|
ntohl(*(uint32_t*)graph->data),
|
|
*(unsigned char*)(graph->data + 4),
|
|
*(unsigned char*)(graph->data + 5),
|
|
*(unsigned char*)(graph->data + 6),
|
|
*(unsigned char*)(graph->data + 7));
|
|
printf("num_commits: %u\n", graph->num_commits);
|
|
printf("chunks:");
|
|
|
|
if (graph->chunk_oid_fanout)
|
|
printf(" oid_fanout");
|
|
if (graph->chunk_oid_lookup)
|
|
printf(" oid_lookup");
|
|
if (graph->chunk_commit_data)
|
|
printf(" commit_metadata");
|
|
if (graph->chunk_generation_data)
|
|
printf(" generation_data");
|
|
if (graph->chunk_generation_data_overflow)
|
|
printf(" generation_data_overflow");
|
|
if (graph->chunk_extra_edges)
|
|
printf(" extra_edges");
|
|
if (graph->chunk_bloom_indexes)
|
|
printf(" bloom_indexes");
|
|
if (graph->chunk_bloom_data)
|
|
printf(" bloom_data");
|
|
printf("\n");
|
|
|
|
printf("options:");
|
|
if (graph->bloom_filter_settings)
|
|
printf(" bloom(%"PRIu32",%"PRIu32",%"PRIu32")",
|
|
graph->bloom_filter_settings->hash_version,
|
|
graph->bloom_filter_settings->bits_per_entry,
|
|
graph->bloom_filter_settings->num_hashes);
|
|
if (graph->read_generation_data)
|
|
printf(" read_generation_data");
|
|
if (graph->topo_levels)
|
|
printf(" topo_levels");
|
|
printf("\n");
|
|
}
|
|
|
|
static void dump_graph_bloom_filters(struct commit_graph *graph)
|
|
{
|
|
uint32_t i;
|
|
|
|
for (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {
|
|
struct bloom_filter filter = { 0 };
|
|
size_t j;
|
|
|
|
if (load_bloom_filter_from_graph(graph, &filter, i) < 0) {
|
|
fprintf(stderr, "missing Bloom filter for graph "
|
|
"position %"PRIu32"\n", i);
|
|
continue;
|
|
}
|
|
|
|
for (j = 0; j < filter.len; j++)
|
|
printf("%02x", filter.data[j]);
|
|
if (filter.len)
|
|
printf("\n");
|
|
}
|
|
}
|
|
|
|
int cmd__read_graph(int argc, const char **argv)
|
|
{
|
|
struct commit_graph *graph = NULL;
|
|
struct odb_source *source;
|
|
int ret = 0;
|
|
|
|
setup_git_directory(the_repository);
|
|
source = the_repository->objects->source;
|
|
|
|
prepare_repo_settings(the_repository);
|
|
|
|
graph = read_commit_graph_one(the_repository, source->path);
|
|
if (!graph) {
|
|
ret = 1;
|
|
goto done;
|
|
}
|
|
|
|
if (argc <= 1)
|
|
dump_graph_info(graph);
|
|
else if (!strcmp(argv[1], "bloom-filters"))
|
|
dump_graph_bloom_filters(graph);
|
|
else {
|
|
fprintf(stderr, "unknown sub-command: '%s'\n", argv[1]);
|
|
ret = 1;
|
|
}
|
|
|
|
done:
|
|
free_commit_graph(graph);
|
|
return ret;
|
|
}
|