dilim/git
1
0 Fork 0
mirror of https://github.com/git/git.git synced 2026-11-06 13:28:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'cc/pack-index-v3' into seen

Browse Source 
The third part of the SHA-1/SHA-256 interoperability work has been
added, implementing support for reading and writing pack index v3.
This allows packed objects to be mapped between hash algorithms by
storing object IDs in both the main and compatibility hash
algorithms.

* cc/pack-index-v3:
  doc: move pack index v3 documentation
  packfile: allow looking up compat object IDs in packs
  loose: map object IDs using pack index v3
  packfile: validate pack index v3 tables
  packfile: add support for reading pack index v3
  object-file: handle compatibility algorithm
  builtin: populate pack index v3 fields when indexing
  pack-write: add support for pack index v3
  object-file-convert: implement mapping in terms of object DB
  odb: implement interoperability functions
  loose: add a flag to control whether we write mapped OIDs
  loose: rename repo_add_loose_object_map
  loose: allow looking up in object map with source
This commit is contained in:
Junio C Hamano committed 2026-11-05 20:45:35 -08:00
2 parents 37010e7ffa 0443b66ed9
commit ee0bb65a64
25 files changed
+1189 -164
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Documentation/gitformat-pack.adoc
+75
View File
@@ -318,6 +318,81 @@ Pack file entry: <+
Index checksum of all of the above.
== Version 3 pack-*.idx files have the format:
- A header appears at the beginning and consists of the following:
* The 4-byte pack index signature: '\377t0c'
* 4-byte version number: 3
* 4-byte length of the header section, including the signature and
version number
* 4-byte number of objects contained in the pack
* 4-byte number of object formats in this pack index: 1 or 2
* For each object format:
** 4-byte format identifier (e.g., 'sha1' for SHA-1)
** 4-byte length in bytes of shortened object names. This is the
shortest possible length needed to make names in the shortened
object name table unambiguous.
** 8-byte integer, recording where tables relating to this format
are stored in this index file, as an offset from the beginning.
* 8-byte offset to the trailer from the beginning of this file.
* Zero or more additional key/value pairs (4-byte key, 4-byte
value). Only one key is supported: 'PSRC'. See the "Loose objects
and unreachable objects" section of the hash function transition
document (`Documentation/technical/hash-function-transition.adoc`)
for supported values and how this is used. All other keys are reserved. Readers must ignore
unrecognized keys.
- Zero or more NUL bytes. This can optionally be used to improve the
alignment of the full object name table below.
- Tables for the first object format:
* A sorted table of shortened object names. These are prefixes of
the names of all objects in this pack file, packed together
without offset values to reduce the cache footprint of the binary
search for a specific object name.
* A table of full object names in pack order. This allows resolving
a reference to "the nth object in the pack file" (from a
reachability bitmap or from the next table of another object
format) to its object name.
* A table of 4-byte values mapping object name order to pack order.
For an object in the table of sorted shortened object names, the
value at the corresponding index in this table is the index in the
previous table for that same object.
This can be used to look up the object in reachability bitmaps or
to look up its name in another object format.
* A table of 4-byte CRC32 values of the packed object data, in the
order that the objects appear in the pack file. This is to allow
compressed data to be copied directly from pack to pack during
repacking without undetected data corruption.
* A table of 4-byte offset values. The index of this table in pack order
indicates where that object can be found in the pack file. These are
usually 31-bit pack file offsets, but large offsets are encoded as
an index into the next table with the most significant bit set.
* A table of 8-byte offset entries (empty for pack files less than
2 GiB). Pack files are organized with heavily used objects toward
the front, so most object references should not need to refer to
this table.
- Zero or more NUL bytes.
- Tables for the second object format, with the same layout as above,
up to and not including the table of CRC32 values.
- Zero or more NUL bytes.
- The trailer consists of the following:
* A copy of the hash checksum at the end of the corresponding
packfile.
* Hash checksum of all of the above.
Note that the two hashes at the end of the index file are in whatever the
repository's main algorithm is. In the usual case when there are multiple
algorithms, the main algorithm will be SHA-256 and the compatibility algorithm
will be SHA-1. Git always writes the main algorithm first, but doesn't require this.
All 4-byte numbers are in network order and must be 4-byte aligned in the file,
so the NUL padding may be required in some cases.
== pack-*.rev files have the format:
- A 4-byte magic number '0x52494458' ('RIDX').
builtin/index-pack.c
+98 -9
View File
@@ -31,6 +31,8 @@
#include "run-command.h"
#include "setup.h"
#include "strvec.h"
#include "loose.h"
#include "object-file-convert.h"
static const char index_pack_usage[] =
"git index-pack [-v] [-o <index-file>] [--keep | --keep=<msg>] [--[no-]rev-index] [--verify] [--strict[=<msg-id>=<severity>...]] [--fsck-objects[=<msg-id>=<severity>...]] (<pack-file> | --stdin [--fix-thin] [<pack-file>])";
@@ -182,6 +184,10 @@ static pthread_mutex_t deepest_delta_mutex;
#define deepest_delta_lock() lock_mutex(&deepest_delta_mutex)
#define deepest_delta_unlock() unlock_mutex(&deepest_delta_mutex)
static pthread_mutex_t repo_mutex;
#define repo_lock() lock_mutex(&repo_mutex)
#define repo_unlock() unlock_mutex(&repo_mutex)
static pthread_key_t key;
static inline void lock_mutex(pthread_mutex_t *mutex)
@@ -205,6 +211,7 @@ static void init_thread(void)
init_recursive_mutex(&read_mutex);
pthread_mutex_init(&counter_mutex, NULL);
pthread_mutex_init(&work_mutex, NULL);
pthread_mutex_init(&repo_mutex, NULL);
if (show_stat)
pthread_mutex_init(&deepest_delta_mutex, NULL);
pthread_key_create(&key, NULL);
@@ -225,6 +232,7 @@ static void cleanup_thread(void)
pthread_mutex_destroy(&read_mutex);
pthread_mutex_destroy(&counter_mutex);
pthread_mutex_destroy(&work_mutex);
pthread_mutex_destroy(&repo_mutex);
if (show_stat)
pthread_mutex_destroy(&deepest_delta_mutex);
for (i = 0; i < nr_threads; i++)
@@ -469,13 +477,15 @@ static int is_delta_type(enum object_type type)
}
static void *unpack_entry_data(off_t offset, size_t size,
enum object_type type, struct object_id *oid)
enum object_type type, struct object_id *oid,
struct object_id *compat_oid)
{
static char fixed_buf[8192];
int status;
git_zstream stream;
void *buf;
struct git_hash_ctx c;
struct git_hash_ctx c, c_compat;
const struct git_hash_algo *compat = the_repository->compat_hash_algo;
char hdr[32];
int hdrlen;
@@ -483,8 +493,15 @@ static void *unpack_entry_data(off_t offset, size_t size,
hdrlen = format_object_header(hdr, sizeof(hdr), type, size);
git_hash_init(&c, the_hash_algo);
git_hash_update(&c, hdr, hdrlen);
} else
if (compat && compat_oid) {
git_hash_init(&c_compat, compat);
if (type == OBJ_BLOB)
git_hash_update(&c_compat, hdr, hdrlen);
}
} else {
oid = NULL;
compat_oid = NULL;
}
if (type == OBJ_BLOB &&
size > repo_settings_get_big_file_threshold(the_repository))
buf = fixed_buf;
@@ -504,6 +521,8 @@ static void *unpack_entry_data(off_t offset, size_t size,
use(input_len - stream.avail_in);
if (oid)
git_hash_update(&c, last_out, stream.next_out - last_out);
if (compat && compat_oid && type == OBJ_BLOB)
git_hash_update(&c_compat, last_out, stream.next_out - last_out);
if (buf == fixed_buf) {
stream.next_out = buf;
stream.avail_out = sizeof(fixed_buf);
@@ -514,13 +533,43 @@ static void *unpack_entry_data(off_t offset, size_t size,
git_inflate_end(&stream);
if (oid)
git_hash_final_oid(oid, &c);
if (compat && compat_oid) {
if (type != OBJ_BLOB) {
/*
* This is not a blob. Let's remap it to the proper
* algorithm.
*/
struct strbuf outbuf = STRBUF_INIT;
int ret;
repo_lock();
ret = convert_object_file(the_repository, &outbuf,
the_repository->hash_algo,
compat,
buf, size, type, 0);
repo_unlock();
if (ret < 0)
bad_object(offset, _("could not convert object"));
hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX,
type_name(type),(uintmax_t)outbuf.len) + 1;
git_hash_update(&c_compat, hdr, hdrlen);
git_hash_update(&c_compat, outbuf.buf, outbuf.len);
strbuf_release(&outbuf);
}
git_hash_final_oid(compat_oid, &c_compat);
repo_lock();
odb_write_oid_map(the_repository->objects, oid, compat_oid, 0);
repo_unlock();
}
return buf == fixed_buf ? NULL : buf;
}
static void *unpack_raw_entry(struct object_entry *obj,
off_t *ofs_offset,
struct object_id *ref_oid,
struct object_id *oid)
struct object_id *oid,
struct object_id *compat_oid)
{
unsigned char *p;
size_t size, c;
@@ -582,7 +631,7 @@ static void *unpack_raw_entry(struct object_entry *obj,
}
obj->hdr_size = consumed_bytes - obj->idx.offset;
data = unpack_entry_data(obj->idx.offset, obj->size, obj->type, oid);
data = unpack_entry_data(obj->idx.offset, obj->size, obj->type, oid, compat_oid);
obj->idx.crc32 = input_crc32;
return data;
}
@@ -1069,6 +1118,45 @@ static struct base_data *resolve_delta(struct object_entry *delta_obj,
bad_object(delta_obj->idx.offset, _("failed to apply delta"));
hash_object_file(the_hash_algo, result_data, result_size,
delta_obj->real_type, &delta_obj->idx.oid);
if (the_repository->compat_hash_algo) {
struct strbuf outbuf = STRBUF_INIT;
int ret = 0;
char *buf;
size_t len;
if (delta_obj->real_type != OBJ_BLOB) {
repo_lock();
ret = convert_object_file(the_repository, &outbuf,
the_repository->hash_algo,
the_repository->compat_hash_algo,
result_data, result_size,
delta_obj->real_type, 0);
repo_unlock();
buf = outbuf.buf;
len = outbuf.len;
} else {
buf = result_data;
len = result_size;
}
if (ret < 0)
bad_object(delta_obj->idx.offset,
_("could not convert object from delta"));
hash_object_file(the_repository->compat_hash_algo, buf, len,
delta_obj->real_type, &delta_obj->idx.compat_oid);
/*
* We add this entry to the loose object map but do not write it
* to disk because we may need it available when mapping other
* objects. Storing it temporarily in the loose object map lets
* us keep it around for that purpose.
*/
repo_lock();
odb_write_oid_map(the_repository->objects,
&delta_obj->idx.oid,
&delta_obj->idx.compat_oid, 0);
repo_unlock();
strbuf_release(&outbuf);
}
sha1_object(result_data, NULL, result_size, delta_obj->real_type,
&delta_obj->idx.oid);
@@ -1265,7 +1353,8 @@ static void parse_pack_objects(unsigned char *hash)
struct object_entry *obj = &objects[i];
void *data = unpack_raw_entry(obj, &ofs_delta->offset,
&ref_delta_oid,
&obj->idx.oid);
&obj->idx.oid,
&obj->idx.compat_oid);
obj->real_type = obj->type;
if (obj->type == OBJ_OFS_DELTA) {
nr_ofs_deltas++;
@@ -1669,7 +1758,7 @@ static int git_index_pack_config(const char *k, const char *v,
if (!strcmp(k, "pack.indexversion")) {
opts->version = git_config_int(k, v, ctx->kvi);
if (opts->version > 2)
if (opts->version > 3)
die(_("bad pack.indexVersion=%"PRIu32), opts->version);
return 0;
}
@@ -1752,7 +1841,7 @@ static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)
/* Read the attributes from the existing idx file */
opts->version = p->index_version;
if (opts->version == 2)
if (opts->version >= 2)
read_v2_anomalous_offsets(p, opts);
/*
@@ -1991,7 +2080,7 @@ int cmd_index_pack(int argc,
} else if (starts_with(arg, "--index-version=")) {
char *c;
opts.version = strtoul(arg + 16, &c, 10);
if (opts.version > 2)
if (opts.version > 3)
die(_("bad %s"), arg);
if (*c == ',')
opts.off32_limit = strtoul(c+1, &c, 0);
builtin/pack-objects.c
+7 -1
View File
@@ -47,6 +47,9 @@
#include "blob.h"
#include "tree.h"
#include "path-walk.h"
#include "trace2.h"
#include "loose.h"
#include "object-file-convert.h"
/*
* Objects we are going to pack are collected in the `to_pack` structure.
@@ -1849,6 +1852,9 @@ static struct object_entry *create_object_entry(const struct object_id *oid,
entry = packlist_alloc(&to_pack, oid);
entry->hash = hash;
if (repo_oid_to_algop(the_repository, &entry->idx.oid,
the_repository->compat_hash_algo, &entry->idx.compat_oid) < 0)
die(_("can't map object %s while writing pack"), oid_to_hex(oid));
oe_set_type(entry, type);
if (exclude)
entry->preferred_base = 1;
@@ -5137,7 +5143,7 @@ static int option_parse_index_version(const struct option *opt,
BUG_ON_OPT_NEG(unset);
popts->version = strtoul(val, &c, 10);
if (popts->version > 2)
if (popts->version > 3)
die(_("unsupported index version %s"), val);
if (*c == ',' && c[1])
popts->off32_limit = strtoul(c+1, &c, 0);
hash-lookup.c
+41 -9
View File
@@ -104,18 +104,13 @@ int oid_pos(const struct object_id *oid, const void *table, size_t nr,
return index_pos_to_insert_pos(lo);
}
int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
const unsigned char *table, size_t stride, uint32_t *result)
static int bsearch_hash_1(const unsigned char *hash, const unsigned char *table,
size_t stride, size_t cmpsize, uint32_t hi,
uint32_t lo, uint32_t *result)
{
uint32_t hi, lo;
hi = ntohl(fanout_nbo[*hash]);
lo = ((*hash == 0x0) ? 0 : ntohl(fanout_nbo[*hash - 1]));
while (lo < hi) {
unsigned mi = lo + (hi - lo) / 2;
int cmp = hashcmp(table + mi * stride, hash,
the_repository->hash_algo);
int cmp = memcmp(table + mi * stride, hash, cmpsize);
if (!cmp) {
if (result)
@@ -132,3 +127,40 @@ int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
*result = lo;
return 0;
}
int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
const unsigned char *table, size_t stride, uint32_t *result)
{
uint32_t hi, lo;
hi = ntohl(fanout_nbo[*hash]);
lo = ((*hash == 0x0) ? 0 : ntohl(fanout_nbo[*hash - 1]));
return bsearch_hash_1(hash, table, stride, the_hash_algo->rawsz, hi, lo, result);
}
/* full_oid is assumed to follow directly behind short_oid in memory. */
int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,
const uint32_t *pack_map, const unsigned char *short_oid,
size_t stride, size_t hashsz, uint32_t *result)
{
uint32_t nr = (full_oid - short_oid) / stride;
uint32_t obj_off, index;
int res = bsearch_hash_1(hash, short_oid, stride, stride, nr,
0, &obj_off);
if (result)
*result = obj_off;
if (!res)
return 0;
/*
* obj_off is now the offset in the shortened list of object names that
* we've found. We now need to look it up in the mapping of pack order
* to find the full object ID. index is the offset in pack order.
*/
index = ntohl(pack_map[obj_off]);
if (index >= nr)
return 0;
if (!memcmp(hash, full_oid + (hashsz * index), hashsz))
return 1;
return 0;
}
hash-lookup.h
+18 -1
View File
@@ -20,7 +20,7 @@ int oid_pos(const struct object_id *oid,
* first byte is less than or equal to i
* - table: a sorted list of hashes with optional extra information in between
* - stride: distance between two consecutive elements in table (should be
* GIT_MAX_RAWSZ or greater)
* GIT_MAX_RAWSZ or less)
* - result: if not NULL, this function stores the element index of the
* position found (if the search is successful) or the index of the least
* element that is greater than hash (if the search is not successful)
@@ -29,4 +29,21 @@ int oid_pos(const struct object_id *oid,
*/
int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
const unsigned char *table, size_t stride, uint32_t *result);
/*
* Performs a binary search for hash, returning 1 if it is found and 0 if not.
*
* - hash is the object ID to search for.
* - short_oid is a sorted table of equal-length prefixes of the hashes (each of
* which is stride bytes in length.
* - full_oid, which must follow in memory directly behind short_oid, is a list
* of full object IDs in an arbitrary order (usually pack order); each item is
* hashsz bytes in length.
* - pack_map is a 4-byte mapping of indices of items in short_oid to full_oid.
* - result may be NULL; if not, it is the index of the item in the list of
* object IDs if found and the least element greater than it otherwise.
*/
int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,
const uint32_t *pack_map, const unsigned char *short_oid,
size_t stride, size_t hashsz, uint32_t *result);
#endif
loose.c
+34 -16
View File
@@ -220,9 +220,10 @@ errout:
return -1;
}
int repo_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid)
int source_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid,
int flags)
{
int inserted = 0;
@@ -230,32 +231,49 @@ int repo_add_loose_object_map(struct odb_source_loose *loose,
return 0;
inserted = insert_loose_map(loose->map, loose->cache, oid, compat_oid);
if (inserted)
if (inserted && (flags & OBJ_MAP_WRITE))
return write_one_object(loose, oid, compat_oid);
return 0;
}
int source_loose_object_map_oid(struct odb_source_loose *loose,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest)
{
kh_oid_map_t *map;
khiter_t pos;
struct loose_object_map *loose_map = loose->map;
struct repository *repo = loose->base.odb->repo;
if (!loose_map)
return -1;
map = (to == repo->compat_hash_algo) ?
loose_map->to_compat :
loose_map->to_storage;
pos = kh_get_oid_map(map, *src);
if (pos < kh_end(map)) {
oidcpy(dest, kh_value(map, pos));
return 0;
}
return -1;
}
int repo_loose_object_map_oid(struct repository *repo,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest)
{
struct odb_source_files *files = odb_source_files_downcast(repo->objects->source);
kh_oid_map_t *map;
khiter_t pos;
for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
struct loose_object_map *loose_map = dir->loose->map;
if (!loose_map)
int ret;
if (!dir->loose)
continue;
map = (to == repo->compat_hash_algo) ?
loose_map->to_compat :
loose_map->to_storage;
pos = kh_get_oid_map(map, *src);
if (pos < kh_end(map)) {
oidcpy(dest, kh_value(map, pos));
return 0;
}
ret = source_loose_object_map_oid(dir->loose, src, to, dest);
if (ret != -1)
return ret;
}
return -1;
}
loose.h
+12 -3
View File
@@ -3,6 +3,9 @@
#include "khash.h"
/* Should we write this object to disk? */
#define OBJ_MAP_WRITE (1 << 6)
struct repository;
struct odb_source_loose;
@@ -18,9 +21,6 @@ int repo_loose_object_map_oid(struct repository *repo,
const struct object_id *src,
const struct git_hash_algo *dest_algo,
struct object_id *dest);
int repo_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid);
int repo_read_loose_object_map(struct repository *repo);
/*
@@ -32,4 +32,13 @@ int repo_read_loose_object_map(struct repository *repo);
int loose_object_idx_merge(struct repository *repo, const char *src,
const char *dst);
int source_loose_object_map_oid(struct odb_source_loose *loose,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest);
int source_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid,
int flags);
#endif
object-file-convert.c
+2 -12
View File
@@ -7,6 +7,7 @@
#include "repository.h"
#include "hash.h"
#include "object.h"
#include "odb.h"
#include "loose.h"
#include "commit.h"
#include "gpg-interface.h"
@@ -35,18 +36,7 @@ int repo_oid_to_algop(struct repository *repo, const struct object_id *srcoid,
oidcpy(dest, src);
return 0;
}
if (repo_loose_object_map_oid(repo, src, to, dest)) {
/*
* We may have loaded the object map at repo initialization but
* another process (perhaps upstream of a pipe from us) may have
* written a new object into the map. If the object is missing,
* let's reload the map to see if the object has appeared.
*/
repo_read_loose_object_map(repo);
if (repo_loose_object_map_oid(repo, src, to, dest))
return -1;
}
return 0;
return odb_map_oid(repo->objects, src, to, dest, 0);
}
static int decode_tree_entry_raw(struct object_id *oid, const char **path,
object-file.c
+16 -2
View File
@@ -748,6 +748,7 @@ static int hash_stream(struct odb_stream *stream,
*/
static void stream_to_pack(struct transaction_packfile *state,
struct git_hash_ctx *ctx,
struct git_hash_ctx *compat_ctx,
struct odb_stream *stream)
{
git_zstream s;
@@ -775,6 +776,9 @@ static void stream_to_pack(struct transaction_packfile *state,
is_finished = true;
git_hash_update(ctx, ibuf, rsize);
if (compat_ctx)
git_hash_update(compat_ctx,
ibuf, rsize);
s.next_in = ibuf;
s.avail_in = rsize;
@@ -883,16 +887,22 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas
struct odb_transaction_files,
base);
struct transaction_packfile *state = &transaction->packfile;
struct git_hash_ctx ctx;
struct git_hash_ctx ctx, compat_ctx;
struct object_id compat_oid;
unsigned char obuf[16384];
unsigned header_len;
struct hashfile_checkpoint checkpoint;
struct pack_idx_entry *idx;
const struct git_hash_algo *compat = transaction->base.source->odb->repo->compat_hash_algo;
header_len = format_object_header((char *)obuf, sizeof(obuf),
stream->type, stream->size);
git_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);
git_hash_update(&ctx, obuf, header_len);
if (compat) {
git_hash_init(&compat_ctx, compat);
git_hash_update(&compat_ctx, obuf, header_len);
}
/*
* If writing another object to the packfile could result in it
@@ -917,8 +927,10 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas
hashfile_checkpoint(state->f, &checkpoint);
idx->offset = state->offset;
crc32_begin(state->f);
stream_to_pack(state, &ctx, stream);
stream_to_pack(state, &ctx, compat ? &compat_ctx : NULL, stream);
git_hash_final_oid(result_oid, &ctx);
if (compat)
git_hash_final_oid(&compat_oid, &compat_ctx);
idx->crc32 = crc32_end(state->f);
if (already_written(transaction, result_oid)) {
@@ -927,6 +939,8 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas
free(idx);
} else {
oidcpy(&idx->oid, result_oid);
if (compat)
oidcpy(&idx->compat_oid, &compat_oid);
ALLOC_GROW(state->written,
state->nr_written + 1,
state->alloc_written);
odb.c
+17
View File
@@ -625,6 +625,23 @@ out:
return ret;
}
int odb_map_oid(struct object_database *odb,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest,
enum odb_map_oid_flags flags)
{
return odb_source_map_oid(odb->source, src, to, dest, flags);
}
int odb_write_oid_map(struct object_database *odb,
const struct object_id *oid,
const struct object_id *compat_oid,
enum odb_write_oid_map_flags flags)
{
return odb_source_write_oid_map(odb->source, oid, compat_oid, flags);
}
/*
* Return the slot of the most-significant bit set in "val". There are various
* ways to do this quickly with fls() or __builtin_clzl(), but speed is
odb.h
+24
View File
@@ -2,6 +2,7 @@
#define ODB_H
#include "hashmap.h"
#include "loose.h"
#include "object.h"
#include "oid-array.h"
#include "oidset.h"
@@ -650,6 +651,29 @@ static inline int odb_write_object(struct object_database *odb,
struct odb_stream;
enum odb_map_oid_flags {
/* Reload this source to obtain any new information. */
ODB_MAP_OID_RELOAD = (1 << 0),
};
int odb_map_oid(struct object_database *odb,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest,
enum odb_map_oid_flags flags);
enum odb_write_oid_map_flags {
/*
* Serialize this data to disk instead of persisting it only in memory.
*/
ODB_WRITE_OID_MAP_WRITE = OBJ_MAP_WRITE,
};
int odb_write_oid_map(struct object_database *odb,
const struct object_id *oid,
const struct object_id *compat_oid,
enum odb_write_oid_map_flags flags);
int odb_write_object_stream(struct object_database *odb,
struct odb_stream *stream,
struct object_id *oid);
odb/source-files.c
+38
View File
@@ -6,6 +6,7 @@
#include "gettext.h"
#include "hex.h"
#include "lockfile.h"
#include "loose.h"
#include "object-file.h"
#include "odb.h"
#include "odb/source.h"
@@ -1290,6 +1291,41 @@ struct odb_files_dir *odb_source_files_find_dir(struct object_database *odb, con
return dir;
}
static int odb_source_files_map_oid(struct odb_source *source,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest,
enum odb_map_oid_flags flags)
{
struct odb_source_files *files = odb_source_files_downcast(source);
for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
int ret;
ret = odb_source_map_oid(&dir->loose->base, src, to, dest, flags);
if (ret != -1)
return ret;
ret = odb_source_map_oid(&dir->packed->base, src, to, dest, flags);
if (ret != -1)
return ret;
ret = odb_source_map_oid(&dir->loose->base, src, to, dest,
flags | ODB_MAP_OID_RELOAD);
if (ret != -1)
return ret;
}
return -1;
}
static inline int odb_source_files_write_oid_map(struct odb_source *source,
const struct object_id *oid,
const struct object_id *compat_oid,
enum odb_write_oid_map_flags flags)
{
struct odb_source_files *files = odb_source_files_downcast(source);
return odb_source_write_oid_map(&files->dirs->loose->base, oid, compat_oid, flags);
}
struct odb_source_files *odb_source_files_new(struct object_database *odb,
const char *payload,
enum odb_new_flags flags)
@@ -1332,6 +1368,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
files->base.write_object = odb_source_files_write_object;
files->base.write_object_stream = odb_source_files_write_object_stream;
files->base.begin_transaction = odb_source_files_begin_transaction;
files->base.map_oid = odb_source_files_map_oid;
files->base.write_oid_map = odb_source_files_write_oid_map;
files->base.optimize = odb_source_files_optimize;
files->base.optimize_required = odb_source_files_optimize_required;
files->base.generate_pack = odb_source_files_generate_pack;
odb/source-loose.c
+29 -2
View File
@@ -864,7 +864,7 @@ static int odb_source_loose_write_object(struct odb_source *source,
return -1;
if (compat_oid)
return repo_add_loose_object_map(loose, oid, compat_oid);
return source_add_loose_object_map(loose, oid, compat_oid, OBJ_MAP_WRITE);
return 0;
}
@@ -991,7 +991,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
loose_cache_mark_seen(loose, oid);
if (compat)
err = repo_add_loose_object_map(loose, oid, &compat_oid);
err = source_add_loose_object_map(loose, oid, &compat_oid, OBJ_MAP_WRITE);
cleanup:
strbuf_release(&tmp_file);
@@ -1039,6 +1039,31 @@ static void odb_source_loose_reparent(const char *old_cwd,
loose->base.path = path;
}
static int odb_source_loose_map_oid(struct odb_source *source, const struct object_id *src,
const struct git_hash_algo *to, struct object_id *dest,
enum odb_map_oid_flags flags)
{
struct odb_source_loose *loose = odb_source_loose_downcast(source);
/*
* We may have loaded the object map at repo initialization but
* another process (perhaps upstream of a pipe from us) may have
* written a new object into the map. If the object is missing,
* let's reload the map to see if the object has appeared.
*/
if (flags & ODB_MAP_OID_RELOAD)
loose_object_map_load(loose);
return source_loose_object_map_oid(loose, src, to, dest);
}
static inline int odb_source_loose_write_oid_map(struct odb_source *source,
const struct object_id *oid,
const struct object_id *compat_oid,
enum odb_write_oid_map_flags flags)
{
struct odb_source_loose *loose = odb_source_loose_downcast(source);
return source_add_loose_object_map(loose, oid, compat_oid, flags);
}
static void odb_source_loose_free(struct odb_source *source)
{
struct odb_source_loose *loose = odb_source_loose_downcast(source);
@@ -1160,6 +1185,8 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
loose->base.write_object = odb_source_loose_write_object;
loose->base.write_object_stream = odb_source_loose_write_object_stream;
loose->base.begin_transaction = odb_source_loose_begin_transaction;
loose->base.map_oid = odb_source_loose_map_oid;
loose->base.write_oid_map = odb_source_loose_write_oid_map;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(odb_source_loose_reparent, loose);
odb/source-packed.c
+92 -24
View File
@@ -286,37 +286,64 @@ static int for_each_prefixed_object_in_pack(
int len = opts->prefix_hex_len > p->repo->hash_algo->hexsz ?
p->repo->hash_algo->hexsz : opts->prefix_hex_len;
int ret;
struct object_id wanted;
num = p->num_objects;
bsearch_pack(opts->prefix, p, &first);
/*
* At this point, "first" is the location of the lowest object
* with an object name that could match "bin_pfx". See if we have
* 0, 1 or more objects that actually match(es).
*/
for (i = first; i < num; i++) {
struct object_id oid;
for (uint32_t algo = 1; algo < GIT_HASH_NALGOS; algo++) {
const struct git_hash_algo *algop = &hash_algos[algo];
nth_packed_object_id(&oid, p, i);
if (!match_hash(len, opts->prefix->hash, oid.hash))
break;
/*
* Skip this algorithm unless it's the desired one or we've
* asked for any algorithm.
*/
if (opts->prefix->algo != GIT_HASH_UNKNOWN &&
opts->prefix->algo != algo)
continue;
/*
* Skip this algorithm unless it's the main algorithm or we have
* a compatibility algorithm and it's that algorithm.
*/
if (algop != p->repo->hash_algo &&
(!p->repo->compat_hash_algo ||
algop != p->repo->compat_hash_algo))
continue;
if (p->index_version < 3 && algop != p->repo->hash_algo)
continue;
if (data->request) {
struct object_info oi = *data->request;
off_t offset = nth_packed_object_offset(p, i);
oidcpy(&wanted, opts->prefix);
wanted.algo = algo;
ret = packed_object_info(source, p, offset, &oi);
if (ret)
goto out;
bsearch_pack(&wanted, p, &first);
ret = data->cb(&oid, &oi, data->cb_data);
if (ret)
goto out;
} else {
ret = data->cb(&oid, NULL, data->cb_data);
if (ret)
goto out;
/*
* At this point, "first" is the location of the lowest object
* with an object name that could match "bin_pfx". See if we have
* 0, 1 or more objects that actually match(es).
*/
for (i = first; i < num; i++) {
struct object_id oid;
if (nth_packed_object_id_algop(&oid, p, i, algop, algop) ||
!match_hash(len, opts->prefix->hash, oid.hash))
break;
if (data->request) {
struct object_info oi = *data->request;
off_t offset = nth_packed_object_offset_algop(p, i, algop);
ret = packed_object_info(source, p, offset, &oi);
if (ret)
goto out;
ret = data->cb(&oid, &oi, data->cb_data);
if (ret)
goto out;
} else {
ret = data->cb(&oid, NULL, data->cb_data);
if (ret)
goto out;
}
}
}
@@ -854,6 +881,45 @@ static void odb_source_packed_close(struct odb_source *source)
packed->midx = NULL;
}
static int odb_source_packed_map_oid(struct odb_source *source,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest,
enum odb_map_oid_flags flags UNUSED)
{
struct odb_source_packed *packed = odb_source_packed_downcast(source);
struct repository *repo = packed->base.odb->repo;
struct packfile_list_entry *e;
for (e = packfile_store_get_packs(packed); e; e = e->next) {
struct packed_git *p = e->pack;
uint32_t off;
const struct git_hash_algo *src_algo = src->algo ?
&hash_algos[src->algo] :
repo->hash_algo;
if (open_pack_index(p) || !p->num_objects)
continue;
/* Only v3 indices store object IDs in both algorithms. */
if (p->index_version < 3)
continue;
if (!bsearch_pack(src, p, &off))
continue;
if (nth_packed_object_id_algop(dest, p, off, src_algo, to))
continue;
return 0;
}
return -1;
}
static inline int odb_source_packed_write_oid_map(struct odb_source *source UNUSED,
const struct object_id *oid UNUSED,
const struct object_id *compat_oid UNUSED,
enum odb_write_oid_map_flags flags UNUSED)
{
return -1;
}
static void odb_source_packed_free(struct odb_source *source)
{
struct odb_source_packed *packed = odb_source_packed_downcast(source);
@@ -1005,6 +1071,8 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
packed->base.write_object = odb_source_packed_write_object;
packed->base.write_object_stream = odb_source_packed_write_object_stream;
packed->base.begin_transaction = odb_source_packed_begin_transaction;
packed->base.map_oid = odb_source_packed_map_oid;
packed->base.write_oid_map = odb_source_packed_write_oid_map;
if (!is_absolute_path(path))
chdir_notify_register(odb_source_packed_reparent, packed);
odb/source.h
+65
View File
@@ -285,6 +285,42 @@ struct odb_source {
*/
int (*fsck)(struct odb_source *source,
struct odb_fsck_options *options);
/*
* This callback is expected to provide the mapping of an object from
* the algorithm specified in src to the algorithm specified in to,
* leaving the result in dest. flags is a combination of `enum
* odb_map_oid_flags` values: with ODB_MAP_OID_RELOAD, the source should
* reload its mapping data before the lookup, as another process may
* have written new mappings in the meantime.
*
* The algo field in src must point to a valid algorithm (specifically,
* it must not be zero). to must likewise point to a valid algorithm;
* it must be non-NULL.
*
* The callback is expected to return 0 on success, -1 if the object is
* not found, and a different negative error code on error.
*/
int (*map_oid)(struct odb_source *source, const struct object_id *src,
const struct git_hash_algo *to, struct object_id *dest,
enum odb_map_oid_flags flags);
/*
* This callback is expected to write a mapping of an object between oid
* and compat_oid. flags is a combination of `enum
* odb_write_oid_map_flags` values: with ODB_WRITE_OID_MAP_WRITE, the
* mapping should be persisted to disk; otherwise it may only be kept in
* memory.
*
* The algo fields in oid and compat_oid must point to valid algorithms
* (specifically, they must not be zero).
*
* The callback is expected to return 0 on success, and a negative
* error code on error.
*/
int (*write_oid_map)(struct odb_source *source, const struct object_id *oid,
const struct object_id *compat_oid,
enum odb_write_oid_map_flags flags);
};
/*
@@ -540,4 +576,33 @@ static inline int odb_source_fsck(struct odb_source *source,
return source->fsck(source, opts);
}
/*
* Map the specified object ID into the specified algorithm.
*
* Returns 0 on success, -1 if the object is not found, and a different negative
* error code otherwise.
*/
static inline int odb_source_map_oid(struct odb_source *source,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest,
int flags)
{
return source->map_oid(source, src, to, dest, flags);
}
/*
* Create a mapping between the two object IDs, which should be in the main and
* compatibility hash algorithms.
*
* Returns 0 on success, a negative error otherwise.
*/
static inline int odb_source_write_oid_map(struct odb_source *source,
const struct object_id *oid,
const struct object_id *compat_oid,
int flags)
{
return source->write_oid_map(source, oid, compat_oid, flags);
}
#endif
pack-check.c
+3 -1
View File
@@ -44,7 +44,9 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs,
} while (len);
index_crc = p->index_data;
index_crc += 2 + 256 + (size_t)p->num_objects * (p->repo->hash_algo->rawsz/4) + nr;
if (p->index_version == 3)
nr = nth_packed_object_pack_order_algop(p, nr, p->repo->hash_algo);
index_crc += (p->crc_offset / 4) + nr;
return data_crc != ntohl(*index_crc);
}
pack-revindex.c
+21 -3
View File
@@ -138,11 +138,28 @@ static void create_pack_revindex(struct packed_git *p)
const unsigned hashsz = p->repo->hash_algo->rawsz;
ALLOC_ARRAY(p->revindex, num_ent + 1);
index += 4 * 256;
if (p->index_version > 1) {
if (p->index_version == 3) {
const uint32_t *off_32 =
(uint32_t *)(index + 8 + (size_t)p->num_objects * (hashsz + 4));
(uint32_t *)(index + st_add(p->crc_offset, st_mult(p->num_objects, 4)));
const uint32_t *off_64 = off_32 + p->num_objects;
for (i = 0; i < num_ent; i++) {
const uint32_t pack_order = nth_packed_object_pack_order_algop(p, i, p->repo->hash_algo);
const uint32_t off = ntohl(off_32[pack_order]);
const uint32_t lowoff = off & 0x7fffffff;
if (!(off & 0x80000000)) {
p->revindex[i].offset = off;
} else {
const uint32_t *ptr = off_64 + st_mult(lowoff, 2);
check_pack_index_ptr(p, ptr);
p->revindex[i].offset = get_be64(ptr);
}
p->revindex[i].nr = i;
}
} else if (p->index_version == 2) {
const uint32_t *off_32 =
(uint32_t *)(index + st_add(p->crc_offset, st_mult(p->num_objects, 4)));
const uint32_t *off_64 = off_32 + p->num_objects;
for (i = 0; i < num_ent; i++) {
const uint32_t off = ntohl(*off_32++);
@@ -155,6 +172,7 @@ static void create_pack_revindex(struct packed_git *p)
p->revindex[i].nr = i;
}
} else {
index += 4 * 256;
for (i = 0; i < num_ent; i++) {
const uint32_t hl = *((uint32_t *)(index + (hashsz + 4) * i));
p->revindex[i].offset = ntohl(hl);
pack-write.c
+233 -30
View File
@@ -22,13 +22,20 @@ void reset_pack_idx_option(struct pack_idx_option *opts)
opts->delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT;
}
static int sha1_compare(const void *_a, const void *_b)
static int oid_compare(const void *_a, const void *_b)
{
struct pack_idx_entry *a = *(struct pack_idx_entry **)_a;
struct pack_idx_entry *b = *(struct pack_idx_entry **)_b;
return oidcmp(&a->oid, &b->oid);
}
static int oid_compare_compat(const void *_a, const void *_b)
{
struct pack_idx_entry *a = *(struct pack_idx_entry **)_a;
struct pack_idx_entry *b = *(struct pack_idx_entry **)_b;
return oidcmp(&a->compat_oid, &b->compat_oid);
}
static int cmp_uint32(const void *a_, const void *b_)
{
uint32_t a = *((uint32_t *)a_);
@@ -50,30 +57,156 @@ static int need_large_offset(off_t offset, const struct pack_idx_option *opts)
sizeof(ofsval), cmp_uint32);
}
static inline int last_matching_offset(const struct object_id *a, const struct object_id *b,
const struct git_hash_algo *algop)
{
size_t i;
for (i = 0; i < algop->rawsz; i++)
if (a->hash[i] != b->hash[i])
return i;
/* We should never hit this case. */
return i;
}
static const struct object_id *main_oid(const struct pack_idx_entry *ent)
{
return &ent->oid;
}
static const struct object_id *compat_oid(const struct pack_idx_entry *ent)
{
return &ent->compat_oid;
}
static uint32_t find_short_name_len(struct pack_idx_entry **sorted, size_t nr_objects,
const struct git_hash_algo *algop,
const struct object_id *(*fn)(const struct pack_idx_entry *))
{
size_t i, len = 1;
for (i = 1; i < nr_objects; i++) {
const struct object_id *oid1 = fn(sorted[i - 1]);
const struct object_id *oid2 = fn(sorted[i]);
size_t offset = last_matching_offset(oid1, oid2, algop);
if (offset >= len)
len = offset + 1;
}
return len > algop->rawsz ? algop->rawsz : len;
}
struct pack_idx_format_header {
uint32_t id;
uint32_t short_name_len;
uint64_t offset;
};
/* This struct contains fields beyond the signature and version number. */
struct pack_idx_header_v3 {
uint32_t hdr_len;
uint32_t nr_objects;
uint32_t nr_formats;
};
static void write_v3_header(struct repository *repo,
int nr_objects, int nr_large_offset,
uint32_t short_name_len[2], int padding[2],
struct hashfile *f)
{
struct pack_idx_header_v3 hdr;
struct pack_idx_format_header format[2];
const struct git_hash_algo *compat = repo->compat_hash_algo;
uint32_t nformats = compat ? 2 : 1;
/*
* signature (4) + version (4) + length (4) + nobjects(4) + nformats (4) +
* each format: identifier (4) + length of shortened names (4) +
* table offset (8) +
* offset to trailer (8)
*/
uint32_t hdrlen = 4 + 4 + 4 + 4 + 4 + nformats * (4 + 4 + 8) + 8;
uint64_t offset = hdrlen;
uint64_t short_table = short_name_len[0] * nr_objects;
hdr.hdr_len = htonl(hdrlen);
hdr.nr_objects = htonl(nr_objects);
hdr.nr_formats = htonl(nformats);
hashwrite(f, &hdr, sizeof(hdr));
/* padding */
if (short_table & 3) {
padding[0] = 4 - (short_table & 3);
offset += padding[0];
}
format[0].id = htonl(repo->hash_algo->format_id);
format[0].offset = htonll(offset);
format[0].short_name_len = htonl(short_name_len[0]);
hashwrite(f, &format[0], sizeof(format[0]));
offset += (uint64_t)(short_name_len[0] + repo->hash_algo->rawsz + 4 + 4 + 4) * nr_objects +
8 * nr_large_offset;
if (compat) {
short_table = short_name_len[1] * nr_objects;
/* padding */
if (short_table & 3) {
padding[1] = 4 - (short_table & 3);
offset += padding[1];
}
format[1].id = htonl(compat->format_id);
format[1].offset = htonll(offset);
format[1].short_name_len = htonl(short_name_len[1]);
hashwrite(f, &format[1], sizeof(format[1]));
offset += (uint64_t)(short_name_len[1] + compat->rawsz + 4) * nr_objects;
}
offset = htonll(offset);
hashwrite(f, &offset, sizeof(offset));
}
/*
* The *sha1 contains the pack content SHA1 hash.
* The objects array passed in will be sorted by SHA1 on exit.
* The *hash contains the pack content hash.
* The objects array passed in should be in pack order and will be sorted by
* object ID on return.
*/
const char *write_idx_file(struct repository *repo,
const char *index_name, struct pack_idx_entry **objects,
int nr_objects, const struct pack_idx_option *opts,
const unsigned char *sha1)
const unsigned char *hash)
{
struct hashfile *f;
struct pack_idx_entry **sorted_by_sha, **list, **last;
off_t last_obj_offset = 0;
struct pack_idx_entry **sorted_by_sha, **sorted_by_sha_compat = NULL, **list, **last;
int i, fd;
uint32_t index_version;
unsigned int nr_large_offset = 0;
const struct git_hash_algo *compat = repo->compat_hash_algo;
uint32_t short_name_len[2] = { 0 };
int padding_size[2] = { 0 };
uint8_t zero[4] = { 0 };
index_version = opts->version;
if (nr_objects) {
sorted_by_sha = objects;
list = sorted_by_sha;
last = sorted_by_sha + nr_objects;
for (i = 0; i < nr_objects; ++i) {
if (objects[i]->offset > last_obj_offset)
last_obj_offset = objects[i]->offset;
objects[i]->idx = i;
if (need_large_offset(objects[i]->offset, opts))
nr_large_offset++;
}
QSORT(sorted_by_sha, nr_objects, sha1_compare);
/*
* If we're using index v3, we need both pack order and sorted
* order.
*/
if (index_version >= 3) {
DUP_ARRAY(sorted_by_sha, objects, nr_objects);
if (compat) {
DUP_ARRAY(sorted_by_sha_compat, objects,
nr_objects);
QSORT(sorted_by_sha_compat, nr_objects, oid_compare_compat);
}
}
QSORT(sorted_by_sha, nr_objects, oid_compare);
}
else
sorted_by_sha = list = last = NULL;
@@ -95,7 +228,8 @@ const char *write_idx_file(struct repository *repo,
}
/* if last object's offset is >= 2^31 we should use index V2 */
index_version = need_large_offset(last_obj_offset, opts) ? 2 : opts->version;
if (index_version == 1 && nr_large_offset)
index_version = 2;
/* index versions 2 and above need a header */
if (index_version >= 2) {
@@ -105,50 +239,84 @@ const char *write_idx_file(struct repository *repo,
hashwrite(f, &hdr, sizeof(hdr));
}
/*
* Write the first-level table (the list is sorted,
* but we use a 256-entry lookup to be able to avoid
* having to do eight extra binary search iterations).
*/
for (i = 0; i < 256; i++) {
struct pack_idx_entry **next = list;
while (next < last) {
struct pack_idx_entry *obj = *next;
if (obj->oid.hash[0] != i)
break;
next++;
if (index_version >= 3) {
short_name_len[0] = find_short_name_len(sorted_by_sha,
nr_objects,
repo->hash_algo,
main_oid);
if (compat)
short_name_len[1] = find_short_name_len(sorted_by_sha_compat,
nr_objects,
compat, compat_oid);
write_v3_header(repo, nr_objects, nr_large_offset, short_name_len, padding_size, f);
hashwrite(f, zero, padding_size[0]);
/* Write the sorted abbreviated names */
list = sorted_by_sha;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
hashwrite(f, obj->oid.hash, short_name_len[0]);
if ((opts->flags & WRITE_IDX_STRICT) &&
(i && oideq(&list[-2]->oid, &obj->oid)))
die("The same object %s appears twice in the pack",
oid_to_hex(&obj->oid));
}
} else {
/*
* Write the first-level table (the list is sorted,
* but we use a 256-entry lookup to be able to avoid
* having to do eight extra binary search iterations).
*/
for (i = 0; i < 256; i++) {
struct pack_idx_entry **next = list;
while (next < last) {
struct pack_idx_entry *obj = *next;
if (obj->oid.hash[0] != i)
break;
next++;
}
hashwrite_be32(f, next - sorted_by_sha);
list = next;
}
hashwrite_be32(f, next - sorted_by_sha);
list = next;
}
/*
* Write the actual SHA1 entries..
*/
list = sorted_by_sha;
list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
if (index_version < 2)
hashwrite_be32(f, obj->offset);
hashwrite(f, obj->oid.hash, repo->hash_algo->rawsz);
if ((opts->flags & WRITE_IDX_STRICT) &&
if ((opts->flags & WRITE_IDX_STRICT) && index_version < 3 &&
(i && oideq(&list[-2]->oid, &obj->oid)))
die("The same object %s appears twice in the pack",
oid_to_hex(&obj->oid));
}
if (index_version >= 3) {
/* write the sorted-to-pack index */
list = sorted_by_sha;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
uint32_t index = htonl(obj->idx);
hashwrite(f, &index, 4);
}
}
if (index_version >= 2) {
unsigned int nr_large_offset = 0;
/* write the crc32 table */
list = sorted_by_sha;
list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
hashwrite_be32(f, obj->crc32);
}
/* write the 32-bit offset table */
list = sorted_by_sha;
list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
uint32_t offset;
@@ -160,7 +328,7 @@ const char *write_idx_file(struct repository *repo,
}
/* write the large offset table */
list = sorted_by_sha;
list = objects;
while (nr_large_offset) {
struct pack_idx_entry *obj = *list++;
uint64_t offset = obj->offset;
@@ -172,10 +340,45 @@ const char *write_idx_file(struct repository *repo,
}
}
hashwrite(f, sha1, repo->hash_algo->rawsz);
if (index_version >= 3 && compat) {
hashwrite(f, zero, padding_size[1]);
/* Write the sorted abbreviated names */
list = sorted_by_sha_compat;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
hashwrite(f, obj->compat_oid.hash, short_name_len[1]);
if ((opts->flags & WRITE_IDX_STRICT) &&
(i && oideq(&list[-2]->compat_oid, &obj->compat_oid)))
die("The same object %s appears twice in the pack",
oid_to_hex(&obj->compat_oid));
}
list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
hashwrite(f, obj->compat_oid.hash, compat->rawsz);
}
/* write the sorted-to-pack index */
list = sorted_by_sha_compat;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
uint32_t index = htonl(obj->idx);
hashwrite(f, &index, 4);
}
}
hashwrite(f, hash, repo->hash_algo->rawsz);
finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
CSUM_HASH_IN_STREAM | CSUM_CLOSE |
((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
if (sorted_by_sha != objects) {
QSORT(objects, nr_objects, oid_compare);
free(sorted_by_sha);
}
free(sorted_by_sha_compat);
return index_name;
}
pack.h
+2
View File
@@ -79,7 +79,9 @@ struct pack_idx_header {
*/
struct pack_idx_entry {
struct object_id oid;
struct object_id compat_oid;
uint32_t crc32;
uint32_t idx;
off_t offset;
};
packfile.c
+170 -15
View File
@@ -88,7 +88,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)
return -1;
}
idx_size = xsize_t(st.st_size);
if (idx_size < 4 * 256 + hashsz + hashsz) {
if (idx_size < hashsz * 3) {
close(fd);
return error("index file %s is too small", path);
}
@@ -109,14 +109,14 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
struct pack_idx_header *hdr = idx_map;
uint32_t version, nr, i, *index;
if (idx_size < 4 * 256 + hashsz + hashsz)
if (idx_size < hashsz * 3)
return error("index file %s is too small", path);
if (!idx_map)
return error("empty data");
if (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {
version = ntohl(hdr->idx_version);
if (version < 2 || version > 2)
if (version < 2 || version > 3)
return error("index file %s is version %"PRIu32
" and is not supported by this binary"
" (try upgrading GIT to a newer version)",
@@ -124,15 +124,20 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
} else
version = 1;
if (version < 3 && idx_size < 4 * 256 + hashsz + hashsz)
return error("index file %s is too small", path);
nr = 0;
index = idx_map;
if (version > 1)
index += 2; /* skip index header */
for (i = 0; i < 256; i++) {
uint32_t n = ntohl(index[i]);
if (n < nr)
return error("non-monotonic index %s", path);
nr = n;
if (version <= 2) {
for (i = 0; i < 256; i++) {
uint32_t n = ntohl(index[i]);
if (n < nr)
return error("non-monotonic index %s", path);
nr = n;
}
}
if (version == 1) {
@@ -174,6 +179,82 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
(sizeof(off_t) <= 4))
return error("pack too large for current definition of off_t in %s", path);
p->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));
} else if (version == 3) {
uint32_t hdrlen, num_formats, main_algo;
uint64_t trailer_offset;
const uint32_t *q = index;
/* Fixed part of the header (after signature and version) */
hdrlen = ntohl(*q++);
nr = ntohl(*q++);
num_formats = ntohl(*q++);
if (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))
return error("v3 index %s has an invalid number of formats: %"PRIu32,
path, num_formats);
/* Then 16 bytes per format and the 8-byte offset of the trailer */
if (hdrlen < 20 + num_formats * (4 + 4 + 8) + 8 ||
hdrlen > idx_size - 2 * hashsz)
return error("v3 index %s has an invalid header length", path);
trailer_offset = get_be64(q + num_formats * 4);
if (trailer_offset != idx_size - 2 * hashsz)
return error("v3 index %s has a bad size", path);
memset(p->formats, 0, sizeof(p->formats));
for (i = 0; i < num_formats; i++) {
uint32_t format_id = ntohl(*q++);
uint32_t format = hash_algo_by_id(format_id);
struct packed_git_format *pfp;
uint64_t end;
if (format == GIT_HASH_UNKNOWN)
return error("v3 index %s has an unknown format_id 0x%08"PRIx32,
path, format_id);
/*
* We store at the value of the constant minus 1 to
* avoid having to store a useless entry at offset 0.
*/
pfp = &p->formats[format-1];
if (pfp->data_offset)
return error("v3 index %s has duplicate format_id 0x%08"PRIx32,
path, format_id);
pfp->short_name_len = ntohl(*q++);
pfp->data_offset = get_be64(q);
q += 2;
if (!pfp->short_name_len ||
pfp->short_name_len > hash_algos[format].rawsz)
return error("v3 index %s has an invalid short name length",
path);
if (pfp->data_offset < hdrlen ||
pfp->data_offset > trailer_offset)
return error("v3 index %s has an invalid table offset", path);
/* No overflow as pfp->data_offset < idx_size and nr < 2^32 */
pfp->full_oid_offset = pfp->data_offset +
(uint64_t)pfp->short_name_len * nr;
pfp->order_map_offset = pfp->full_oid_offset +
(uint64_t)hash_algos[format].rawsz * nr;
end = pfp->order_map_offset + (uint64_t)nr * 4;
/* After the first format tables: CRC32 and 4-byte offset tables */
if (!i) {
p->crc_offset = end;
end += (uint64_t)nr * (4 + 4); /* CRC32 + offset */
}
if (pfp->order_map_offset % 4 || end > trailer_offset)
return error("v3 index %s has invalid tables", path);
}
main_algo = hash_algo_by_length(hashsz);
if (main_algo == GIT_HASH_UNKNOWN ||
!p->formats[main_algo - 1].data_offset)
return error("v3 index %s is missing data for the main hash algorithm",
path);
}
p->index_version = version;
@@ -1730,6 +1811,26 @@ out:
return data;
}
static int bsearch_pack_v3(const struct object_id *oid, const struct packed_git *p, uint32_t *result)
{
int algo = oid->algo ? oid->algo : hash_algo_by_ptr(p->repo->hash_algo);
const struct packed_git_format *pfp = &p->formats[algo - 1];
const unsigned char *data = p->index_data;
/* This index has no data for the algorithm of the object ID. */
if (!pfp->data_offset) {
if (result)
*result = 0;
return 0;
}
return bsearch_hash_v3(oid->hash, data + pfp->full_oid_offset,
(const uint32_t *)(data + pfp->order_map_offset),
data + pfp->data_offset,
pfp->short_name_len,
hash_algos[algo].rawsz, result);
}
int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32_t *result)
{
const unsigned char *index_fanout = p->index_data;
@@ -1740,6 +1841,9 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32
if (!index_fanout)
BUG("bsearch_pack called without a valid pack-index");
if (p->index_version == 3)
return bsearch_pack_v3(oid, p, result);
index_lookup = index_fanout + 4 * 256;
if (p->index_version == 1) {
index_lookup_width = hashsz + 4;
@@ -1757,9 +1861,19 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32
int nth_packed_object_id(struct object_id *oid,
struct packed_git *p,
uint32_t n)
{
return nth_packed_object_id_algop(oid, p, n, p->repo->hash_algo,
p->repo->hash_algo);
}
int nth_packed_object_id_algop(struct object_id *oid,
struct packed_git *p,
uint32_t n,
const struct git_hash_algo *src,
const struct git_hash_algo *dest)
{
const unsigned char *index = p->index_data;
const unsigned int hashsz = p->repo->hash_algo->rawsz;
const unsigned int hashsz = dest->rawsz;
if (!index) {
if (open_pack_index(p))
return -1;
@@ -1767,13 +1881,25 @@ int nth_packed_object_id(struct object_id *oid,
}
if (n >= p->num_objects)
return -1;
index += 4 * 256;
if (p->index_version < 3 && (src != dest || dest != p->repo->hash_algo))
return -1; /* older indices only store main object IDs */
if (p->index_version == 1) {
index += 4 * 256;
oidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),
p->repo->hash_algo);
dest);
} else if (p->index_version == 2) {
index += 4 * 256 + 8;
oidread(oid, index + st_mult(hashsz, n), dest);
} else {
index += 8;
oidread(oid, index + st_mult(hashsz, n), p->repo->hash_algo);
int algo = hash_algo_by_ptr(dest);
const struct packed_git_format *pfp = &p->formats[algo-1];
uint32_t pack_order_idx;
if (!pfp->data_offset ||
!p->formats[hash_algo_by_ptr(src) - 1].data_offset)
return -1; /* no data for one of the algorithms */
pack_order_idx = nth_packed_object_pack_order_algop(p, n, src);
oidread(oid, index + st_add(pfp->full_oid_offset, st_mult(hashsz, pack_order_idx)), dest);
}
return 0;
}
@@ -1793,15 +1919,28 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
}
off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
{
return nth_packed_object_offset_algop(p, n, p->repo->hash_algo);
}
off_t nth_packed_object_offset_algop(const struct packed_git *p, uint32_t n,
const struct git_hash_algo *algop)
{
const unsigned char *index = p->index_data;
const unsigned int hashsz = p->repo->hash_algo->rawsz;
index += 4 * 256;
if (p->index_version < 3 && algop != p->repo->hash_algo)
BUG("pack index v%d only stores main object IDs",
p->index_version);
if (p->index_version == 1) {
index += 4 * 256;
return ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));
} else {
uint32_t off;
index += st_add(8, st_mult(p->num_objects, hashsz + 4));
if (p->index_version == 3)
n = nth_packed_object_pack_order_algop(p, n, algop);
index += st_add(p->crc_offset, st_mult(p->num_objects, 4));
off = ntohl(*((uint32_t *)(index + st_mult(4, n))));
if (!(off & 0x80000000))
return off;
@@ -1812,6 +1951,22 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
}
}
uint32_t nth_packed_object_pack_order_algop(const struct packed_git *p, uint32_t n,
const struct git_hash_algo *algop)
{
const unsigned char *index = p->index_data;
const unsigned int algo = hash_algo_by_ptr(algop);
const struct packed_git_format *pfp = &p->formats[algo-1];
uint32_t pos = ntohl(*(uint32_t *)(index +
st_add(pfp->order_map_offset,
st_mult(4, n))));
if (pos >= p->num_objects)
die(_("invalid object position in pack index for %s (corrupt index?)"),
p->pack_name);
return pos;
}
off_t find_pack_entry_one(const struct object_id *oid,
struct packed_git *p)
{
packfile.h
+33
View File
@@ -1,6 +1,7 @@
#ifndef PACKFILE_H
#define PACKFILE_H
#include "hash.h"
#include "list.h"
#include "object.h"
#include "odb.h"
@@ -14,12 +15,22 @@
struct object_info;
struct odb_stream;
struct packed_git_format {
/* This is also the location of the short OID table. */
uint64_t data_offset;
uint64_t full_oid_offset;
uint64_t order_map_offset;
uint32_t short_name_len;
};
struct packed_git {
struct pack_window *windows;
off_t pack_size;
const void *index_data;
size_t index_size;
uint32_t num_objects;
uint32_t num_formats;
struct packed_git_format formats[GIT_HASH_NALGOS - 1];
size_t crc_offset;
struct oidset bad_objects;
int index_version;
@@ -285,12 +296,34 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32
*/
int nth_packed_object_id(struct object_id *, struct packed_git *, uint32_t n);
/*
* The same as nth_packed_object_id, but the nth object is counted in src
* and the object ID looked up is in dest.
*/
int nth_packed_object_id_algop(struct object_id *, struct packed_git *,
uint32_t n, const struct git_hash_algo *src,
const struct git_hash_algo *dest);
/*
* Return the offset of the nth object within the specified packfile.
* The index must already be opened.
*/
off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
/*
* The same as nth_packed_object_offset, but the nth object is counted in
* algop. For indices older than v3, algop must be the main algorithm.
*/
off_t nth_packed_object_offset_algop(const struct packed_git *, uint32_t n,
const struct git_hash_algo *algop);
/*
* Return the index into pack order of the nth object within the specified
* packfile. The index must already be opened and it must be a v3 index.
*/
uint32_t nth_packed_object_pack_order_algop(const struct packed_git *, uint32_t n,
const struct git_hash_algo *algop);
/*
* If the object named by oid is present in the specified packfile,
* return its offset within the packfile; otherwise, return 0.
t/t1016-compatObjectFormat.sh
+97 -3
View File
@@ -45,6 +45,15 @@ compat_hash () {
esac
}
create_compat_repo () {
local hash="$1"
local repo="$2"
git init --object-format="$hash" "$repo" &&
git -C "$repo" config set core.repositoryformatversion 1 &&
git -C "$repo" config set extensions.compatObjectFormat $(compat_hash "$hash")
}
hello_oid () {
case "$1" in
"sha1")
@@ -192,9 +201,7 @@ do
for repo in alt borrow
do
test_when_finished "rm -rf $repo" &&
git init --object-format=$hash $repo &&
git -C $repo config set core.repositoryformatversion 1 &&
git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
create_compat_repo $hash $repo || exit 1
done &&
git -C alt commit --allow-empty --message A &&
@@ -205,6 +212,93 @@ do
git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
test_cmp expect actual
'
test_expect_success 'index-pack writes pack index v3' '
test_when_finished "rm -rf v3-repo v3.idx" &&
create_compat_repo $hash v3-repo &&
echo content >v3-repo/file &&
git -C v3-repo add file &&
git -C v3-repo commit --message A &&
git -C v3-repo commit --allow-empty --message B &&
git -C v3-repo repack -a -d &&
pack=$(echo v3-repo/.git/objects/pack/pack-*.pack) &&
git -C v3-repo index-pack --index-version=3 \
-o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
echo "ff 74 4f 63 00 00 00 03 " >expect &&
test_copy_bytes 8 <v3.idx | test-tool hexdump >actual &&
test_cmp expect actual
'
test_expect_success 'add a large file streamed into a pack' '
test_when_finished "rm -rf large-repo" &&
create_compat_repo $hash large-repo &&
git -C large-repo config set core.bigFileThreshold 1k &&
test-tool genrandom large 5000 >large-repo/large &&
git -C large-repo add large &&
ls large-repo/.git/objects/pack/pack-*.pack &&
git -C large-repo hash-object large >expect &&
git -C large-repo rev-parse :large >actual &&
test_cmp expect actual &&
git -C large-repo cat-file blob :large >actual &&
test_cmp large-repo/large actual
'
test_expect_success 'packed objects are only mapped using pack index v3' '
test_when_finished "rm -rf map-repo v3.idx loose-object-idx" &&
create_compat_repo $hash map-repo &&
echo content >map-repo/file &&
git -C map-repo add file &&
git -C map-repo commit --message A &&
oid=$(git -C map-repo rev-parse HEAD) &&
git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
git -C map-repo repack -a -d &&
pack=$(echo map-repo/.git/objects/pack/pack-*.pack) &&
git -C map-repo index-pack --index-version=3 \
-o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
mv map-repo/.git/objects/loose-object-idx loose-object-idx &&
# A v2 index only has object IDs in the main algorithm, so it
# must not provide a mapping.
git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
! test_cmp expect actual &&
test $(wc -c <actual) -ne $(($(test_oid --hash=$(compat_hash $hash) hexsz) + 1)) &&
# A v3 index provides it.
rm -f "${pack%.pack}.idx" "${pack%.pack}.rev" &&
mv v3.idx "${pack%.pack}.idx" &&
git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
test_cmp expect actual
'
test_expect_success 'abbreviated compat object IDs are found using pack index v3' '
test_when_finished "rm -rf abbrev-repo v3.idx" &&
create_compat_repo $hash abbrev-repo &&
echo content >abbrev-repo/file &&
git -C abbrev-repo add file &&
git -C abbrev-repo commit --message A &&
compat_oid=$(git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) HEAD) &&
abbrev=$(echo $compat_oid | cut -c 1-12) &&
# A multi-pack-index only stores object IDs in the main
# algorithm.
GIT_TEST_MULTI_PACK_INDEX=0 git -C abbrev-repo repack -a -d &&
pack=$(echo abbrev-repo/.git/objects/pack/pack-*.pack) &&
git -C abbrev-repo index-pack --index-version=3 \
-o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
rm abbrev-repo/.git/objects/loose-object-idx &&
# A v2 index only has object IDs in the main algorithm.
test_must_fail git -C abbrev-repo cat-file -t $abbrev &&
rm -f "${pack%.pack}.idx" "${pack%.pack}.rev" &&
mv v3.idx "${pack%.pack}.idx" &&
echo commit >expect &&
git -C abbrev-repo cat-file -t $abbrev >actual &&
test_cmp expect actual &&
echo $compat_oid >expect &&
git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) $abbrev >actual &&
test_cmp expect actual
'
done
cd "$base"
t/t5302-pack-index.sh
+38 -33
View File
@@ -36,50 +36,55 @@ test_expect_success 'setup' '
git update-ref HEAD $commit
'
test_expect_success 'pack-objects with index version 1' '
test_expect_success !COMPAT_HASH 'pack-objects with index version 1' '
pack1=$(git pack-objects --index-version=1 test-1 <obj-list) &&
git verify-pack -v "test-1-${pack1}.pack"
'
test_expect_success 'pack-objects with index version 2' '
test_expect_success !COMPAT_HASH 'pack-objects with index version 2' '
pack2=$(git pack-objects --index-version=2 test-2 <obj-list) &&
git verify-pack -v "test-2-${pack2}.pack"
'
test_expect_success 'both packs should be identical' '
test_expect_success 'pack-objects with index version 3' '
packv3=$(git pack-objects --index-version=3 test-v3 <obj-list) &&
git verify-pack -v "test-v3-${packv3}.pack"
'
test_expect_success !COMPAT_HASH 'both packs should be identical' '
cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
'
test_expect_success 'index v1 and index v2 should be different' '
test_expect_success !COMPAT_HASH 'index v1 and index v2 should be different' '
! cmp "test-1-${pack1}.idx" "test-2-${pack2}.idx"
'
test_expect_success 'index-pack with index version 1' '
test_expect_success !COMPAT_HASH 'index-pack with index version 1' '
git index-pack --index-version=1 -o 1.idx "test-1-${pack1}.pack"
'
test_expect_success 'index-pack with index version 2' '
test_expect_success !COMPAT_HASH 'index-pack with index version 2' '
git index-pack --index-version=2 -o 2.idx "test-1-${pack1}.pack"
'
test_expect_success 'index-pack results should match pack-objects ones' '
test_expect_success !COMPAT_HASH 'index-pack results should match pack-objects ones' '
cmp "test-1-${pack1}.idx" "1.idx" &&
cmp "test-2-${pack2}.idx" "2.idx"
'
test_expect_success 'index-pack --verify on index version 1' '
test_expect_success !COMPAT_HASH 'index-pack --verify on index version 1' '
git index-pack --verify "test-1-${pack1}.pack"
'
test_expect_success 'index-pack --verify on index version 2' '
test_expect_success !COMPAT_HASH 'index-pack --verify on index version 2' '
git index-pack --verify "test-2-${pack2}.pack"
'
test_expect_success 'pack-objects --index-version=2, is not accepted' '
test_expect_success !COMPAT_HASH 'pack-objects --index-version=2, is not accepted' '
test_must_fail git pack-objects --index-version=2, test-3 <obj-list
'
test_expect_success 'index v2: force some 64-bit offsets with pack-objects' '
test_expect_success !COMPAT_HASH 'index v2: force some 64-bit offsets with pack-objects' '
pack3=$(git pack-objects --index-version=2,0x40000 test-3 <obj-list)
'
@@ -91,29 +96,29 @@ else
say "# skipping tests concerning 64-bit offsets"
fi
test_expect_success OFF64_T 'index v2: verify a pack with some 64-bit offsets' '
test_expect_success OFF64_T,!COMPAT_HASH 'index v2: verify a pack with some 64-bit offsets' '
git verify-pack -v "test-3-${pack3}.pack"
'
test_expect_success OFF64_T '64-bit offsets: should be different from previous index v2 results' '
test_expect_success OFF64_T,!COMPAT_HASH '64-bit offsets: should be different from previous index v2 results' '
! cmp "test-2-${pack2}.idx" "test-3-${pack3}.idx"
'
test_expect_success OFF64_T 'index v2: force some 64-bit offsets with index-pack' '
test_expect_success OFF64_T,!COMPAT_HASH 'index v2: force some 64-bit offsets with index-pack' '
git index-pack --index-version=2,0x40000 -o 3.idx "test-1-${pack1}.pack"
'
test_expect_success OFF64_T '64-bit offsets: index-pack result should match pack-objects one' '
test_expect_success OFF64_T,!COMPAT_HASH '64-bit offsets: index-pack result should match pack-objects one' '
cmp "test-3-${pack3}.idx" "3.idx"
'
test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2 (cheat)' '
test_expect_success OFF64_T,!COMPAT_HASH 'index-pack --verify on 64-bit offset v2 (cheat)' '
# This cheats by knowing which lower offset should still be encoded
# in 64-bit representation.
git index-pack --verify --index-version=2,0x40000 "test-3-${pack3}.pack"
'
test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '
test_expect_success OFF64_T,!COMPAT_HASH 'index-pack --verify on 64-bit offset v2' '
git index-pack --verify "test-3-${pack3}.pack"
'
@@ -142,7 +147,7 @@ index_obj_offset()
( read offs extra && echo "$offs" )
}
test_expect_success '[index v1] 1) stream pack to repository' '
test_expect_success !COMPAT_HASH '[index v1] 1) stream pack to repository' '
git index-pack --index-version=1 --stdin < "test-1-${pack1}.pack" &&
git prune-packed &&
git count-objects | ( read nr rest && test "$nr" -eq 1 ) &&
@@ -150,7 +155,7 @@ test_expect_success '[index v1] 1) stream pack to repository' '
cmp "test-1-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v1] 2) create a stealth corruption in a delta base reference' '
# This test assumes file_101 is a delta smaller than 16 bytes.
# It should be against file_100 but we substitute its base for file_099
@@ -167,27 +172,27 @@ test_expect_success \
git cat-file blob $sha1_101 > file_101_foo1
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v1] 3) corrupted delta happily returned wrong data' '
test -f file_101_foo1 && ! cmp file_101 file_101_foo1
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v1] 4) confirm that the pack is actually corrupted' '
test_must_fail git fsck --full $commit
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v1] 5) pack-objects happily reuses corrupted data' '
pack4=$(git pack-objects test-4 <obj-list) &&
test -f "test-4-${pack4}.pack"
'
test_expect_success '[index v1] 6) newly created pack is BAD !' '
test_expect_success !COMPAT_HASH '[index v1] 6) newly created pack is BAD !' '
test_must_fail git verify-pack -v "test-4-${pack4}.pack"
'
test_expect_success '[index v2] 1) stream pack to repository' '
test_expect_success !COMPAT_HASH '[index v2] 1) stream pack to repository' '
rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
git prune-packed &&
@@ -196,7 +201,7 @@ test_expect_success '[index v2] 1) stream pack to repository' '
cmp "test-2-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v2] 2) create a stealth corruption in a delta base reference' '
# This test assumes file_101 is a delta smaller than 16 bytes.
# It should be against file_100 but we substitute its base for file_099
@@ -212,23 +217,23 @@ test_expect_success \
git cat-file blob $sha1_101 > file_101_foo2
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v2] 3) corrupted delta happily returned wrong data' '
test -f file_101_foo2 && ! cmp file_101 file_101_foo2
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v2] 4) confirm that the pack is actually corrupted' '
test_must_fail git fsck --full $commit
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v2] 5) pack-objects refuses to reuse corrupted data' '
test_must_fail git pack-objects test-5 <obj-list &&
test_must_fail git pack-objects --no-reuse-object test-6 <obj-list
'
test_expect_success \
test_expect_success !COMPAT_HASH \
'[index v2] 6) verify-pack detects CRC mismatch' '
rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
@@ -246,12 +251,12 @@ test_expect_success \
test_expect_success 'running index-pack in the object store' '
rm -f .git/objects/pack/* &&
cp test-1-${pack1}.pack .git/objects/pack/pack-${pack1}.pack &&
cp test-v3-${packv3}.pack .git/objects/pack/pack-${packv3}.pack &&
(
cd .git/objects/pack &&
git index-pack pack-${pack1}.pack
git index-pack pack-${packv3}.pack
) &&
test -f .git/objects/pack/pack-${pack1}.idx
test -f .git/objects/pack/pack-${packv3}.idx
'
test_expect_success 'index-pack --strict warns upon missing tagger in tag' '
@@ -273,7 +278,7 @@ EOF
test_grep "^warning:.* expected .tagger. line" err
'
test_expect_success 'index-pack --fsck-objects also warns upon missing tagger in tag' '
test_expect_success !COMPAT_HASH 'index-pack --fsck-objects also warns upon missing tagger in tag' '
git index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&
test_grep "^warning:.* expected .tagger. line" err
'
t/t5308-pack-detect-duplicates.sh
+7
View File
@@ -79,4 +79,11 @@ test_expect_success 'index-pack can reject packs with duplicates' '
test_expect_code 1 git cat-file -e $LO_SHA1
'
test_expect_success 'index-pack can reject packs with duplicates with index v3' '
clear_packs &&
create_pack dups.pack 2 &&
test_must_fail git index-pack --strict --index-version=3 --stdin <dups.pack &&
test_expect_code 1 git cat-file -e $LO_SHA1
'
test_done
t/t5313-pack-bounds-checks.sh
+17
View File
@@ -45,6 +45,14 @@ extended_table () {
echo $(($(ofs_table "$1") + 4*$1))
}
# Offset in a single-format v3 .idx with no extended offsets to the
# start of its order map. $1 is its number of objects, nr.
# The order map (4*nr) is followed by the CRC32 table (4*nr), the
# 4-byte offset table (4*nr) and by the trailer (2*rawsz).
v3_order_map () {
echo $(($(wc -c <"$idx") - 2*$(test_oid rawsz) - 12*$1))
}
test_expect_success 'setup' '
test_oid_cache <<-EOF
oid000 sha1:1485
@@ -171,6 +179,15 @@ test_expect_success 'bogus offset inside v2 extended table' '
test_must_fail git index-pack --verify $pack
'
test_expect_success 'bogus order map entry (v3)' '
do_pack $object --index-version=3 &&
munge $idx $(v3_order_map 1) "\377\377\377\377" &&
clear_base &&
test_must_fail git cat-file blob $object &&
test_must_fail git cat-file --batch-all-objects --batch-check 2>err &&
test_grep "corrupt index" err
'
test_expect_success 'bogus OFS_DELTA in packfile' '
# Generate a pack with a delta in it.
base=$(test-tool genrandom foo 3000 | git hash-object --stdin -w) &&