Replace the snapshots.ErrAlreadyStaged sentinel error with a
read-only-mounts signal: a snapshotter (e.g. erofs serving a layer
content cache hit) now returns Prepare mounts normally, with no
error, when the layer content is already staged into the active
snapshot. The unpacker's isStaged helper checks the last mount's
ReadOnly() to decide whether to skip fetch+apply and just commit.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The cache used to serve a hit by committing the layer during Prepare and
returning ErrAlreadyExists, which is incompatible with parallel unpacking:
the "rebase" capability defers the parent to Commit time, so a
commit-at-Prepare layer ends up parentless. The erofs snapshotter therefore
had to disable rebase whenever the cache was enabled, making every cache
*miss* fall back to a fully sequential (slower-than-cold) pull.
Introduce snapshots.ErrAlreadyStaged, returned from Prepare to mean "the
active snapshot's content is staged; skip the layer download and apply, but
still Commit it" (where the parent is applied). Unlike ErrAlreadyExists it
does not end the layer's lifecycle. The unpacker handles it by emitting a
status whose bottom half runs the normal serialized commit (rebasing the
parent in), and the metadata snapshotter threads it through Prepare like a
normal active snapshot. The erofs cache hit now stages the blob and returns
ErrAlreadyStaged instead of committing, so it can advertise "rebase"
unconditionally: hits skip download+conversion and misses stay parallel.
Also promote the "rebase" capability string to snapshots.RebaseCap (shared by
the unpacker, erofs, and overlay).
Follow up: fetch is range-based, so only a fully cached contiguous
prefix skips downloads; a miss in a lower layer still pulls everything above
it. Per-layer fetch-skip is left as a follow-up.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Disable stat usage since we never consume the data.
This can reduce unnecessary contention during transactions.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
We should reject commit request if target parent is not committed.
Just in case we rebase active snapshot on active one.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Adding a close method allows the mount manager to close any open file
descriptors. The method will also be called automatically by containerd
on shutdown.
Ensure the tests call Close to avoid leaking file descriptors or errors
on Windows cleaning up directories that are in use.
Signed-off-by: Derek McGowan <derek@mcg.dev>
We've long been able to use these and they have a couple
great benefits:
1. Forces you to always access them atomically. With the pointer
variants it's completely valid to access the regular ol' int64/uint32
etc. without using the atomic.* methods. These wrappers don't provide
access to the underlying value so it forces correct usage always.
2. Conveys intent much better. Seeing the type be atomic.Int32 immediately
lets the reader know that this var will be used in a concurrent context,
and we no longer need comments like "this MUST be accessed atomically"
or similar.
Signed-off-by: Danny Canter <danny@dcantah.dev>
Uses the new github.com/containerd/errdefs/pkg module which is intended
to hold less stable utility functions separately from the stable
github.com/containerd/errdefs error types.
Includes temporary update to hcsshim until a release is cut there
Signed-off-by: Derek McGowan <derek@mcg.dev>
Core should not have a dependency on API types.
This was causing a transative dependency on grpc when importing the core
snapshots package.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The ParentIDs array in the Snapshot type is populated in the reverse order i.e the
immediate parent is at the 0th index and the oldest parent is at the last index. It can be
seen here:
https://github.com/containerd/containerd/blob/main/core/snapshots/storage/bolt.go#L492
When applying these layers, the parent layer at the last index should be applied first and
the parent layer at the 0th index should be applied last. However, the comment above the
Snapshot type says the exact opposite thing. This commit fixes that comment.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
Allow the api to stay at the same v1 go package name and keep using a
1.x version number. This indicates the API is still at 1.x and allows
sharing proto types with containerd 1.6 and 1.7 releases.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Maksym Pavlenko