When a layer fails to prepare during parallel unpack we break out of the
launch loop but never return the error, so unpack() can report success and
label the image with a chainID that was never created. Keep the error and
return it once the already queued layers have been drained, so those still
commit as they would in sequential mode. Also end the layer's tracing span,
which leaked on this path.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Replace the snapshots.ErrAlreadyStaged sentinel error with a
read-only-mounts signal: a snapshotter (e.g. erofs serving a layer
content cache hit) now returns Prepare mounts normally, with no
error, when the layer content is already staged into the active
snapshot. The unpacker's isStaged helper checks the last mount's
ReadOnly() to decide whether to skip fetch+apply and just commit.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The cache used to serve a hit by committing the layer during Prepare and
returning ErrAlreadyExists, which is incompatible with parallel unpacking:
the "rebase" capability defers the parent to Commit time, so a
commit-at-Prepare layer ends up parentless. The erofs snapshotter therefore
had to disable rebase whenever the cache was enabled, making every cache
*miss* fall back to a fully sequential (slower-than-cold) pull.
Introduce snapshots.ErrAlreadyStaged, returned from Prepare to mean "the
active snapshot's content is staged; skip the layer download and apply, but
still Commit it" (where the parent is applied). Unlike ErrAlreadyExists it
does not end the layer's lifecycle. The unpacker handles it by emitting a
status whose bottom half runs the normal serialized commit (rebasing the
parent in), and the metadata snapshotter threads it through Prepare like a
normal active snapshot. The erofs cache hit now stages the blob and returns
ErrAlreadyStaged instead of committing, so it can advertise "rebase"
unconditionally: hits skip download+conversion and misses stay parallel.
Also promote the "rebase" capability string to snapshots.RebaseCap (shared by
the unpacker, erofs, and overlay).
Follow up: fetch is range-based, so only a fully cached contiguous
prefix skips downloads; a miss in a lower layer still pulls everything above
it. Per-layer fetch-skip is left as a follow-up.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
This PR adds opt-in tracing spans/attributes in CRI image pull and selected sandbox-related paths to improve debugging and correlation (e.g., sandbox.id/pod metadata). If maintainers prefer a smaller diff, I’m happy to split this into a pull-only PR plus follow-ups.
• follow-up after pull-only PR
• focuses on task/metadata/sandbox/cni setup spans
Signed-off-by: Cindy Li <cindyli@pinterest.com>
Avoid using internal interfaces for exported types and always prefer
using interfaces over structs to limit unused imports.
Signed-off-by: Derek McGowan <derek@mcg.dev>
We've long been able to use these and they have a couple
great benefits:
1. Forces you to always access them atomically. With the pointer
variants it's completely valid to access the regular ol' int64/uint32
etc. without using the atomic.* methods. These wrappers don't provide
access to the underlying value so it forces correct usage always.
2. Conveys intent much better. Seeing the type be atomic.Int32 immediately
lets the reader know that this var will be used in a concurrent context,
and we no longer need comments like "this MUST be accessed atomically"
or similar.
Signed-off-by: Danny Canter <danny@dcantah.dev>
This PR optimizes the chain ID calculation in unpack so we
only (pre-)calculate the chain ID for layer_i exactly once,
by calling `identity.ChainIDs(diffIDs)`.
Currently in `unpack` for every layer_i, we calculate the chain id
of layer_i-1 and layer_i *repeatedly*. Because each
`identity.ChainID(diffIDs)` call involves:
1. Copy `diffIDs` to a new slice
2. Calculate chain ID for every index (by calling `ChainIDs(diffIDs)`)
3. Return the final one as result of `identity.ChainID(diffIDs)`.
This means, given an image with N layers, for every layer_i:
1. we create 2 new slices and copy the diffIDs sofar;
2. we're recalculating all chain IDs from layer_0 to layer_i-1 twice;
3. in total, the chain ID for layer_i is calculated 2 * (N - layer_i) times;
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
When an upstream client (e.g. kubelet) stops or restarts, the CRI
connection to the containerd gets interrupted which is treated as a
cancellation of context which subsequently cancels an ongoing operation,
including an image pull. This generally gets followed by containerd's
GC routine that tries to delete the prepared snapshots for the image
layer(s) corresponding to the image in the pull operation that got
cancelled. However, if the upstream client immediately retries (or
starts a new) image pull operation, containerd initiates a new image
pull and starts unpacking the image layers into snapshots. This may
create a race condition: the GC routine (corresponding to the failed
image pull operation) trying to clean up the same snapshot that the new
image pull operation is preparing, thus leading to the "parent snapshot
does not exist: not found" error.
Race Condition Scenario:
Assume an image consisting of 2 layers (L1 and L2, L1 being the bottom
layer) that are supposed to get unpacked into snapshots S1 and S2
respectively.
During an image pull operation, containerd unpacks(L1) which involves
Stat()'ing the chainID. This Stat() fails as the chainID does not
exist and Prepare(L1) gets called. Once S1 gets prepared, containerd
processes L2 - unpack(L2) which again involves Stat()'ing the chainID
which fails as the chainID for S2 does not exist which results in the
call to Prepare(L2). However, if the image pull operation gets
cancelled before Prepare(L2) is called, then the GC routine tries to
clean up S1.
When the image pull operation is retried by the upstream client,
containerd follows the same series of operations. unpack(L1) gets
called which then calls Stat(chainID) for L1. However, this time,
Stat(L1) succedes as S1 already exists (from the previous image pull
operation) and thus containerd goes to the next iteration to
unpack(L2). Now, GC cleans up S1 and when Prepare(L2) gets called, it
returns back the "parent snapshot does not exist: not found" error.
Fix:
Removing the "Stat() + early return" fixes the race condition. Now
during the image pull operation corresponding to the client retry,
although the chainID (for L1) already exists, containerd does not
return early and goes on to Prepare(L1). Since L1 is already prepared,
it adds a new lease to S1 and then returns `ErrAlreadyExists`. This
new lease prevents GC from cleaning up S1 when containerd processes
L2 (unpack(L2) -> Prepare(L2)).
Fixes: https://github.com/containerd/containerd/issues/3787
Signed-off-by: Saket Jajoo <saketjajoo@google.com>
When a set of layers are provided to the unpacker, then the unpacker
should still fetch them regardless of whether they will be used for
unpack. The image handler filters are responsible for removing content
which is not intended to be fetched. Currently there is no way to use an
unpacker and also fetch all platforms.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Packages related to transfer and unpacking provide core interfaces which
use other core interfaces and part of common functionality.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Maksym Pavlenko