The snapshot garbage collector holds the snapshotter write lock while
calling the underlying Snapshotter.Remove. cleanupSnapshotter also
detaches cancellation (context.WithoutCancel), so a Remove that never
returns - e.g. a proxy snapshotter whose RPC connection has wedged -
holds the write lock forever. Every subsequent snapshot operation on
the node then blocks behind the lock, including CRI RunPodSandbox,
turning the node into a permanent scheduling black hole while it still
reports Ready.
Observed in production with a proxy snapshotter (soci) that stopped
answering after a file-descriptor exhaustion storm: the GC goroutine
sat in WaitGroup.Wait for 4.5 hours with the Remove RPC parked in
waitOnHeader, and 17 CRI operations queued behind the metadata lock.
Bound each GC-path Remove with a timeout registered via pkg/timeout
(io.containerd.timeout.gc.snapshotter.remove, default 30m,
operator-tunable through config.toml like the other timeout keys). A
timed-out Remove returns DeadlineExceeded, which is not a
FailedPrecondition, so it propagates out of pruneBranch and ends the
pass: a globally wedged snapshotter costs one timeout per GC cycle
rather than one per orphaned snapshot, the deferred Cleanup is skipped
because the pass failed, and the snapshots left behind stay orphaned
for the next pass.
Signed-off-by: Nahum Litvin <nahuml@wix.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Extend the garbage-collection framework so a collectible resource can emit
forward references during graph traversal, in addition to the existing
back-reference mechanism.
A CollectionContext may now implement the optional collectionWithReferences
interface:
References(ctx context.Context, node gc.Node, fn func(gc.Node))
When the GC visits a node whose resource type was registered by an external
collector, gcContext.references consults the per-type References
implementation after the built-in core resource types are handled.
This is the forward-reference analogue of collectionWithBackRefs. Whereas
ActiveWithBackRefs must enumerate every edge up front and the gcContext
holds all of them in its backRefs map for the entire collection, References
is invoked on demand for a single node. A collector whose resources fan
out to many other nodes can therefore emit those edges without retaining
them in memory for the gc context.
This commit is intentionally a no-op: no plugin registers a collector that
uses collectionWithReferences yet. It is isolated here so that concurrent
development efforts that depend on this interface can be proposed and
reviewed upstream independently.
Signed-off-by: Derek McGowan <derek@mcg.dev>
TestImagesCreateUpdateDelete asserts that an image's updatedat is
strictly after its createdat. Both timestamps are stamped via
time.Now().UTC(), which strips the monotonic reading, so the comparison
falls back to the wall clock. On platforms with coarse timer resolution
(e.g. Windows, which advances system time at the ~15.6ms tick), the
Create and Update calls can land in the same tick and produce identical
timestamps, making the strict After() check fail intermittently.
Wait for the wall clock to advance past the creation timestamp before
updating so the assertion stays meaningful without depending on clock
resolution. On fine-resolution clocks the loop runs zero iterations.
Signed-off-by: Austin Vazquez <austin.vazquez@docker.com>
Conditional gc references allows establishing a conditional reference,
which can be used for expiration of specific connections without needing
to updated multiple objects.
For example, content can hold a temporary relationship to a snapshot
that can expire if the snapshot is unused after a specific time. This
allows the just updating the snapshot label when it is used without
needing to update other objects or create an expiring lease to hold the
connection.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This PR adds opt-in tracing spans/attributes in CRI image pull and selected sandbox-related paths to improve debugging and correlation (e.g., sandbox.id/pod metadata). If maintainers prefer a smaller diff, I’m happy to split this into a pull-only PR plus follow-ups.
• follow-up after pull-only PR
• focuses on task/metadata/sandbox/cni setup spans
Signed-off-by: Cindy Li <cindyli@pinterest.com>
Error strings should not be capitalized (unless beginning with proper
nouns or acronyms) or end with punctuation, per Go style guide and
staticcheck ST1005.
This change lowercases the first letter of error messages in fmt.Errorf
calls throughout the codebase, while preserving uppercase for acronyms
at the start of error strings (e.g., NRI, RDT, CDI, UID, GID, VHD, FUSE).
Also fixes typo 'faild' -> 'failed' in resolver_test.go.
Changes:
- core/metadata: target.Digest, target.Size, target.MediaType
- core/remotes/docker: fetcher.Seek, fetcherByDigest
- integration: various test error messages
- internal/cri/server: containerConfig
- plugins/snapshots: erofs
Fixes#12011
Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
Allows an object to create a backreference, allowing objects to be
referred to by objects which already exist without updating the labels
on the original object. This is useful for ephemeral objects as well as
objects with a 1 to many relationship or created after another object.
Signed-off-by: Derek McGowan <derek@mcg.dev>
the `cleanup.Background` utility was introduced in f606c4eba7,
at which time the project used go1.19, and Go's stdlib context did not yet
have [`context.WithoutCancel`], which was introduced in go1.21.
This patch replaces `cleanup.Background` for `context.WithoutCancel`, which
is near-identical, and part of go stdlib;
`cleanup.Background`:
type clearCancel struct {
context.Context
}
func (cc clearCancel) Deadline() (deadline time.Time, ok bool) {
return
}
func (cc clearCancel) Done() <-chan struct{} {
return nil
}
func (cc clearCancel) Err() error {
return nil
}
// Background creates a new context which clears out the parent errors
func Background(ctx context.Context) context.Context {
return clearCancel{ctx}
}
`context.WithoutCancel`:
// WithoutCancel returns a derived context that points to the parent context
// and is not canceled when parent is canceled.
// The returned context returns no Deadline or Err, and its Done channel is nil.
// Calling [Cause] on the returned context returns nil.
func WithoutCancel(parent Context) Context {
if parent == nil {
panic("cannot create context from nil parent")
}
return withoutCancelCtx{parent}
}
type withoutCancelCtx struct {
c Context
}
func (withoutCancelCtx) Deadline() (deadline time.Time, ok bool) {
return
}
func (withoutCancelCtx) Done() <-chan struct{} {
return nil
}
func (withoutCancelCtx) Err() error {
return nil
}
func (c withoutCancelCtx) Value(key any) any {
return value(c, key)
}
func (c withoutCancelCtx) String() string {
return contextName(c.c) + ".WithoutCancel"
}
[`context.WithoutCancel`]: https://pkg.go.dev/context@go1.21.0#WithoutCancel
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
We've long been able to use these and they have a couple
great benefits:
1. Forces you to always access them atomically. With the pointer
variants it's completely valid to access the regular ol' int64/uint32
etc. without using the atomic.* methods. These wrappers don't provide
access to the underlying value so it forces correct usage always.
2. Conveys intent much better. Seeing the type be atomic.Int32 immediately
lets the reader know that this var will be used in a concurrent context,
and we no longer need comments like "this MUST be accessed atomically"
or similar.
Signed-off-by: Danny Canter <danny@dcantah.dev>
Other similar events were already moved to the metadata store. The
metadata store has more information that can be used for a future
content created event.
Signed-off-by: Derek McGowan <derek@mcg.dev>
The boltdb instance in metadata is only used for getting transactions
and can also be overriden via the context to have a wider control of the
transaction boundary. Using the transactor interface allows callers of
metadata to have more control of the transaction lifecycle.
Since boltdb must be fsync'ed on commit, operations which perform many
database operations can be costly and slow. While providing transactor
via context can be used to group together operations, it does not
provide a way to manage the commit fsyncs more globally.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Currently the metadata snapshotter is not consistently adding keys to a
lease when already exists is returned. When a lease is provided, any
already exists errors should add the relevant key to the lease. It is
not expected that clients must explicitly lease a key after calling
Prepare/Commit.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Allow the api to stay at the same v1 go package name and keep using a
1.x version number. This indicates the API is still at 1.x and allows
sharing proto types with containerd 1.6 and 1.7 releases.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Nahum Litvin