This reverts commit f078cebbd1.
Propagating State deadline errors causes Get to fail and List to omit
tasks when a shim is temporarily unresponsive, such as during slow
cleanup in Delete. A timeout does not mean the task no longer exists,
and omitting it makes container lifecycle management harder for clients
and CRI.
Restore the previous behavior of returning UNKNOWN on State timeouts,
so tasks remain visible and callers can reconcile their state.
Refs: https://github.com/containerd/containerd/pull/14380#pullrequestreview-5689176711
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Commit records a snapshot in the metadata store, which syncs its own
file, while layer.erofs and the snapshot directory may still be in the
page cache. A power loss shortly after a pull can then leave a committed
snapshot whose blob is corrupt or whose directory is missing. A later
pull does nothing because the chain ID exists, and running the image
fails to mount the layer.
With fsync_on_commit = true, Commit fsyncs everything in the snapshot
directory, the directory itself and snapshots/ before the snapshot is
recorded. Blobs served from a layer content cache are not synced. The
option is off by default and lives in the snapshotter because the
differ's SyncFs does not reach Commit, where the walking differ's blob
is created.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Mykhaylo Sul <mykhaylo.sul@oss.qualcomm.com>
Remove the `enable_cdi` configuration option in
`[plugins."io.containerd.cri.v1.runtime"]`, which was deprecated in
v2.2. CDI support is now always enabled.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
The mount handler built the device name from the source path, taking the
base of the parent directory as a snapshot id. That reads an identity
out of a layout only a snapshot directory has. A layer served from a
layer content cache is mounted from the cache, whose parent directory is
a shard named for the first two characters of the digest. Every blob in
a shard then names one device.
A second layer in a shard fails to mount. dmverity.Open finds the name
taken, VerifyDevice compares the existing device's root hash against the
one being mounted, and the mismatch is an error. No layer is served
against the wrong root hash, and none of the colliding layers can be
mounted.
Name the device for the source and the mount point together. Mount
closes the device it creates when the mount fails, so the name has to be
unique to the mount. Neither half is unique alone: a cache serves one
blob to every snapshot of that layer, and a mount point is reused once
the mount before it is gone. A device left behind by a crash is reused
only for the same blob in the same place, where its root hash still
matches. Unmount reads the name back from the mounted source and needs
nothing from either.
Signed-off-by: Derek McGowan <derek@mcg.dev>
A layer served from the layer content cache was symlinked into the
snapshot as its layer.erofs. That put a path inside the snapshot
directory aliasing a blob owned by the operator and shared with every
other snapshot of that layer. Everything written to that path went
straight into the cache entry. Each write site then needed its own
guard: the differ had to refuse to apply, Commit had to refuse to
convert through a dangling link, and Remove had to avoid clearing
IMMUTABLE_FL on a file it did not own. A missed guard corrupted content
shared across images. A pruned entry left a dangling link that read as
an absent blob on one path and as populated content on another.
Record where the blob is on the snapshot and mount it from there.
Nothing in the snapshot directory refers to content it does not own. A
stray write creates a local file, conversion cannot reach the cache, and
removal has nothing to follow. The guards Commit and Remove needed for a
symlink are gone. The differ still refuses, on the read-only mounts the
snapshotter hands out.
resolveBlob is the one place a blob is located. It reports whether the
blob is already populated, meaning there is nothing to apply or convert,
and whether it belongs to this snapshot, meaning it may be written.
Mounts are then assembled without consulting the filesystem to find out
what a layer is. A source that does not resolve is an error on every
path, including the parentless path in mounts(). A flattened parent's
device= is resolved the same way.
fsverity is only measured for a blob the snapshot owns. Measuring a
cache entry would require the operator to enable fsverity on content
this snapshotter never wrote. NewSnapshotter already rejects fsverity
together with the layer content cache.
The record is kept in labels outside the containerd.io/snapshot/
namespace. Labels in that namespace are inherited from image
annotations, which would let an image name any path on the host. The
same filter runs on everything the metadata snapshotter passes down. A
client cannot set these labels, and cannot send them back after reading
them through Stat. Commit and Update carry them across.
The dm-verity sidecar is read from beside the cache entry. Nothing is
copied into the snapshot. The sidecar cannot fall out of step with the
blob it pins. An entry required to have one is rejected at Prepare.
Signed-off-by: Derek McGowan <derek@mcg.dev>
The differ found an already-populated snapshot by looking for a
symlinked layer.erofs in the layer directory. That check needs a
particular on-disk layout, and it needs erofsutils.MountsToLayer to
infer the snapshot directory from a mount source that lives inside it.
Neither holds for a snapshot whose content is not a local file.
Refuse on the signal the unpacker already uses. Mounts handed to Apply
that are read-only belong to a snapshot the snapshotter populated
itself, whose content is shared with every other snapshot of that layer.
This does not need path inference, covers parented and parentless
snapshots alike, and stays correct for any content the snapshotter does
not store as a plain file in the snapshot directory.
The error stays ErrFailedPrecondition. plugins/services/diff/local.go
falls through to the next differ on ErrNotImplemented. Another differ
would fail on this layer for the same reason.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Staging a cached layer blob was gated on the Prepare having no parent,
which sequential unpacking only does for the first layer, so a warm
cache missed on every layer but layer 0 and a pull ended up slower than
with no cache at all.
Serve any image-layer extraction, and mount a staged blob read-only
whatever its parents, so the differ is never handed a writable view of a
blob shared with every other snapshot of that layer. A read-only mount
is a report to the caller and not an enforcement, so the erofs differ
also refuses a layer whose blob is a symlink, which would otherwise
truncate the cache entry through it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The UpdateSandboxRequest and UpdateSandboxResponse messages have been
present since the Sandbox API was introduced, but no RPC ever used them,
and controllerLocal.Update was an unconditional no-op. A shim therefore
had no way to learn about an update of the sandbox metadata object, a CRI
pod level resource resize for example.
Add the missing UpdateSandbox RPC and extend UpdateSandboxRequest with the
sandbox object and the updated fieldpaths, mirroring the daemon side
ControllerUpdateRequest, which is what Controller.Update carries. The
pre-existing resources and annotations fields are kept but marked
deprecated, containerd has no input to populate them from.
controllerLocal.Update now calls the new RPC. Implementing it is optional:
shims that don't support updates answer with a ttrpc Unimplemented error,
which maps to errdefs.ErrNotImplemented and is already tolerated by the
CRI caller in UpdatePodSandboxResources.
next.txtpb additionally picks up unrelated churn, the committed descriptor
set does not match what the pinned buf version produces.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The preceding commit advertises the default platform with the erofs OS feature. Move construction of that platform set into snapshotterPlatforms so the client-side matching contract can be tested without changing the advertisement.
EROFS identifies the image layer package format rather than a Linux-only host capability, so the feature-bearing entry remains derived from platforms.DefaultSpec. Test that the advertised set matches both the plain default platform and the same platform carrying the erofs feature.
Assisted-by: Claude Opus 5
Signed-off-by: Martín Fernández <mfernandez@crunchloop.io>
Advertise a second platform - the default platform spec with OSFeatures
["erofs"] - from the EROFS snapshotter plugin, alongside the plain
default platform it already advertised. This mirrors what the EROFS
differ plugin already advertises.
This is required by client.Image.Unpack's checkSnapshotterSupport, which
rejects unpacking an image whose config declares os.features onto a
snapshotter that doesn't advertise a matching platform: without this,
pulling and unpacking an EROFS image layer format specification image
(https://github.com/erofs/erofs-image-spec) - which requires "erofs" in
os.features - with the erofs snapshotter would fail that check.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Signed-off-by: Martín Fernández <mfernandez@crunchloop.io>
There was no way to tell whether a cache was doing anything: hits and
misses were only visible as trace log lines. Count lookups by result,
hits by the directory that served them so caches with different lifetimes
can be told apart, the blob bytes served, and lookup errors by reason,
which also makes an unreadable cache (a down FUSE mount, say) visible
instead of looking like a cold one.
Also add a span around the differ's Apply, carrying the layer descriptor
and which path handled it. The unpacker traces the fetch and the layer
lifecycle but not the apply, so a slow layer couldn't be pinned on the
download or the conversion.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
A new-* directory can exist before createSnapshot acquires the metadata
writer transaction. A concurrent Remove performs a global orphan scan and
can delete the staging directory because it is not in the ID map yet.
Exclude new-* staging directories from orphan cleanup and use a shared
local constant for directory creation and filtering.
Signed-off-by: Chris Ayoub <cayoub@openai.com>
The layer content cache was a single directory, so every source of
pre-converted blobs had to be merged into one tree. A shared read-only fleet
cache, a host-local cache, and a per-image warm cache could not coexist.
Replace layer_content_cache with layer_content_caches, a list. Each directory
is checked in order and the first hit is staged into the snapshot; a layer
found in none of them falls back to the normal download-and-convert path.
Cache directories are no longer required to exist at startup. A missing one is
indistinguishable from an empty one at lookup time (both are simply a miss),
and it may well be mounted after the daemon starts, so the only check left is
that each path is absolute -- a relative one would be symlinked into the
snapshot dir and dangle.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Replace the snapshots.ErrAlreadyStaged sentinel error with a
read-only-mounts signal: a snapshotter (e.g. erofs serving a layer
content cache hit) now returns Prepare mounts normally, with no
error, when the layer content is already staged into the active
snapshot. The unpacker's isStaged helper checks the last mount's
ReadOnly() to decide whether to skip fetch+apply and just commit.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The cache used to serve a hit by committing the layer during Prepare and
returning ErrAlreadyExists, which is incompatible with parallel unpacking:
the "rebase" capability defers the parent to Commit time, so a
commit-at-Prepare layer ends up parentless. The erofs snapshotter therefore
had to disable rebase whenever the cache was enabled, making every cache
*miss* fall back to a fully sequential (slower-than-cold) pull.
Introduce snapshots.ErrAlreadyStaged, returned from Prepare to mean "the
active snapshot's content is staged; skip the layer download and apply, but
still Commit it" (where the parent is applied). Unlike ErrAlreadyExists it
does not end the layer's lifecycle. The unpacker handles it by emitting a
status whose bottom half runs the normal serialized commit (rebasing the
parent in), and the metadata snapshotter threads it through Prepare like a
normal active snapshot. The erofs cache hit now stages the blob and returns
ErrAlreadyStaged instead of committing, so it can advertise "rebase"
unconditionally: hits skip download+conversion and misses stay parallel.
Also promote the "rebase" capability string to snapshots.RebaseCap (shared by
the unpacker, erofs, and overlay).
Follow up: fetch is range-based, so only a fully cached contiguous
prefix skips downloads; a miss in a lower layer still pulls everything above
it. Per-layer fetch-skip is left as a follow-up.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
mounts() moved `first` to the merged fsmeta mount's index whenever
mountFsMeta matched, collapsing the overlay's lowerdir range to the
fsmeta alone and dropping any plain lowers already appended above it.
When fsmerge was added, fsmeta was only ever written for the top
parent of the chain being prepared, so the loop matched at i == 0,
where first already equalled the fsmeta's index and the reassignment
was a no-op. It only mattered once the fsmeta could sit below the top
parent, i.e. a chain extending an already-merged sub-chain. The
snapshotter no longer generates fsmeta itself, so today this only
affects externally supplied fsmeta files.
first only marks the start of the lowerdir range, so it must stay at
the first lower's index.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
NewSnapshotter auto-appends "index=off" unless the configured
mount_options already carry an index option — but the check used
hasOption(..., "index", false), which compares against the literal
string "index". No valid mount option ever equals "index" (it is
always valued, "index=on"/"index=off"), so the append happened
unconditionally whenever the kernel exposes the overlay index
parameter.
Mount options are last-wins in the kernel, so a user-configured
"index=on" was silently overridden. Worse, configuring
mount_options = ["index=on", "nfs_export=on"] (the documented way to
make overlay mounts NFS-exportable) produced
"index=on,nfs_export=on,index=off", which the kernel rejects with
EINVAL (nfs_export=on conflicts with an explicit index=off) — breaking
every snapshot mount including image unpack.
Make hasOption match the key of a "key[=value]" option so callers need
not care whether the option takes a value, and add a regression test.
Signed-off-by: AprilNEA <github@sku.moe>
Disable stat usage since we never consume the data.
This can reduce unnecessary contention during transactions.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Perform an early validation check on both container checkpoint and
restore paths to fail-fast if the CRIU binary is missing or is older
than the minimum required version (3.16.0).
To support runtime-configured environments, the validation respects the
custom PATH from the shim manager environment if configured, skipping
any relative paths to avoid incorrect daemon-relative resolution. If not
configured, it falls back to a standard system PATH lookup. The check
result is cached using sync.Once to prevent redundant process spawning.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Switch block CIM snapshotter and diff plugin init checks from
IsBlockCimSupported() to IsBlockCimWriteSupported() to allow plugins
to load when the CimWriter.dll is available, even on OS builds
without native block CIM support.
Signed-off-by: Maksim An <maksiman@microsoft.com>
This change updates an incorrect default config value for the block CIM
snapshotter. The `ntfsFormat` should always be false for the block CIM
snapshotter. If the scratch needs to be formatted with NTFS, a format
option named `WithNTFSFormat` can be provided.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
dmsetup.Status() returns a nil DeviceStatus when it fails. The mkfs
failure path records that error, but still unconditionally dereferences
status when logging the pool status.
Avoid the panic by logging a fallback status string when dmsetup.Status()
does not return a status.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Signed-off-by: Mikhail Dmitrichenko <m.dmitrichenko222@gmail.com>
On Windows, os.RemoveAll and os.ReadFile can fail with a sharing
violation when a file handle has not been fully released by the OS.
Unlike Unix, Windows does not allow deleting or reading a file
while any handle is open, and handles may be held briefly after
Close returns. This causes a race between the gRPC server-side
writer Close (which writes updatedat.tmp) and both Abort (which
calls RemoveAll on the ingest directory) and status (which reads
the updatedat file), surfacing as a flaky
TestContentClient/CommitErrorState on Windows CI.
Add platform-specific removePath and readFileWithRetry helpers
that retry with incremental backoff on Windows. On non-Windows
platforms they pass through to os.RemoveAll and os.ReadFile
directly. This follows the same pattern as ensureRemoveAll in
the CRI server helpers, which retries on EBUSY for mount races
on Linux.
Signed-off-by: ayush-panta <ayushkp@amazon.com>
Wei Fu