The layer content cache was a single directory, so every source of
pre-converted blobs had to be merged into one tree. A shared read-only fleet
cache, a host-local cache, and a per-image warm cache could not coexist.
Replace layer_content_cache with layer_content_caches, a list. Each directory
is checked in order and the first hit is staged into the snapshot; a layer
found in none of them falls back to the normal download-and-convert path.
Cache directories are no longer required to exist at startup. A missing one is
indistinguishable from an empty one at lookup time (both are simply a miss),
and it may well be mounted after the daemon starts, so the only check left is
that each path is absolute -- a relative one would be symlinked into the
snapshot dir and dangle.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Replace the snapshots.ErrAlreadyStaged sentinel error with a
read-only-mounts signal: a snapshotter (e.g. erofs serving a layer
content cache hit) now returns Prepare mounts normally, with no
error, when the layer content is already staged into the active
snapshot. The unpacker's isStaged helper checks the last mount's
ReadOnly() to decide whether to skip fetch+apply and just commit.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The cache used to serve a hit by committing the layer during Prepare and
returning ErrAlreadyExists, which is incompatible with parallel unpacking:
the "rebase" capability defers the parent to Commit time, so a
commit-at-Prepare layer ends up parentless. The erofs snapshotter therefore
had to disable rebase whenever the cache was enabled, making every cache
*miss* fall back to a fully sequential (slower-than-cold) pull.
Introduce snapshots.ErrAlreadyStaged, returned from Prepare to mean "the
active snapshot's content is staged; skip the layer download and apply, but
still Commit it" (where the parent is applied). Unlike ErrAlreadyExists it
does not end the layer's lifecycle. The unpacker handles it by emitting a
status whose bottom half runs the normal serialized commit (rebasing the
parent in), and the metadata snapshotter threads it through Prepare like a
normal active snapshot. The erofs cache hit now stages the blob and returns
ErrAlreadyStaged instead of committing, so it can advertise "rebase"
unconditionally: hits skip download+conversion and misses stay parallel.
Also promote the "rebase" capability string to snapshots.RebaseCap (shared by
the unpacker, erofs, and overlay).
Follow up: fetch is range-based, so only a fully cached contiguous
prefix skips downloads; a miss in a lower layer still pulls everything above
it. Per-layer fetch-skip is left as a follow-up.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
mounts() moved `first` to the merged fsmeta mount's index whenever
mountFsMeta matched, collapsing the overlay's lowerdir range to the
fsmeta alone and dropping any plain lowers already appended above it.
When fsmerge was added, fsmeta was only ever written for the top
parent of the chain being prepared, so the loop matched at i == 0,
where first already equalled the fsmeta's index and the reassignment
was a no-op. It only mattered once the fsmeta could sit below the top
parent, i.e. a chain extending an already-merged sub-chain. The
snapshotter no longer generates fsmeta itself, so today this only
affects externally supplied fsmeta files.
first only marks the start of the lowerdir range, so it must stay at
the first lower's index.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
NewSnapshotter auto-appends "index=off" unless the configured
mount_options already carry an index option — but the check used
hasOption(..., "index", false), which compares against the literal
string "index". No valid mount option ever equals "index" (it is
always valued, "index=on"/"index=off"), so the append happened
unconditionally whenever the kernel exposes the overlay index
parameter.
Mount options are last-wins in the kernel, so a user-configured
"index=on" was silently overridden. Worse, configuring
mount_options = ["index=on", "nfs_export=on"] (the documented way to
make overlay mounts NFS-exportable) produced
"index=on,nfs_export=on,index=off", which the kernel rejects with
EINVAL (nfs_export=on conflicts with an explicit index=off) — breaking
every snapshot mount including image unpack.
Make hasOption match the key of a "key[=value]" option so callers need
not care whether the option takes a value, and add a regression test.
Signed-off-by: AprilNEA <github@sku.moe>
Disable stat usage since we never consume the data.
This can reduce unnecessary contention during transactions.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Perform an early validation check on both container checkpoint and
restore paths to fail-fast if the CRIU binary is missing or is older
than the minimum required version (3.16.0).
To support runtime-configured environments, the validation respects the
custom PATH from the shim manager environment if configured, skipping
any relative paths to avoid incorrect daemon-relative resolution. If not
configured, it falls back to a standard system PATH lookup. The check
result is cached using sync.Once to prevent redundant process spawning.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Switch block CIM snapshotter and diff plugin init checks from
IsBlockCimSupported() to IsBlockCimWriteSupported() to allow plugins
to load when the CimWriter.dll is available, even on OS builds
without native block CIM support.
Signed-off-by: Maksim An <maksiman@microsoft.com>
This change updates an incorrect default config value for the block CIM
snapshotter. The `ntfsFormat` should always be false for the block CIM
snapshotter. If the scratch needs to be formatted with NTFS, a format
option named `WithNTFSFormat` can be provided.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
dmsetup.Status() returns a nil DeviceStatus when it fails. The mkfs
failure path records that error, but still unconditionally dereferences
status when logging the pool status.
Avoid the panic by logging a fallback status string when dmsetup.Status()
does not return a status.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Signed-off-by: Mikhail Dmitrichenko <m.dmitrichenko222@gmail.com>
On Windows, os.RemoveAll and os.ReadFile can fail with a sharing
violation when a file handle has not been fully released by the OS.
Unlike Unix, Windows does not allow deleting or reading a file
while any handle is open, and handles may be held briefly after
Close returns. This causes a race between the gRPC server-side
writer Close (which writes updatedat.tmp) and both Abort (which
calls RemoveAll on the ingest directory) and status (which reads
the updatedat file), surfacing as a flaky
TestContentClient/CommitErrorState on Windows CI.
Add platform-specific removePath and readFileWithRetry helpers
that retry with incremental backoff on Windows. On non-Windows
platforms they pass through to os.RemoveAll and os.ReadFile
directly. This follows the same pattern as ensureRemoveAll in
the CRI server helpers, which retries on EBUSY for mount races
on Linux.
Signed-off-by: ayush-panta <ayushkp@amazon.com>
Fix issue 13388
```
[...]
May 12 16:57:23 kind-control-plane kubelet[257]: failed to extract layer (application/vnd.docker.image.rootfs
.diff.tar sha256:6f1cdceb6a3146f0ccb986521156bef8a422cdbb0863396f7f751f575ba308f4) to overlayfs as "extract-920875437
-7QPF sha256:31e64620332e54e3e4fb246d8325ed2c9f1c2cc64a95f0bb23b4b7e82834c95a": failed to mount /var/lib/containerd/t
mpmounts/containerd-mount2180142388: mount source: "overlay", target: "/var/lib/containerd/tmpmounts/containerd-mount
2180142388", fstype: overlay, flags: 0, data: "upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/sn
apshots/279/fs", err: invalid argument
[...]
```
This was a regression introduced in PR 13115.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The grpc, grpc-tcp, and ttrpc server plugins enumerated their services
through ic.GetByType, which short-circuits on the first plugin whose
Instance() returned an error. A single failed gRPC plugin (e.g. CRI
under rootless, which cannot watch /etc/cni/net.d) therefore prevented
the server plugins from initialising, leaving /run/containerd/containerd.sock
uncreated.
Iterate the plugin set directly and skip plugins that failed to
initialise, restoring the pre-c15ec2485 behaviour where the listener
is still created and only the failed services are missing.
Fixes: c15ec2485 ("Add server plugins for grpc and ttrpc")
Fixes: https://github.com/containerd/containerd/issues/13362
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Currently the metadata may be generated after a snapshot is committed,
causing a difference in the overlayfs mount that may cause ESTALE errors.
The committed snapshot must be immutable and should always return the
same set of mounts after commit, even if configuration is changed.
This setting may be added back later to be performed before commit,
either as part of the unpack or as a step before commit. This is getting
pulled out from the upcoming release.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Update the EROFS snapshotter to pass the dm-verity metadata path
through the `X-containerd.dmverity` mount option. Previously, the
system relied on implicit mode detection; providing the explicit
path allows downstream components (such as Kata Containers mount
handlers) to reliably locate and read `.dmverity` files.
Correspondingly, update the EROFS mount handler to parse and utilize
this explicit metadata path for dm-verity device initialization.
This enables the runtime to retrieve root hashes and other necessary
metadata directly, ensuring robust layer integrity verification.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The gRPC sandbox controller service only forwarded the `options` field
when calling the local controller. The `netns_path`, `rootfs`, and
`annotations` fields were silently dropped, causing clients using the
gRPC proxy path to receive incomplete sandbox configurations.
Event topics were missing the leading `/` prefix ("sandboxes/create"
instead of "/sandboxes/create"), causing the event exchange to reject
the publish and return an error to the caller.
Add unit tests for the controller service that exercise all RPC
methods.
Signed-off-by: William Myers <willmyrs@amazon.com>
The new server plugin type allows services which have listeners to be
configured as plugins rather than defined directly in the global
configuration. This provides more configuration consistency and allows
containerd to be extended for new types of api handlers.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This is all specific to the content store -- prior to this change, `ctr content fetch docker.io/tianon/test:sha512-blobs` fails with errors about `sha256` digests, and after this change it succeeds and ingests the `sha512` blobs successfully.
In the interest of full disclosure, this change was mostly authored by myself without any AI-assistance, but Claude was used to find the appropriate tests to update and to understand why the metadata service would clear out `Digest` when passing details to the content store and thus the appropriate fix being in the Writer's Commit method (re-hashing after Close when the digest algorithm was previously unknown but is known and non-canonical during Commit). I had a very active hand in instructing Claude on exactly which changes to make and where to make them, and I spent a lot of time reviewing them to make sure I not only understand but agree with the changes that it authored.
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
If fsmerge is enabled and no write is needed, it can return an overlay
mount with a single lowerdir, which is illegal for overlayfs.
For example, it can cause the following Nerdctl error:
: I'm not sure why ctr works, but the issue is real.
```bash
$ nerdctl run --runtime io.containerd.kata.v2 --snapshotter=erofs -it --rm nginx:latest /bin/bash
FATA[0000] failed to mount {Type:overlay Source:overlay Target: Options:[lowerdir=/run/containerd/
io.containerd.mount-manager.v1.bolt/t/7/1]} on "/tmp/initialC2039543827": mount source: "overlay",
target: "/tmp/initialC2039543827", fstype: overlay, flags: 0, data: "lowerdir=/run/containerd/io.
containerd.mount-manager.v1.bolt/t/7/1", err: invalid argument
```
Switch to using a bind mount instead.
Fixes: 9a7500a974 ("Add support for EROFS fsmerge feature")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Although EROFS has native compression support (and each filesystem can
contain multiple compression algorithms), in many cases, people only
consider using zstd compression when transporting on the wire in order
to reduce the pulling time but maintain the optimal runtime performance.
Only `+zstd` is considered: it has skippable frames which will be used
for the seekable EROFS implementation in future containerd versions.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
The default walking applier performs a real temporary mount for
unpacking, but the mount manager failed to adapt to the walking
differ.
This fixes the EROFS snapshotter together with the default walking
differ, otherwise it reports:
```
ctr: apply layer error for "[]": failed to extract layer sha256:[]:
failed to mount /var/lib/containerd/tmpmounts/containerd-mount3992073457:
internal mount option "X-containerd.mkfs.fs=ext4" was not consumed by
the mount manager
```
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
