Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
NewSnapshotter auto-appends "index=off" unless the configured
mount_options already carry an index option — but the check used
hasOption(..., "index", false), which compares against the literal
string "index". No valid mount option ever equals "index" (it is
always valued, "index=on"/"index=off"), so the append happened
unconditionally whenever the kernel exposes the overlay index
parameter.
Mount options are last-wins in the kernel, so a user-configured
"index=on" was silently overridden. Worse, configuring
mount_options = ["index=on", "nfs_export=on"] (the documented way to
make overlay mounts NFS-exportable) produced
"index=on,nfs_export=on,index=off", which the kernel rejects with
EINVAL (nfs_export=on conflicts with an explicit index=off) — breaking
every snapshot mount including image unpack.
Make hasOption match the key of a "key[=value]" option so callers need
not care whether the option takes a value, and add a regression test.
Signed-off-by: AprilNEA <github@sku.moe>
Disable stat usage since we never consume the data.
This can reduce unnecessary contention during transactions.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Perform an early validation check on both container checkpoint and
restore paths to fail-fast if the CRIU binary is missing or is older
than the minimum required version (3.16.0).
To support runtime-configured environments, the validation respects the
custom PATH from the shim manager environment if configured, skipping
any relative paths to avoid incorrect daemon-relative resolution. If not
configured, it falls back to a standard system PATH lookup. The check
result is cached using sync.Once to prevent redundant process spawning.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Switch block CIM snapshotter and diff plugin init checks from
IsBlockCimSupported() to IsBlockCimWriteSupported() to allow plugins
to load when the CimWriter.dll is available, even on OS builds
without native block CIM support.
Signed-off-by: Maksim An <maksiman@microsoft.com>
This change updates an incorrect default config value for the block CIM
snapshotter. The `ntfsFormat` should always be false for the block CIM
snapshotter. If the scratch needs to be formatted with NTFS, a format
option named `WithNTFSFormat` can be provided.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
dmsetup.Status() returns a nil DeviceStatus when it fails. The mkfs
failure path records that error, but still unconditionally dereferences
status when logging the pool status.
Avoid the panic by logging a fallback status string when dmsetup.Status()
does not return a status.
Found by Linux Verification Center (linuxtesting.org) with SVACE.
Signed-off-by: Mikhail Dmitrichenko <m.dmitrichenko222@gmail.com>
On Windows, os.RemoveAll and os.ReadFile can fail with a sharing
violation when a file handle has not been fully released by the OS.
Unlike Unix, Windows does not allow deleting or reading a file
while any handle is open, and handles may be held briefly after
Close returns. This causes a race between the gRPC server-side
writer Close (which writes updatedat.tmp) and both Abort (which
calls RemoveAll on the ingest directory) and status (which reads
the updatedat file), surfacing as a flaky
TestContentClient/CommitErrorState on Windows CI.
Add platform-specific removePath and readFileWithRetry helpers
that retry with incremental backoff on Windows. On non-Windows
platforms they pass through to os.RemoveAll and os.ReadFile
directly. This follows the same pattern as ensureRemoveAll in
the CRI server helpers, which retries on EBUSY for mount races
on Linux.
Signed-off-by: ayush-panta <ayushkp@amazon.com>
Fix issue 13388
```
[...]
May 12 16:57:23 kind-control-plane kubelet[257]: failed to extract layer (application/vnd.docker.image.rootfs
.diff.tar sha256:6f1cdceb6a3146f0ccb986521156bef8a422cdbb0863396f7f751f575ba308f4) to overlayfs as "extract-920875437
-7QPF sha256:31e64620332e54e3e4fb246d8325ed2c9f1c2cc64a95f0bb23b4b7e82834c95a": failed to mount /var/lib/containerd/t
mpmounts/containerd-mount2180142388: mount source: "overlay", target: "/var/lib/containerd/tmpmounts/containerd-mount
2180142388", fstype: overlay, flags: 0, data: "upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/sn
apshots/279/fs", err: invalid argument
[...]
```
This was a regression introduced in PR 13115.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The grpc, grpc-tcp, and ttrpc server plugins enumerated their services
through ic.GetByType, which short-circuits on the first plugin whose
Instance() returned an error. A single failed gRPC plugin (e.g. CRI
under rootless, which cannot watch /etc/cni/net.d) therefore prevented
the server plugins from initialising, leaving /run/containerd/containerd.sock
uncreated.
Iterate the plugin set directly and skip plugins that failed to
initialise, restoring the pre-c15ec2485 behaviour where the listener
is still created and only the failed services are missing.
Fixes: c15ec2485 ("Add server plugins for grpc and ttrpc")
Fixes: https://github.com/containerd/containerd/issues/13362
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Currently the metadata may be generated after a snapshot is committed,
causing a difference in the overlayfs mount that may cause ESTALE errors.
The committed snapshot must be immutable and should always return the
same set of mounts after commit, even if configuration is changed.
This setting may be added back later to be performed before commit,
either as part of the unpack or as a step before commit. This is getting
pulled out from the upcoming release.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Update the EROFS snapshotter to pass the dm-verity metadata path
through the `X-containerd.dmverity` mount option. Previously, the
system relied on implicit mode detection; providing the explicit
path allows downstream components (such as Kata Containers mount
handlers) to reliably locate and read `.dmverity` files.
Correspondingly, update the EROFS mount handler to parse and utilize
this explicit metadata path for dm-verity device initialization.
This enables the runtime to retrieve root hashes and other necessary
metadata directly, ensuring robust layer integrity verification.
Signed-off-by: Alex Lyn <alex.lyn@antgroup.com>
The gRPC sandbox controller service only forwarded the `options` field
when calling the local controller. The `netns_path`, `rootfs`, and
`annotations` fields were silently dropped, causing clients using the
gRPC proxy path to receive incomplete sandbox configurations.
Event topics were missing the leading `/` prefix ("sandboxes/create"
instead of "/sandboxes/create"), causing the event exchange to reject
the publish and return an error to the caller.
Add unit tests for the controller service that exercise all RPC
methods.
Signed-off-by: William Myers <willmyrs@amazon.com>
The new server plugin type allows services which have listeners to be
configured as plugins rather than defined directly in the global
configuration. This provides more configuration consistency and allows
containerd to be extended for new types of api handlers.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This is all specific to the content store -- prior to this change, `ctr content fetch docker.io/tianon/test:sha512-blobs` fails with errors about `sha256` digests, and after this change it succeeds and ingests the `sha512` blobs successfully.
In the interest of full disclosure, this change was mostly authored by myself without any AI-assistance, but Claude was used to find the appropriate tests to update and to understand why the metadata service would clear out `Digest` when passing details to the content store and thus the appropriate fix being in the Writer's Commit method (re-hashing after Close when the digest algorithm was previously unknown but is known and non-canonical during Commit). I had a very active hand in instructing Claude on exactly which changes to make and where to make them, and I spent a lot of time reviewing them to make sure I not only understand but agree with the changes that it authored.
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
If fsmerge is enabled and no write is needed, it can return an overlay
mount with a single lowerdir, which is illegal for overlayfs.
For example, it can cause the following Nerdctl error:
: I'm not sure why ctr works, but the issue is real.
```bash
$ nerdctl run --runtime io.containerd.kata.v2 --snapshotter=erofs -it --rm nginx:latest /bin/bash
FATA[0000] failed to mount {Type:overlay Source:overlay Target: Options:[lowerdir=/run/containerd/
io.containerd.mount-manager.v1.bolt/t/7/1]} on "/tmp/initialC2039543827": mount source: "overlay",
target: "/tmp/initialC2039543827", fstype: overlay, flags: 0, data: "lowerdir=/run/containerd/io.
containerd.mount-manager.v1.bolt/t/7/1", err: invalid argument
```
Switch to using a bind mount instead.
Fixes: 9a7500a974 ("Add support for EROFS fsmerge feature")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Although EROFS has native compression support (and each filesystem can
contain multiple compression algorithms), in many cases, people only
consider using zstd compression when transporting on the wire in order
to reduce the pulling time but maintain the optimal runtime performance.
Only `+zstd` is considered: it has skippable frames which will be used
for the seekable EROFS implementation in future containerd versions.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
The default walking applier performs a real temporary mount for
unpacking, but the mount manager failed to adapt to the walking
differ.
This fixes the EROFS snapshotter together with the default walking
differ, otherwise it reports:
```
ctr: apply layer error for "[]": failed to extract layer sha256:[]:
failed to mount /var/lib/containerd/tmpmounts/containerd-mount3992073457:
internal mount option "X-containerd.mkfs.fs=ext4" was not consumed by
the mount manager
```
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
klog has a long-standing bug where setting -logtostderr=true causes the
-stderrthreshold flag to be silently ignored. All log messages are sent
to stderr regardless of their severity.
klog v2.140.0 introduced two new flags that allow callers to opt into the
correct behavior:
-legacy_stderr_threshold_behavior=false
-stderrthreshold=INFO
Set them in the CRI plugin's setGLogLevel() function, right after
klog.InitFlags() and before any other flag configuration.
Reference: kubernetes/klog#212
Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>
If no snapshotter is specified and `os.features` contains "erofs",
unpacking should use the EROFS snapshotter and differ.
This enhances the usability of native EROFS container images.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
