The preceding commit advertises the default platform with the erofs OS feature. Move construction of that platform set into snapshotterPlatforms so the client-side matching contract can be tested without changing the advertisement.
EROFS identifies the image layer package format rather than a Linux-only host capability, so the feature-bearing entry remains derived from platforms.DefaultSpec. Test that the advertised set matches both the plain default platform and the same platform carrying the erofs feature.
Assisted-by: Claude Opus 5
Signed-off-by: Martín Fernández <mfernandez@crunchloop.io>
Advertise a second platform - the default platform spec with OSFeatures
["erofs"] - from the EROFS snapshotter plugin, alongside the plain
default platform it already advertised. This mirrors what the EROFS
differ plugin already advertises.
This is required by client.Image.Unpack's checkSnapshotterSupport, which
rejects unpacking an image whose config declares os.features onto a
snapshotter that doesn't advertise a matching platform: without this,
pulling and unpacking an EROFS image layer format specification image
(https://github.com/erofs/erofs-image-spec) - which requires "erofs" in
os.features - with the erofs snapshotter would fail that check.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Signed-off-by: Martín Fernández <mfernandez@crunchloop.io>
The layer content cache was a single directory, so every source of
pre-converted blobs had to be merged into one tree. A shared read-only fleet
cache, a host-local cache, and a per-image warm cache could not coexist.
Replace layer_content_cache with layer_content_caches, a list. Each directory
is checked in order and the first hit is staged into the snapshot; a layer
found in none of them falls back to the normal download-and-convert path.
Cache directories are no longer required to exist at startup. A missing one is
indistinguishable from an empty one at lookup time (both are simply a miss),
and it may well be mounted after the daemon starts, so the only check left is
that each path is absolute -- a relative one would be symlinked into the
snapshot dir and dangle.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Replace the snapshots.ErrAlreadyStaged sentinel error with a
read-only-mounts signal: a snapshotter (e.g. erofs serving a layer
content cache hit) now returns Prepare mounts normally, with no
error, when the layer content is already staged into the active
snapshot. The unpacker's isStaged helper checks the last mount's
ReadOnly() to decide whether to skip fetch+apply and just commit.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
The cache used to serve a hit by committing the layer during Prepare and
returning ErrAlreadyExists, which is incompatible with parallel unpacking:
the "rebase" capability defers the parent to Commit time, so a
commit-at-Prepare layer ends up parentless. The erofs snapshotter therefore
had to disable rebase whenever the cache was enabled, making every cache
*miss* fall back to a fully sequential (slower-than-cold) pull.
Introduce snapshots.ErrAlreadyStaged, returned from Prepare to mean "the
active snapshot's content is staged; skip the layer download and apply, but
still Commit it" (where the parent is applied). Unlike ErrAlreadyExists it
does not end the layer's lifecycle. The unpacker handles it by emitting a
status whose bottom half runs the normal serialized commit (rebasing the
parent in), and the metadata snapshotter threads it through Prepare like a
normal active snapshot. The erofs cache hit now stages the blob and returns
ErrAlreadyStaged instead of committing, so it can advertise "rebase"
unconditionally: hits skip download+conversion and misses stay parallel.
Also promote the "rebase" capability string to snapshots.RebaseCap (shared by
the unpacker, erofs, and overlay).
Follow up: fetch is range-based, so only a fully cached contiguous
prefix skips downloads; a miss in a lower layer still pulls everything above
it. Per-layer fetch-skip is left as a follow-up.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add a layer_content_cache option to the erofs snapshotter: a directory of
pre-converted, diffID-keyed erofs layer blobs (in production a read-only
mountpoint-s3/FUSE mount) that the snapshotter sources layers from on pull
instead of downloading and converting them per node.
On an image-layer extraction Prepare, if the layer's blob is present in the
cache the snapshotter stages it as a symlink, commits it as the target
chainID in the same transaction, and returns ErrAlreadyExists. This uses the
existing remote-snapshot protocol, so containerd skips both the layer
download and the tar->erofs conversion; no core changes are needed. Any miss
(cache disabled, no snapshot.ref/diff-id labels, blob absent, unreadable
cache) falls through to the normal path, so pulls keep working.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Currently the metadata may be generated after a snapshot is committed,
causing a difference in the overlayfs mount that may cause ESTALE errors.
The committed snapshot must be immutable and should always return the
same set of mounts after commit, even if configuration is changed.
This setting may be added back later to be performed before commit,
either as part of the unpack or as a step before commit. This is getting
pulled out from the upcoming release.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This disables the slow_chown feature (nobody in their right mind
is going to be choosing erofs and want to slowly chown each file),
indicates that we support idmaps if the kernel supports it, and makes
sure to chown the upperdir.
This is more or less exactly how the overlay snapshotter does things,
minus the slow_chown part (which has discussions about dropping
altogether at some point anyways).
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
EROFS has supported a tiny metadata-only image to reference external
blobs since Linux 5.16. This eliminates the need to mount each EROFS
layer one by one and is also useful for VM-based containers (e.g.
nerdbox and Kata containers.)
Similar to LCOW/CimFS, `snapshots.UnpackKeyPrefix` is used to
trigger fsmerge generation (typically < 100 ms) on demand in Prepare().
In the future, we can also generate fsmeta in Commit() of the final
unpacking layer (by introducing an annotation to keep the chainID).
However, in the case of intermediate layer reuse, the Prepare() handling
will still be required.
```toml
[plugins."io.containerd.snapshotter.v1.erofs"]
max_unmerged_layers = 1 # enable fsmerge if image layers >= 2
```
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Rename `default_size_mb` to `default_size` so that it's similar to
Docker's `--storage-opt size=30G` flag. For example,
``` toml
[plugins."io.containerd.snapshotter.v1.erofs"]
default_size = "5GB"
```
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Enabling the IMMUTABLE_FL file attribute causes dirty data to be
flushed synchronously at least on EXT4, which can greatly impact
container launch performance. In contrast, the overlayfs snapshotter
does not use syncfs by default.
Most users may not need IMMUTABLE_FL, let's make IMMUTABLE_FL optional
to align with the behavior of the overlayfs snapshotter and recover the
original performance.
1. tensorflow
Test commands:
$ nerdctl image pull --snapshotter=X --unpack="false" tensorflow/tensorflow:2.19.0
$ time nerdctl container --snapshotter=X run -d tensorflow/tensorflow:2.19.0 /bin/sh
Results:
overlayfs | 0m18.748s
erofs (no IMMUTABLE_FL) | 0m10.090s
erofs (with IMMUTABLE_FL) | 0m21.074s
2. ubuntu 22.04
Test commands:
$ nerdctl image pull --snapshotter=X --unpack="false" ubuntu:22.04
$ time nerdctl container --snapshotter=X run -d ubuntu:22.04 /bin/sh
Results:
overlayfs | 0m1.147s
erofs (no IMMUTABLE_FL) | 0m0.795s
erofs (with IMMUTABLE_FL) | 0m1.094s
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Add fsverity support to erofs snapshotter to enable data integrity
verification for erofs layers:
- Add an config option `EnableFsverity` for erofs snapshotter
- Add fsverity verification during mount operations
- Enable fsverity on erofs layers during commit
- Add documentation for fsverity support in erofs snapshotter.
- Add TestErofsFsverity to verify fsverity enablement and data protection
The feature can be enabled via config.toml, such as:
```toml
[plugins.'io.containerd.snapshotter.v1.erofs']
root_path = ''
ovl_mount_options = []
enable_fsverity = true
```
Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>
It allows us to mount each EROFS blob layer (generated by the EROFS
differ) independently, or use the "unpacked" fs/ directories (if
some other differ is used.)
Currently, it's somewhat like the overlay snapshotter, but I tend
to separate the new EROFS logic into a self-contained component,
rather than keeping it tangled in the very beginning.
Existing users who use the overlay snapshotter won't be impacted
at all but they have a chance to use this new snapshotter to
leverage the EROFS filesystem.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Martín Fernández