Enabling the IMMUTABLE_FL file attribute causes dirty data to be
flushed synchronously at least on EXT4, which can greatly impact
container launch performance. In contrast, the overlayfs snapshotter
does not use syncfs by default.
Most users may not need IMMUTABLE_FL, let's make IMMUTABLE_FL optional
to align with the behavior of the overlayfs snapshotter and recover the
original performance.
1. tensorflow
Test commands:
$ nerdctl image pull --snapshotter=X --unpack="false" tensorflow/tensorflow:2.19.0
$ time nerdctl container --snapshotter=X run -d tensorflow/tensorflow:2.19.0 /bin/sh
Results:
overlayfs | 0m18.748s
erofs (no IMMUTABLE_FL) | 0m10.090s
erofs (with IMMUTABLE_FL) | 0m21.074s
2. ubuntu 22.04
Test commands:
$ nerdctl image pull --snapshotter=X --unpack="false" ubuntu:22.04
$ time nerdctl container --snapshotter=X run -d ubuntu:22.04 /bin/sh
Results:
overlayfs | 0m1.147s
erofs (no IMMUTABLE_FL) | 0m0.795s
erofs (with IMMUTABLE_FL) | 0m1.094s
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Avoid introducing utils package outside of internal. This package
should not be imported by other modules.
Signed-off-by: Derek McGowan <derek@mcg.dev>
follow-up to 09f34d18b7
- Use strings.Cut instead of trimming prefixes and strings.Split
to reduce allocations
- Use a switch for mount-type for slightly better readability
than if / else if / else.
- Fix GoDoc to start with the function name.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Unlike the walking differ, which implements a generic method to
accommodate all kinds of snapshotters, the EROFS differ is just
implemented for EROFS and EROFS snapshotter so it can utilize the
recent DiffDirChanges() [1] to avoid traversing the entire rootfs
directory in order to improve `nerdctl commit` performance.
Additionally, I think `baseDir` is unnecessary too (in principle,
only `upperdir` is useful for OCI format convention). However,
addressing this requires more work, so left as is for now.
It's also useful to implement a customized Compare() method for
EROFS differ so that we can dump the native EROFS-formatted blob
to the content store later.
[1] https://github.com/containerd/continuity/pull/145
Signed-off-by: Gao Xiang <xiang@kernel.org>
Otherwise, the following error avoids snapshot GC:
level=warning msg="snapshot garbage collection failed" error="failed
to clear IMMUTABLE_FL: failed to open: open /var/lib/containerd/io.
containerd.snapshotter.v1.erofs/snapshots/2/layer.erofs: no such file
or directory
Fixes: b477cf8e97 ("erofs-snapshotter: protect layer blobs with FS_IMMUTABLE_FL")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Currently, containerd cannot dynamically select between EROFS block
or file-based mounting approaches based on the specific runtime (or
the Linux kernel version of the runtime) due to its static mount
structure.
For example, the EROFS snapshotter fails on Linux 5.4 (Ubuntu 20.04)
with `bin/nerdctl run --net=host --snapshotter=erofs busybox:latest`:
FATA[0005] failed to mount {Type:erofs Source:/var/lib/containerd/
io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs Target:
Options:[ro]} on "/tmp/initialC1374142795": block device required
Temporarily fix this by appending `-oloop` for single-layer images.
The upcoming mount manager will make it better [1].
[1] https://github.com/containerd/containerd/issues/11303
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
As documented in ioctl_iflags(2):
```
FS_IMMUTABLE_FL
The file is immutable: no changes are permitted to the file contents
or metadata (permissions, timestamps, ownership, link count, and so
on). (This restriction applies even to the superuser.)
```
For example, any user cannot delete/move layer blobs when
FS_IMMUTABLE_FL is set:
``` sh
# cd /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/4
# mv layer{,1}.erofs
mv: cannot move 'layer.erofs' to 'layer1.erofs': Operation not permitted
# rm layer.erofs
rm: cannot remove 'layer.erofs': Operation not permitted
```
Note that it's a best-effort approach for data loss prevention. IOWs,
just warn out if FS_IMMUTABLE_FL cannot be set anyway (e.g., due to lack
of support in the underlying filesystem.)
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Add fsverity support to erofs snapshotter to enable data integrity
verification for erofs layers:
- Add an config option `EnableFsverity` for erofs snapshotter
- Add fsverity verification during mount operations
- Enable fsverity on erofs layers during commit
- Add documentation for fsverity support in erofs snapshotter.
- Add TestErofsFsverity to verify fsverity enablement and data protection
The feature can be enabled via config.toml, such as:
```toml
[plugins.'io.containerd.snapshotter.v1.erofs']
root_path = ''
ovl_mount_options = []
enable_fsverity = true
```
Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>
Adds a new diff plugin that can import image layers in the block CIM format using the new
block CIM layer writer added in hcsshim repo.
This commit also makes another important change in the way a diff is applied when using
CimFS based layer writers. Currently, the diff plugins call archive.Apply to apply a diff
and pass a function (that can actually apply the diff) as an argument (via
archive.ApplyOptions). This allows the callers to call archive.Apply with either a custom
applier function or if the caller doesn't pass such a function archive.Apply uses the
default naive diff applier.
However, there is drawback to this approach. The applier function passed to the
`archive.Apply` call needs to follow a specific signature. This signature expects it that
all parent layers are represented as an array of strings. In cases like CimFS, we can't
easily represent a set of layers as strings (unless we encode extra data in those strings
in a hacky way). To get around this problem, the diff plugins for CimFS based layers, skip
the archive.Apply call and directly call the layer writer instead.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
It allows us to mount each EROFS blob layer (generated by the EROFS
differ) independently, or use the "unpacked" fs/ directories (if
some other differ is used.)
Currently, it's somewhat like the overlay snapshotter, but I tend
to separate the new EROFS logic into a self-contained component,
rather than keeping it tangled in the very beginning.
Existing users who use the overlay snapshotter won't be impacted
at all but they have a chance to use this new snapshotter to
leverage the EROFS filesystem.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
hcsshim recently [updated](microsoft/hcsshim@1d406d0) the interface of APIs that are used
for importing OCI layers. It now expects that the CimFS snapshotter mounts contain the
full cim paths for parent layers. This change updates the cimfs differ & snapshotter to
use that new interface.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
This is a planned follow-on from #10721 primarily at the request of
@fuweid, exchanging MNT_DETACH at unmount time for MOUNT_ATTR_RDONLY at
mount time. The effect is to increase risk of unmount failure due to
EBUSY (as observed in the wild) but add an additional protection that the
then-leaked bind mount does not act as a conduit for inadvertent modification
of the underlying data, including our own efforts to clean up the mountpoint.
Tests covering the lifecycle of the temporary idmap mounts and integrity
of the underlying lower layer data is also included in the normal and
failed-unmount case.
Fixes#10704
Signed-off-by: Mike Baynton <mike@mbaynton.com>
Commit 8437c567d8 migrated the use of the
userns package to the github.com/moby/sys/user module.
After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).
This patch migrates our code to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].
This patch deprecates the containerd fork of that package, and adds it as
an alias for the moby/sys/user/userns package.
[1]: 3778ae603c
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some of the snapshotters that allow you to change their root location
were already doing this, this just makes all of them follow the same
pattern.
Signed-off-by: Danny Canter <danny@dcantah.dev>
CimFS layers don't need to create a new scratch VHD per image. The scratch VHDs used with CimFS are empty so
we can just create one base VHD and one differencing VHD and copy it for every scratch snapshot.
(Note that UVM VHDs are still unique per image because the VHD information is embedded in the UVM BCD during
import)
Signed-off-by: Amit Barve <ambarve@microsoft.com>
