To make it as DEPRECATED, this PR does the following:
1. Changes config default to use `NetworkPluginBinDirs`;
2. Mark `NetworkPluginBinDir` as deprecated (in config version 3);
3. Add config migration from 2 to 3, which migrates `bin_dir`
in version 2 to `bin_dirs` in version 3.
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
[wip] add deprecation warning
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
This implements container restore as described in:
https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone
For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh
The code changes are based on changes I have done in Podman around 2018
and CRI-O around 2020.
The history behind restoring container via CRI/Kubernetes probably
requires some explanation. The initial proposal to bring
checkpoint/restore to Kubernetes was looking at pod checkpoint and
restoring and the corresponding CRI changes.
https://github.com/kubernetes-sigs/cri-tools/pull/662https://github.com/kubernetes/kubernetes/pull/97194
After discussing this topic for about two years another approach was
implemented as described in KEP-2008:
https://github.com/kubernetes/enhancements/issues/2008
"Forensic Container Checkpointing" allowed us to separate checkpointing
from restoring. For the "Forensic Container Checkpointing" it is enough
to create a checkpoint of the container. Restoring is not necessary as
the analysis of the checkpoint archive can happen without restoring the
container.
While thinking about a way to restore a container it was by coincidence
that we started to look into restoring containers in Kubernetes via
Create and Start. The way it was done in CRI-O is to figure out during
Create if the container image is a checkpoint image and if that is true
we are using another code path. The same was implemented now with this
change in containerd.
With this change it is possible to restore the container from a
checkpoint tar archive that is created during checkpointing via CRI.
To restore a container via Kubernetes we convert the tar archive to an
OCI image as described in the kubernetes.io blog post from above. Using
this OCI image it is possible to restore a container in Kubernetes.
At this point I think it should be doable to restore containers in
CRI-O and containerd no matter if they have been created by containerd or
CRI-O. The biggest difference is the container metadata and that can
be adapted during restore.
Open items:
* It is not clear to me why restoring a container in containerd goes
through task/Create(). But as the restore code already exists this
change extended the existing code path to restore a container in
task/Create() to also restore a container through the CRI via
Create and Start.
* Automatic image pulling. containerd does not pull images
automatically if created via the CRI. There is an option in
crictl to pull images before starting, but that uses the CRI
image pull interface. It is still a separate pull and create
operation. Restoring containers from an OCI image is a bit
different. The checkpoint OCI image does not include the base
image, but just a reference to the image (NAME@DIGEST).
Using crictl with pulling will enable the pulling of the
checkpoint image, but not of the base image the checkpoint is
based on. So during preparation of the checkpoint containerd
will automatically pull the base image, but I was not able how
to pull an image blockingly in containerd. So there is a for
loop waiting for the container image to appear in the internal
store. I think this probably can be implemented better.
Anyway, this is a first step towards container restored in Kubernetes
when using containerd.
Signed-off-by: Adrian Reber <areber@redhat.com>
Otherwise, the following error avoids snapshot GC:
level=warning msg="snapshot garbage collection failed" error="failed
to clear IMMUTABLE_FL: failed to open: open /var/lib/containerd/io.
containerd.snapshotter.v1.erofs/snapshots/2/layer.erofs: no such file
or directory
Fixes: b477cf8e97 ("erofs-snapshotter: protect layer blobs with FS_IMMUTABLE_FL")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Currently, containerd cannot dynamically select between EROFS block
or file-based mounting approaches based on the specific runtime (or
the Linux kernel version of the runtime) due to its static mount
structure.
For example, the EROFS snapshotter fails on Linux 5.4 (Ubuntu 20.04)
with `bin/nerdctl run --net=host --snapshotter=erofs busybox:latest`:
FATA[0005] failed to mount {Type:erofs Source:/var/lib/containerd/
io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs Target:
Options:[ro]} on "/tmp/initialC1374142795": block device required
Temporarily fix this by appending `-oloop` for single-layer images.
The upcoming mount manager will make it better [1].
[1] https://github.com/containerd/containerd/issues/11303
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
As documented in ioctl_iflags(2):
```
FS_IMMUTABLE_FL
The file is immutable: no changes are permitted to the file contents
or metadata (permissions, timestamps, ownership, link count, and so
on). (This restriction applies even to the superuser.)
```
For example, any user cannot delete/move layer blobs when
FS_IMMUTABLE_FL is set:
``` sh
# cd /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/4
# mv layer{,1}.erofs
mv: cannot move 'layer.erofs' to 'layer1.erofs': Operation not permitted
# rm layer.erofs
rm: cannot remove 'layer.erofs': Operation not permitted
```
Note that it's a best-effort approach for data loss prevention. IOWs,
just warn out if FS_IMMUTABLE_FL cannot be set anyway (e.g., due to lack
of support in the underlying filesystem.)
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Add fsverity support to erofs snapshotter to enable data integrity
verification for erofs layers:
- Add an config option `EnableFsverity` for erofs snapshotter
- Add fsverity verification during mount operations
- Enable fsverity on erofs layers during commit
- Add documentation for fsverity support in erofs snapshotter.
- Add TestErofsFsverity to verify fsverity enablement and data protection
The feature can be enabled via config.toml, such as:
```toml
[plugins.'io.containerd.snapshotter.v1.erofs']
root_path = ''
ovl_mount_options = []
enable_fsverity = true
```
Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>
Adds a new diff plugin that can import image layers in the block CIM format using the new
block CIM layer writer added in hcsshim repo.
This commit also makes another important change in the way a diff is applied when using
CimFS based layer writers. Currently, the diff plugins call archive.Apply to apply a diff
and pass a function (that can actually apply the diff) as an argument (via
archive.ApplyOptions). This allows the callers to call archive.Apply with either a custom
applier function or if the caller doesn't pass such a function archive.Apply uses the
default naive diff applier.
However, there is drawback to this approach. The applier function passed to the
`archive.Apply` call needs to follow a specific signature. This signature expects it that
all parent layers are represented as an array of strings. In cases like CimFS, we can't
easily represent a set of layers as strings (unless we encode extra data in those strings
in a hacky way). To get around this problem, the diff plugins for CimFS based layers, skip
the archive.Apply call and directly call the layer writer instead.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
It allows us to mount each EROFS blob layer (generated by the EROFS
differ) independently, or use the "unpacked" fs/ directories (if
some other differ is used.)
Currently, it's somewhat like the overlay snapshotter, but I tend
to separate the new EROFS logic into a self-contained component,
rather than keeping it tangled in the very beginning.
Existing users who use the overlay snapshotter won't be impacted
at all but they have a chance to use this new snapshotter to
leverage the EROFS filesystem.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
The EROFS differ only applies to EROFS layers which are marked by
a special file `.erofslayer` generated by the EROFS snapshotter.
Why it's needed? Since we'd like to parse []mount.Mount directly
without actual mounting and convert OCI layers into EROFS blobs,
`.erofslayer` gives a hint that the active snapshotter supports
the output blob generated by the EROFS differ.
I'd suggest it could be read together with the next commit.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
hcsshim recently [updated](microsoft/hcsshim@1d406d0) the interface of APIs that are used
for importing OCI layers. It now expects that the CimFS snapshotter mounts contain the
full cim paths for parent layers. This change updates the cimfs differ & snapshotter to
use that new interface.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
This is a planned follow-on from #10721 primarily at the request of
@fuweid, exchanging MNT_DETACH at unmount time for MOUNT_ATTR_RDONLY at
mount time. The effect is to increase risk of unmount failure due to
EBUSY (as observed in the wild) but add an additional protection that the
then-leaked bind mount does not act as a conduit for inadvertent modification
of the underlying data, including our own efforts to clean up the mountpoint.
Tests covering the lifecycle of the temporary idmap mounts and integrity
of the underlying lower layer data is also included in the normal and
failed-unmount case.
Fixes#10704
Signed-off-by: Mike Baynton <mike@mbaynton.com>
The cri image service init has a bug where, after getting FSPath
for snapshotter_i, it stores it under defaultSnapshotter instead
of snapshotter_i.
Also make a few other refactor:
1. Dedup the snapshotRoot loading for defaultSnapshotter
2. Remove some unnecessary logic in RuntimePlatforms for-loop
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
* container_update_resources.go: it is Windows and Linux that need special handling
* local*.go: all platforms use the same list of tasks
* temp_unix.go/temp_unsupported.go: Darwin is a Unix
* util_unix.go/util_unsupported.go: use generic unix tag
The only user-visible effect of these changes is that tempMountLocation is now properly handled on Darwin
Signed-off-by: Marat Radchenko <marat@slonopotamus.org>
Other similar events were already moved to the metadata store. The
metadata store has more information that can be used for a future
content created event.
Signed-off-by: Derek McGowan <derek@mcg.dev>
- combine consecutive "WithField" calls to "WithFields", as multiple
calls is known to be expensive.
- include a "snapshotter" field in logs to allow correlating actions
with specific snapshotters.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A contentstore can be created on top of readonly path and
should not fail unless there is an attempt to write into it.
Currently this fails because new ingest directory is created
always, meaning for example that you can't create a store to
read blobs from OCI layout without it contaminating the OCI
layout files.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Uses the new github.com/containerd/errdefs/pkg module which is intended
to hold less stable utility functions separately from the stable
github.com/containerd/errdefs error types.
Includes temporary update to hcsshim until a release is cut there
Signed-off-by: Derek McGowan <derek@mcg.dev>
