Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Switch block CIM snapshotter and diff plugin init checks from
IsBlockCimSupported() to IsBlockCimWriteSupported() to allow plugins
to load when the CimWriter.dll is available, even on OS builds
without native block CIM support.
Signed-off-by: Maksim An <maksiman@microsoft.com>
Although EROFS has native compression support (and each filesystem can
contain multiple compression algorithms), in many cases, people only
consider using zstd compression when transporting on the wire in order
to reduce the pulling time but maintain the optimal runtime performance.
Only `+zstd` is considered: it has skippable frames which will be used
for the seekable EROFS implementation in future containerd versions.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
The default walking applier performs a real temporary mount for
unpacking, but the mount manager failed to adapt to the walking
differ.
This fixes the EROFS snapshotter together with the default walking
differ, otherwise it reports:
```
ctr: apply layer error for "[]": failed to extract layer sha256:[]:
failed to mount /var/lib/containerd/tmpmounts/containerd-mount3992073457:
internal mount option "X-containerd.mkfs.fs=ext4" was not consumed by
the mount manager
```
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
If no snapshotter is specified and `os.features` contains "erofs",
unpacking should use the EROFS snapshotter and differ.
This enhances the usability of native EROFS container images.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Use the same approach for appending UUID arguments in GenerateTarIndexAndAppendTar
as done in ConvertTarErofs for consistency between the two modes.
Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
Use the Linux default rather than the block size from the local macOS
system. The local macOS block size is not relevant as the erofs file
will not be mounted directly on macOS.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Add new snapshotter & differ plugins that can leverage the block CIM layer writers to
store pulled/imported images in the block CIM format.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
Derive filesystem UUIDs (`lsblk -o +UUID`) from the OCI layer digests
(although diffIDs are better in principle, but they're unavailable by
differs in advance) rather than generating a random one. This allows
EROFS to uniquely identify each layer using the content-addressable
filesystem UUID.
It can also be used for reproducible builds. To achieve this, configure
`mkfs_options` with `-T0 --mkfs-time` (However, `--mkfs-time` requires
erofs-utils 1.8+; Otherwise, all inode timestamps will be reset w/o it):
``` toml
[plugins."io.containerd.differ.v1.erofs"]
mkfs_options = ["-T0 --mkfs-time"]
```
Fixes: c73c8e5d52 ("Introduce EROFS differ")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
If the layer media type is expected as an EROFS native layer (ending
with `.erofs`), copy the content as the layer blob.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Avoid introducing utils package outside of internal. This package
should not be imported by other modules.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Unlike the walking differ, which implements a generic method to
accommodate all kinds of snapshotters, the EROFS differ is just
implemented for EROFS and EROFS snapshotter so it can utilize the
recent DiffDirChanges() [1] to avoid traversing the entire rootfs
directory in order to improve `nerdctl commit` performance.
Additionally, I think `baseDir` is unnecessary too (in principle,
only `upperdir` is useful for OCI format convention). However,
addressing this requires more work, so left as is for now.
It's also useful to implement a customized Compare() method for
EROFS differ so that we can dump the native EROFS-formatted blob
to the content store later.
[1] https://github.com/containerd/continuity/pull/145
Signed-off-by: Gao Xiang <xiang@kernel.org>
Adds a new diff plugin that can import image layers in the block CIM format using the new
block CIM layer writer added in hcsshim repo.
This commit also makes another important change in the way a diff is applied when using
CimFS based layer writers. Currently, the diff plugins call archive.Apply to apply a diff
and pass a function (that can actually apply the diff) as an argument (via
archive.ApplyOptions). This allows the callers to call archive.Apply with either a custom
applier function or if the caller doesn't pass such a function archive.Apply uses the
default naive diff applier.
However, there is drawback to this approach. The applier function passed to the
`archive.Apply` call needs to follow a specific signature. This signature expects it that
all parent layers are represented as an array of strings. In cases like CimFS, we can't
easily represent a set of layers as strings (unless we encode extra data in those strings
in a hacky way). To get around this problem, the diff plugins for CimFS based layers, skip
the archive.Apply call and directly call the layer writer instead.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
It allows us to mount each EROFS blob layer (generated by the EROFS
differ) independently, or use the "unpacked" fs/ directories (if
some other differ is used.)
Currently, it's somewhat like the overlay snapshotter, but I tend
to separate the new EROFS logic into a self-contained component,
rather than keeping it tangled in the very beginning.
Existing users who use the overlay snapshotter won't be impacted
at all but they have a chance to use this new snapshotter to
leverage the EROFS filesystem.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
The EROFS differ only applies to EROFS layers which are marked by
a special file `.erofslayer` generated by the EROFS snapshotter.
Why it's needed? Since we'd like to parse []mount.Mount directly
without actual mounting and convert OCI layers into EROFS blobs,
`.erofslayer` gives a hint that the active snapshotter supports
the output blob generated by the EROFS differ.
I'd suggest it could be read together with the next commit.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
hcsshim recently [updated](microsoft/hcsshim@1d406d0) the interface of APIs that are used
for importing OCI layers. It now expects that the CimFS snapshotter mounts contain the
full cim paths for parent layers. This change updates the cimfs differ & snapshotter to
use that new interface.
Signed-off-by: Amit Barve <ambarve@microsoft.com>
