Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Although EROFS has native compression support (and each filesystem can
contain multiple compression algorithms), in many cases, people only
consider using zstd compression when transporting on the wire in order
to reduce the pulling time but maintain the optimal runtime performance.
Only `+zstd` is considered: it has skippable frames which will be used
for the seekable EROFS implementation in future containerd versions.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
If no snapshotter is specified and `os.features` contains "erofs",
unpacking should use the EROFS snapshotter and differ.
This enhances the usability of native EROFS container images.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Use the same approach for appending UUID arguments in GenerateTarIndexAndAppendTar
as done in ConvertTarErofs for consistency between the two modes.
Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
Use the Linux default rather than the block size from the local macOS
system. The local macOS block size is not relevant as the erofs file
will not be mounted directly on macOS.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Derive filesystem UUIDs (`lsblk -o +UUID`) from the OCI layer digests
(although diffIDs are better in principle, but they're unavailable by
differs in advance) rather than generating a random one. This allows
EROFS to uniquely identify each layer using the content-addressable
filesystem UUID.
It can also be used for reproducible builds. To achieve this, configure
`mkfs_options` with `-T0 --mkfs-time` (However, `--mkfs-time` requires
erofs-utils 1.8+; Otherwise, all inode timestamps will be reset w/o it):
``` toml
[plugins."io.containerd.differ.v1.erofs"]
mkfs_options = ["-T0 --mkfs-time"]
```
Fixes: c73c8e5d52 ("Introduce EROFS differ")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
If the layer media type is expected as an EROFS native layer (ending
with `.erofs`), copy the content as the layer blob.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Avoid introducing utils package outside of internal. This package
should not be imported by other modules.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Unlike the walking differ, which implements a generic method to
accommodate all kinds of snapshotters, the EROFS differ is just
implemented for EROFS and EROFS snapshotter so it can utilize the
recent DiffDirChanges() [1] to avoid traversing the entire rootfs
directory in order to improve `nerdctl commit` performance.
Additionally, I think `baseDir` is unnecessary too (in principle,
only `upperdir` is useful for OCI format convention). However,
addressing this requires more work, so left as is for now.
It's also useful to implement a customized Compare() method for
EROFS differ so that we can dump the native EROFS-formatted blob
to the content store later.
[1] https://github.com/containerd/continuity/pull/145
Signed-off-by: Gao Xiang <xiang@kernel.org>
It allows us to mount each EROFS blob layer (generated by the EROFS
differ) independently, or use the "unpacked" fs/ directories (if
some other differ is used.)
Currently, it's somewhat like the overlay snapshotter, but I tend
to separate the new EROFS logic into a self-contained component,
rather than keeping it tangled in the very beginning.
Existing users who use the overlay snapshotter won't be impacted
at all but they have a chance to use this new snapshotter to
leverage the EROFS filesystem.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
The EROFS differ only applies to EROFS layers which are marked by
a special file `.erofslayer` generated by the EROFS snapshotter.
Why it's needed? Since we'd like to parse []mount.Mount directly
without actual mounting and convert OCI layers into EROFS blobs,
`.erofslayer` gives a hint that the active snapshotter supports
the output blob generated by the EROFS differ.
I'd suggest it could be read together with the next commit.
Signed-off-by: cardy.tang <zuniorone@gmail.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
