The X-containerd.* prefix is reserved for containerd-internal mount
options that are processed by containerd components (such as mount
transformers and handlers). These options must not be passed to the
kernel mount syscall.
This change adds filtering at the core mount layer
as a safety net to prevent kernel errors if components forget to clean
up their options.
Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
The loop `for flag := range unprivilegedFlags` iterates over slice
indices (0,1,2,3,4,5,6) rather than the actual flag values (MS_RDONLY,
MS_NODEV, etc). This was a porting error from moby/moby where the data
structure was a map (where `for k := range m` yields keys/values).
As a result, MS_NOEXEC, MS_NOATIME, MS_RELATIME, and MS_NODIRATIME are
never detected or preserved. In user namespaces, this causes bind-mount
remounts to fail with EPERM when any of these flags are locked on the
parent mount, because the kernel requires all CL_UNPRIVILEGED locked
flags to be preserved during remount.
MS_RDONLY (0x1), MS_NOSUID (0x2), and MS_NODEV (0x4) happened to work
by coincidence because their values equal low index numbers.
Fix by using `for _, flag := range` to iterate over values.
Signed-off-by: Luke Hinds <luke@stacklok.com>
Recursive is needed to catch mounts under the bind mount, for example
erofs mount points. i.e. with that in place things like:
/foo/erofs1
/foo/erofs2
bind mount + idmap /foo/ -> /foo-idmapped/
makes it so erofs1 and erofs2 are at /foo-idmapped/, allowing us to use
that when constructing the final overlay mount erofs snapshotters use.
This let's the snapshotter claim to support idmap support later without
issue (right now it goes through remapfs).
As part of this, we want to make the bind mount have private mount
propagation. If you umount /foo-idmapped/erofs1 it shouldn't affect
/foo/erofs1, which could be reused in another container's lowerdirs and
idmapped bind mount.
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
Commit ee8ae9d569 ("Update erofs snapshotter to use mount manager")
temporarily removed the file-backed mount feature to adapt to the new
mount manager infrastructure as a quick start.
After the mount manager was introduced, a specific mount type can be
handled with a mount handler plugin to provide a dedicated mount
process (e.g. setup loopback devices in advance or calling external
mount helpers).
This commit adds a default EROFS mount handler for the Linux hosts
to set up loop devices for mount sources and "device=" external file
blobs if necessary (i.e. when file-backed mounts are unavailable),
allowing common runtimes such as runC to work directly, e.g.
``` sh
mount -t erofs /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs \
/run/containerd/io.containerd.mount-manager.v1.bolt/t/346/1
```
will be handled as
``` sh
mount -t erofs /dev/loop1 /run/containerd/io.containerd.mount-manager.v1.bolt/t/346/1
```
and
``` sh
mount -t erofs /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/7/fsmeta.erofs \
-odevice=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs,\
device=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/2/layer.erofs,\
...
device=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/7/layer.erofs
/run/containerd/io.containerd.mount-manager.v1.bolt/t/335/1
```
will be handled as
``` sh
mount -t erofs /dev/loop1 -odevice=/dev/loop2,device=/dev/loop3,... \
/run/containerd/io.containerd.mount-manager.v1.bolt/t/335/1
```
if file-backed mounts are unavailable.
For other host platforms (e.g. Darwin hosts) or specific runtimes
that require EROFS raw mounts instead of parsed mounts, this plugin
can be explicitly masked off by users.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
When a lot of pods are created all at the same time, the umount fails
with EBUSY sometimes. This causes some mounts to be leaked. A way to
repro this issue is here:
https://github.com/containerd/containerd/issues/12139#issuecomment-3165305904
While using lsof/fuser it was not possible to get the culprits on time
(the mount is busy for a few ms), @fuwei has found what is causing the
mount to be busy:
https://github.com/containerd/containerd/issues/12139#issuecomment-3184544534
When we fork in `GetUsernsFD()`, if a call to prepareIDMappedOverlay()
is ongoing and has an open fd to the path to idmap, then the unmount
callback will fail as the forked process still has an open fd to it.
Let's handle the idmap unmounts in the same way other temp mounts are
done, using the Unmount() helper, that retries the unmount.
This retry fixes it 100% of the times on my system and in the systems
of @halaney that reported the issue too.
This was originally fixed by #10721 by using a detached mount, but the
mount was switched to non-detached again in #10955 and we started to
leak mounts. As @fuwei mentioned, using a detached mount is quite
invisible for the admin and, therefore, a retry is a better alternative.
It seems these umounts are a great candidate for #11303, which will
manage the life-cycle of mounts and can handle these retries whenever
needed.
[1]: To do it, I just run as root "unshare -m", that creates a mntns
with private propagation, and then run the containerd daemon.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Before this patch, the cleanup was unconditionally trying an unmount,
which is not needed if the mount never succeeded in the first place.
This causes logs of failed stuff, that should never be there.
Also, one function was creating the tmpDir and the nested one removing
it (in the cleanup function), which complicates the reasoning about
not leaking resources.
This patch on one hand moves the creation of the tmpDir into the
function that will remove it later and renames the parameter. On the
other hand, it also separates the cleanup function in two functions:
cleanDir() and cleanMount(). Each one will clean the directory or the
mount, only if needed, and a new cleanup function that just calls
cleanDir() and cleanMount() is returned as the cleanup function handler.
Because we now create the tmp directory inside the function, we need to
adjust the test: the directory passed as param now will exist, but it
shoild be empty. Therefore, we change the os.Stat() to an os.Remove().
If it's not empty (the cleanup didn't work), the remove will fail.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
doPrepareIDMappedOverlay() can return nil as the cleanup function. As
the function mandates for the callback to be called, even when errors
are returned, we end-up calling the nil function that of course causes:
panic: runtime error: invalid memory address or nil pointer dereference
The containerd daemon continues to run fine, though.
With the current structure of always calling the cleanup function, we
would either need to check which error it is, to call it only on
specific errors (ugly) or return a "stub function" that doesn't do
anything on those cases (so the call works). None of these options seem
very nice.
Let's just switch the logic to a pattern more common on go: only need to
cleanup if it returned fine. This makes it easier for callers to not do
the wrong thing.
For this change we need to do the cleanup when returning errors on the
function itself, so the user doesn't need to do it for us.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Per layer idmap'ed bind mounts are costly to performance, as shown in
[0]. Each one requires taking various kernel locks, and each one shows
up in the host's mount table leading to some components like systemd
processing all these temporary mounts unnecessarily.
Let's instead go ahead and idmap the common directory of all the
layers to achieve the same effect. Now instead of being a function of
the number of layers, its a constant idmap per overlayfs! This can have
a big impact. For example, imagine running 100 containers at once, each
with 50 layers. That's going from doing 100 * 50 (5000) bind mounts, to
just 100. In reality both the shim and containerd proper do this, so its
actually double that!
[0]: https://github.com/containerd/containerd/issues/12048#issuecomment-3050444019
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
This is a planned follow-on from #10721 primarily at the request of
@fuweid, exchanging MNT_DETACH at unmount time for MOUNT_ATTR_RDONLY at
mount time. The effect is to increase risk of unmount failure due to
EBUSY (as observed in the wild) but add an additional protection that the
then-leaked bind mount does not act as a conduit for inadvertent modification
of the underlying data, including our own efforts to clean up the mountpoint.
Tests covering the lifecycle of the temporary idmap mounts and integrity
of the underlying lower layer data is also included in the normal and
failed-unmount case.
Fixes#10704
Signed-off-by: Mike Baynton <mike@mbaynton.com>
The detached mount is less likely to fail in our case, but if we see any
failure to unmount, we should just skip the removal of directories.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Using os.RemoveAll() is quite risky, as if the unmount failed and we
can delete files from the container rootfs. In fact, we were doing just
that.
Let's use os.Remove() to make sure we only deleted empty dirs.
Big kudos to @mbaynton for reporting this issue with lot of details,
nailing it down to containerd lines of code and showing all the log
lines to understand the big picture.
Fixes: #10704
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Overlayfs needs to do an idmap mount of each layer and the cleanup
function just unmounts and deletes the directories. However, when the
resource is busy, the umount fails.
Let's make the unmount detached so the unmount will eventually be done
when it's not busy anymore. Also, making it detached solves the issues with
the unmount failing because it is busy.
Big kudos to @mbaynton for reporting this issue with lot of details,
nailing it down to containerd lines of code and showing all the log
lines to understand the big picture.
Fixes: #10704
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
After these changes, in order to add Darwin bind-mount implementation, one only needs:
* Adjust HasBindMounts definition in mount.go
* Provide implementation in mount_darwin.go
There was no consensus on adding dependency on bindfs, that seems to be the only working solution for bind-mounts on Darwin as of today, in https://github.com/containerd/containerd/pull/8789, that's why the actual implementation is not added in current PR.
As a bonus, Linux FUSE-related code was moved to a separate file and possibly could be reused on FreeBSD, though this needs testing.
Signed-off-by: Marat Radchenko <marat@slonopotamus.org>
Commit 8437c567d8 migrated the use of the
userns package to the github.com/moby/sys/user module.
After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).
This patch migrates our code to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].
This patch deprecates the containerd fork of that package, and adds it as
an alias for the moby/sys/user/userns package.
[1]: https://github.com/opencontainers/runc/commit/3778ae603c706494fd1e2c2faf83b406e38d687d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Aadhar Agarwal