By restoring the effective capabilities of the thread after setresuid()
we can both:
1. Use the go runtime to setup the uid_map now that we have the
capabilities to do so in the thread again
2. Enable this on distro's which have restrictions around
unprivileged user namespace creation and usage (since the thread
is now privileged)
Let's do it. See [0] for more details on this topic. Unlike
unix::Setresuid()[1], which mimics the glibc implementation and acts on all
threads in the process, unix::Cap{s,g}et() are thread local[2] only as we
want, so we can use that directly.
[0]: https://github.com/containerd/containerd/pull/12317#discussion_r2686960671
[1]: e2fef50def/src/syscall/syscall_linux.go (L1217)
[2]: 6fb913b30f/unix/zsyscall_linux.go (L524)
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
Right now containerd uses os.StartProcess() to create a dummy ptraced
process via a fork/exec in a new user namespace, with the uid_map/gid_map
setup.
This doesn't work so well with user limits[0] in the kernel, at least if
you expect the container user to have separate isolation from the host's
"normal users". The kernel's ucount mechanism forms a ucount tuple of (ns, kuid_t)
for each user limit. Say containerd runs as uid 0 in the initial user namespace,
and the container runs as uid 100000 from the inital user namespace point of view.
When the container tries to do things like inotify_add_watch(), the
kernel:
1. First verifies that the ucount(container user namespace, 100000)
doesn't exceed its "per user per user namespace limits" and
increments the counter there. This limit is set by
/proc/sys/user/max_inotify_watches when in the namespace
2. Then walks up to the ucount who created this namespace,
ucount(initial user namespace, 0) and increments their counter as
well ensuring they don't exceed their limit. In our example this
is the initial user namespace, this limit is set by
/proc/sys/fs/inotify/max_user_watches as well as
/proc/sys/user/max_inotify_watches in that case.
This is done so a user can't escape per user limits by creating a user
namespace and running as different users in that user namespace. The
accounting always rolls back up to the user who created the user
namespace to ensure this, checking limits at each layer.
This means if you have a rogue container they can consume all of
the containerd user's inotify limits. In practice this means global root
is in danger of being denied usage of resources due to the container,
while other less important users are still well within their budget!
In the inotify case for example systemd will fail to start many new
services due to this.
Let's instead create the user namespace *as the container's initial user
namespace user*. This means that all the attribution for these user
limits rolls up to this unimportant user, preventing one container from
exhausting global root's resources, and further isolating each container
from each other (since each pod in k8s runs as its own disjoint set of
users in the initial user namespace).
This is a bit annoying to do in golang. The best option seems to be what
we have here:
1. Lock OS thread
2. Manually setresuid() (seteuid implementation in golang mimics
glibc's and sets the euid for all of the threads, which causes
issues for other threads running at the same time)
3. Create user namespace
4. Undo the setresuid()
5. Update the uid_map/gid_map as root now (can't do this anymore as
part of (3) since to update the map you must have CAP_SETUID
which the container user will not have)
With this in place you can verify the user namespace is owned by the
proper uid via ioctl(fd, NS_GET_OWNER_UID). Some distros block
unprivleged user namespace creation, for those we'll just continue to
create it as containerd's user.
[0]: https://docs.kernel.org/admin-guide/sysctl/user.html#documentation-for-proc-sys-user
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
UnshareAfterEnterUserns() creates a pidfd via os.StartProcess() with
CLONE_PIDFD but fails to close the file descriptor in any code path,
resulting in a file descriptor leak for every container that uses user
namespace isolation.
The leak occurs because:
- The pidfd is created when PidFD field is set in SysProcAttr
- The original defer block only calls PidfdSendSignal() and
pidfdWaitid()
- No code path calls unix.Close(pidfd) to release the file descriptor
This causes one pidfd leak per container launch when user namespace
isolation is enabled (e.g., Kubernetes pods with hostUsers: false). In
production environments with high container churn, this can exhaust the
system's file descriptor limit.
Fix the leak by adding a defer statement immediately after process
creation that ensures unix.Close(pidfd) is always called, regardless of
which code path is taken. This guarantees cleanup even if the function
returns early due to errors or lack of pidfd support.
This follows the same cleanup pattern already established in
core/mount/mount_idmapped_utils_linux.go:getUsernsFD() which properly
closes its pidfd.
Closes: #12166
Signed-off-by: Jose Fernandez <josef@netflix.com>
Now that we have 1.24.x as go min version, I think
we can remove this go code specific to a lower
version.
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
- internal/cri/bandwidth: use lazyregexp to compile regexes on first use
- pkg/identifiers: use lazyregexp to compile regexes on first use
- pkg/progress: use lazyregexp to compile regexes on first use
- pkg/reference: use lazyregexp to compile regexes on first use
- pkg/sys: use lazyregexp to compile regexes on first use
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 8437c567d8 migrated the use of the
userns package to the github.com/moby/sys/user module.
After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).
This patch migrates our code to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].
This patch deprecates the containerd fork of that package, and adds it as
an alias for the moby/sys/user/userns package.
[1]: 3778ae603c
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
We have quite a few pieces of code laying around containerd
that all loop and ignore eintr as they make syscalls directly
(or use a unix/syscall wrapper) because there's no stdlib
equivalent. This adds a small utility to pkg/sys that we can
use for all of these spots.
Signed-off-by: Danny Canter <danny@dcantah.dev>
