Right now containerd uses os.StartProcess() to create a dummy ptraced
process via a fork/exec in a new user namespace, with the uid_map/gid_map
setup.
This doesn't work so well with user limits[0] in the kernel, at least if
you expect the container user to have separate isolation from the host's
"normal users". The kernel's ucount mechanism forms a ucount tuple of (ns, kuid_t)
for each user limit. Say containerd runs as uid 0 in the initial user namespace,
and the container runs as uid 100000 from the inital user namespace point of view.
When the container tries to do things like inotify_add_watch(), the
kernel:
1. First verifies that the ucount(container user namespace, 100000)
doesn't exceed its "per user per user namespace limits" and
increments the counter there. This limit is set by
/proc/sys/user/max_inotify_watches when in the namespace
2. Then walks up to the ucount who created this namespace,
ucount(initial user namespace, 0) and increments their counter as
well ensuring they don't exceed their limit. In our example this
is the initial user namespace, this limit is set by
/proc/sys/fs/inotify/max_user_watches as well as
/proc/sys/user/max_inotify_watches in that case.
This is done so a user can't escape per user limits by creating a user
namespace and running as different users in that user namespace. The
accounting always rolls back up to the user who created the user
namespace to ensure this, checking limits at each layer.
This means if you have a rogue container they can consume all of
the containerd user's inotify limits. In practice this means global root
is in danger of being denied usage of resources due to the container,
while other less important users are still well within their budget!
In the inotify case for example systemd will fail to start many new
services due to this.
Let's instead create the user namespace *as the container's initial user
namespace user*. This means that all the attribution for these user
limits rolls up to this unimportant user, preventing one container from
exhausting global root's resources, and further isolating each container
from each other (since each pod in k8s runs as its own disjoint set of
users in the initial user namespace).
This is a bit annoying to do in golang. The best option seems to be what
we have here:
1. Lock OS thread
2. Manually setresuid() (seteuid implementation in golang mimics
glibc's and sets the euid for all of the threads, which causes
issues for other threads running at the same time)
3. Create user namespace
4. Undo the setresuid()
5. Update the uid_map/gid_map as root now (can't do this anymore as
part of (3) since to update the map you must have CAP_SETUID
which the container user will not have)
With this in place you can verify the user namespace is owned by the
proper uid via ioctl(fd, NS_GET_OWNER_UID). Some distros block
unprivleged user namespace creation, for those we'll just continue to
create it as containerd's user.
[0]: https://docs.kernel.org/admin-guide/sysctl/user.html#documentation-for-proc-sys-user
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
UnshareAfterEnterUserns() creates a pidfd via os.StartProcess() with
CLONE_PIDFD but fails to close the file descriptor in any code path,
resulting in a file descriptor leak for every container that uses user
namespace isolation.
The leak occurs because:
- The pidfd is created when PidFD field is set in SysProcAttr
- The original defer block only calls PidfdSendSignal() and
pidfdWaitid()
- No code path calls unix.Close(pidfd) to release the file descriptor
This causes one pidfd leak per container launch when user namespace
isolation is enabled (e.g., Kubernetes pods with hostUsers: false). In
production environments with high container churn, this can exhaust the
system's file descriptor limit.
Fix the leak by adding a defer statement immediately after process
creation that ensures unix.Close(pidfd) is always called, regardless of
which code path is taken. This guarantees cleanup even if the function
returns early due to errors or lack of pidfd support.
This follows the same cleanup pattern already established in
core/mount/mount_idmapped_utils_linux.go:getUsernsFD() which properly
closes its pidfd.
Closes: #12166
Signed-off-by: Jose Fernandez <josef@netflix.com>
Now that we have 1.24.x as go min version, I think
we can remove this go code specific to a lower
version.
Signed-off-by: Jin Dong <djdongjin95@gmail.com>
Andrew Halaney