Right now containerd uses os.StartProcess() to create a dummy ptraced
process via a fork/exec in a new user namespace, with the uid_map/gid_map
setup.
This doesn't work so well with user limits[0] in the kernel, at least if
you expect the container user to have separate isolation from the host's
"normal users". The kernel's ucount mechanism forms a ucount tuple of (ns, kuid_t)
for each user limit. Say containerd runs as uid 0 in the initial user namespace,
and the container runs as uid 100000 from the inital user namespace point of view.
When the container tries to do things like inotify_add_watch(), the
kernel:
1. First verifies that the ucount(container user namespace, 100000)
doesn't exceed its "per user per user namespace limits" and
increments the counter there. This limit is set by
/proc/sys/user/max_inotify_watches when in the namespace
2. Then walks up to the ucount who created this namespace,
ucount(initial user namespace, 0) and increments their counter as
well ensuring they don't exceed their limit. In our example this
is the initial user namespace, this limit is set by
/proc/sys/fs/inotify/max_user_watches as well as
/proc/sys/user/max_inotify_watches in that case.
This is done so a user can't escape per user limits by creating a user
namespace and running as different users in that user namespace. The
accounting always rolls back up to the user who created the user
namespace to ensure this, checking limits at each layer.
This means if you have a rogue container they can consume all of
the containerd user's inotify limits. In practice this means global root
is in danger of being denied usage of resources due to the container,
while other less important users are still well within their budget!
In the inotify case for example systemd will fail to start many new
services due to this.
Let's instead create the user namespace *as the container's initial user
namespace user*. This means that all the attribution for these user
limits rolls up to this unimportant user, preventing one container from
exhausting global root's resources, and further isolating each container
from each other (since each pod in k8s runs as its own disjoint set of
users in the initial user namespace).
This is a bit annoying to do in golang. The best option seems to be what
we have here:
1. Lock OS thread
2. Manually setresuid() (seteuid implementation in golang mimics
glibc's and sets the euid for all of the threads, which causes
issues for other threads running at the same time)
3. Create user namespace
4. Undo the setresuid()
5. Update the uid_map/gid_map as root now (can't do this anymore as
part of (3) since to update the map you must have CAP_SETUID
which the container user will not have)
With this in place you can verify the user namespace is owned by the
proper uid via ioctl(fd, NS_GET_OWNER_UID). Some distros block
unprivleged user namespace creation, for those we'll just continue to
create it as containerd's user.
[0]: https://docs.kernel.org/admin-guide/sysctl/user.html#documentation-for-proc-sys-user
Signed-off-by: Andrew Halaney <ahalaney@netflix.com>
Andrew Halaney