The OOM handling code is intended to live under pkg/oom/v2. However, the
cgroupv2 package still needs further refinement, such as exporting the
cgroup path and allowing callers to query specific stats instead of
returning all of them.
Until that work is complete, introduce the OOM package as experimental
and place it under containerd-shim-runc-v2.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This change maps ctr --gpus requests to CDI device requests.
This is done by mapping --gpus ID to a nvidia.com/gpu=ID device
request.
This removes the dependence on the nvidia-container-cli and instead
uses existing CDI specifications for nvidia devices if available on
the system.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
- Create /var/lib/containerd with 0o700 (was: 0o711).
- Create config.TempDir with 0o700 (was: 0o711).
- Create /run/containerd/io.containerd.grpc.v1.cri with 0o700 (was: 0o755).
- Create /run/containerd/io.containerd.sandbox.controller.v1.shim with 0o700 (was: 0o711).
- Leave /run/containerd and /run/containerd/io.containerd.runtime.v2.task created with 0o711,
as required by userns-remapped containers.
/run/containerd/io.containerd.runtime.v2.task/<NS>/<ID> is created with:
- 0o700 for non-userns-remapped containers
- 0o710 for userns-remapped containers with the remapped root group as the owner group.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
After runc.Create(init), the container’s cgroup contains a paused init process.
Therefore, we should start monitoring OOM events immediately after creation, in
case the process goes OOM very quickly. Otherwise, we may encounter flaky cases
such as:
```bash
Link https://github.com/containerd/containerd/actions/runs/18787603490/job/53609925150?pr=12399
[k8s.io] Container OOM runtime should output OOMKilled reason [It] should terminate with exitCode 137 and reason OOMKilled
default: sigs.k8s.io/cri-tools/pkg/validate/container_linux.go:147
default:
default: Timeline >>
default: STEP: create Privileged podSandbox @ 10/24/25 18:11:50.889
default: STEP: create container @ 10/24/25 18:11:51.691
default: STEP: create a container that will be killed by OOMKiller @ 10/24/25 18:11:51.691
default: STEP: Get image status for image: registry.k8s.io/e2e-test-images/busybox:1.29-2 @ 10/24/25 18:11:51.691
default: STEP: Create container. @ 10/24/25 18:11:51.691
default: Oct 24 18:11:51.718: INFO: Created container "384bb1538123650106d43dd1fb86fa2c13ae32fb1594b0e70d50cc47eddccbe8"
default:
default: STEP: verifying container status @ 10/24/25 18:11:51.718
default: STEP: start container @ 10/24/25 18:11:51.72
default: STEP: Start container for containerID: 384bb1538123650106d43dd1fb86fa2c13ae32fb1594b0e70d50cc47eddccbe8 @ 10/24/25 18:11:51.72
default: Oct 24 18:11:52.007: INFO: Started container "384bb1538123650106d43dd1fb86fa2c13ae32fb1594b0e70d50cc47eddccbe8"
default:
default: STEP: container is stopped because of OOM @ 10/24/25 18:11:52.007
default: STEP: Get container status for containerID: 384bb1538123650106d43dd1fb86fa2c13ae32fb1594b0e70d50cc47eddccbe8 @ 10/24/25 18:11:52.007
default: STEP: Get container status for containerID: 384bb1538123650106d43dd1fb86fa2c13ae32fb1594b0e70d50cc47eddccbe8 @ 10/24/25 18:11:56.018
default: STEP: Get container status for containerID: 384bb1538123650106d43dd1fb86fa2c13ae32fb1594b0e70d50cc47eddccbe8 @ 10/24/25 18:11:56.022
default: STEP: exit code is 137 @ 10/24/25 18:11:56.023
default: STEP: reason is OOMKilled @ 10/24/25 18:11:56.023
default: [FAILED] in [It] - sigs.k8s.io/cri-tools/pkg/validate/container_linux.go:165 @ 10/24/25 18:11:56.024
default: STEP: stop PodSandbox @ 10/24/25 18:11:56.024
default: STEP: delete PodSandbox @ 10/24/25 18:11:56.932
default: << Timeline
default:
default: [FAILED] Expected
default: <string>: Error
default: to equal
default: <string>: OOMKilled
default: In [It] at: sigs.k8s.io/cri-tools/pkg/validate/container_linux.go:165 @ 10/24/25 18:11:56.024
```
NOTE: We have an EventMonitor goroutine that monitors all containers’ OOMKilled
events, and a dedicated goroutine that monitors container exit events. There is
a race condition where the exit-event goroutine may clean up the task and update
its status faster than the OOM event updater. I don’t have a better idea to
fully resolve this race condition, but this patch aims to minimize the chance
of missing OOM events.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
We have individual goroutine for each sandbox container. If there is any
error in handler, that goroutine will put event in that backoff queue.
So we don't need event subscriber for podsandbox. Otherwise, there will
be two goroutines to cleanup sandbox container.
```
>>>> From EventMonitor
time="2025-10-23T19:30:59.626254404Z" level=debug msg="Received containerd event timestamp - 2025-10-23 19:30:59.624494674 +0000 UTC, namespace - \"k8s.io\", topic - \"/tasks/exit\""
time="2025-10-23T19:30:59.626301912Z" level=debug msg="TaskExit event in podsandbox handler container_id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" pid:203121 exit_status:137 exited_at:{seconds:1761247859 nanos:624467824}"
>>> If EventMonitor handles task exit well, it will close ttrpc
connection and then waitSandboxExit could encounter ttrpc-closed error
time="2025-10-23T19:30:59.688031150Z" level=error msg="failed to delete task" error="ttrpc: closed" id=22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf
```
If both task.Delete calls fail but the shim has already been shut down, it
could trigger a new task.Exit event sent by cleanupAfterDeadShim. This would
result in three events in the EventMonitor's backoff queue, which is unnecessary
and could cause confusion due to duplicate events.
The worst-case scenario caused by two concurrent task.Delete calls is a shim
leak. The timeline for this scenario is as follows:
| Timestamp | Component | Action | Result |
| ------ | ----------- | -------- | -------- |
| T1 | EventMonitor | Sends `task.Delete` | Marked as Req-1 |
| T2 | waitSandboxExit | Sends `task.Delete` | Marked as Req-2 |
| T3 | containerd-shim | Handles Req-2 | Container transitions from stopped to deleted |
| T4 | containerd-shim | Handles Req-1 | Fails - container already deleted<br>Returns error: `cannot delete a deleted process: not found` |
| T5 | EventMonitor | Receives `not found` error | - |
| T6 | EventMonitor | Sends `shim.Shutdown` request | No-op (active container record still exists) |
| T7 | EventMonitor | Closes ttrpc connection | Clean container state dir |
| T8 | containerd-shim | Handles Req-2 | Removes container record from memory |
| T9 | waitSandboxExit | Receives error | Error: `ttrpc: closed` |
| T10 | waitSandboxExit | Sends `shim.Shutdown` request | Fails (connection already closed) |
| T11 | waitSandboxExit | Closes ttrpc connection | No-op (already closed) |
The containerd-shim is still running because shim.Shutdown was sent at T6
before T8. Because container's state dir is deleted at T7, it's unable to clean
it up after containerd restarted.
We should avoid concurrent task.Delete calls here.
I also add subcommand - shutdown - in `ctr shim` for debug.
Fixed: #12344
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Commit ee8ae9d569 ("Update erofs snapshotter to use mount manager")
temporarily removed the file-backed mount feature to adapt to the new
mount manager infrastructure as a quick start.
After the mount manager was introduced, a specific mount type can be
handled with a mount handler plugin to provide a dedicated mount
process (e.g. setup loopback devices in advance or calling external
mount helpers).
This commit adds a default EROFS mount handler for the Linux hosts
to set up loop devices for mount sources and "device=" external file
blobs if necessary (i.e. when file-backed mounts are unavailable),
allowing common runtimes such as runC to work directly, e.g.
``` sh
mount -t erofs /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs \
/run/containerd/io.containerd.mount-manager.v1.bolt/t/346/1
```
will be handled as
``` sh
mount -t erofs /dev/loop1 /run/containerd/io.containerd.mount-manager.v1.bolt/t/346/1
```
and
``` sh
mount -t erofs /var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/7/fsmeta.erofs \
-odevice=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/1/layer.erofs,\
device=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/2/layer.erofs,\
...
device=/var/lib/containerd/io.containerd.snapshotter.v1.erofs/snapshots/7/layer.erofs
/run/containerd/io.containerd.mount-manager.v1.bolt/t/335/1
```
will be handled as
``` sh
mount -t erofs /dev/loop1 -odevice=/dev/loop2,device=/dev/loop3,... \
/run/containerd/io.containerd.mount-manager.v1.bolt/t/335/1
```
if file-backed mounts are unavailable.
For other host platforms (e.g. Darwin hosts) or specific runtimes
that require EROFS raw mounts instead of parsed mounts, this plugin
can be explicitly masked off by users.
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Directly send events for oom rather than starting another go routine as
an event dispatcher. This saves a go routine and needing a map to track
the different oomkill values. The original oom watcher was designed to
match the interface of the much more complicated cgroup v1 oom watcher.
The oom watcher interface is not needed with cgroups v2 manager.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Currently run will generate invalid specifications on non-Linux Unix
platforms. Linux should be the default container platform for run when
the platform is not explicitly provided.
Signed-off-by: Derek McGowan <derek@mcg.dev>
snapshotter.Mounts() maybe get empty Options for different snapshot service.
Empty Options will produce invalid mount command from printMounts:
```
$ ctr -n flintlock snapshot --snapshotter devmapper mount /mnt flintlock/flintlock/demo-2/01K24ZRN9EFAVQVNGXQS26BYVG/root
mount -t ext4 /dev/mapper/fc-dev-thinpool-snap-19 /mnt -o
$ cmd=$(ctr -n flintlock snapshot --snapshotter devmapper mount /mnt flintlock/flintlock/demo-2/01K24ZRN9EFAVQVNGXQS26BYVG/root)
$ $cmd
mount: option requires an argument -- 'o'
Try 'mount --help' for more information.
```
Signed-off-by: Chenyang Yan <memory.yancy@gmail.com>
On Windows, the Go runtime will only ever notify os.Interrupt or
syscall.SIGINT. Using windows.SIGINT, even if it wraps the same integer
value as syscall.SIGINT, will have no effect.
Signed-off-by: Tom Wieczorek <twieczorek@mirantis.com>
We've long been able to use these and they have a couple
great benefits:
1. Forces you to always access them atomically. With the pointer
variants it's completely valid to access the regular ol' int64/uint32
etc. without using the atomic.* methods. These wrappers don't provide
access to the underlying value so it forces correct usage always.
2. Conveys intent much better. Seeing the type be atomic.Int32 immediately
lets the reader know that this var will be used in a concurrent context,
and we no longer need comments like "this MUST be accessed atomically"
or similar.
Signed-off-by: Danny Canter <danny@dcantah.dev>
