Since Go 1.21, os.MkdirTemp/os.TempDir resolve the temp directory via
Windows' GetTempPath2W. For processes running as SYSTEM (as containerd
does when running under the SCM), that API reads the temp location from
the SystemTemp environment variable rather than TMP/TEMP. As a result,
the existing TMP/TEMP overrides no longer steer the layer-extraction
tempdir for the containerd service, so it falls back to the default
C:\\Windows\\SystemTemp and unpacks on the SystemDrive, reintroducing the
cross-volume copy the 'temp' config option was meant to avoid.
Set SystemTemp to config.TempDir alongside TEMP/TMP so the override keeps
working on Go 1.21+.
Factor the env-var setting out of CreateTopLevelDirectories into a small
setTempDirEnv helper and add a focused unit test (TestSetTempDirEnv) that
verifies the expected variables are set: TEMP/TMP/SystemTemp on Windows,
TMPDIR on other platforms.
Ref: https://cs.opensource.google/go/go/+/refs/tags/go1.21.0:src/os/file_windows.go
Signed-off-by: Maksim An <maksiman@microsoft.com>
(cherry picked from commit faff4d66ba)
Signed-off-by: Maksim An <maksiman@microsoft.com>
Adds a new --scrub-logs flag to ctr run on Windows. This flag enables
the ScrubLogs option for the io.containerd.runhcs.v1 shim.
Assisted-by: gemini-cli
Signed-off-by: Samuel Karp <samuelkarp@google.com>
(cherry picked from commit 18a01c0020)
- Create /var/lib/containerd with 0o700 (was: 0o711).
- Create config.TempDir with 0o700 (was: 0o711).
- Create /run/containerd/io.containerd.grpc.v1.cri with 0o700 (was: 0o755).
- Create /run/containerd/io.containerd.sandbox.controller.v1.shim with 0o700 (was: 0o711).
- Leave /run/containerd and /run/containerd/io.containerd.runtime.v2.task created with 0o711,
as required by userns-remapped containers.
/run/containerd/io.containerd.runtime.v2.task/<NS>/<ID> is created with:
- 0o700 for non-userns-remapped containers
- 0o710 for userns-remapped containers with the remapped root group as the owner group.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 51b0cf11dc5af7ed1919beba259e644138b28d96)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
We have individual goroutine for each sandbox container. If there is any
error in handler, that goroutine will put event in that backoff queue.
So we don't need event subscriber for podsandbox. Otherwise, there will
be two goroutines to cleanup sandbox container.
```
>>>> From EventMonitor
time="2025-10-23T19:30:59.626254404Z" level=debug msg="Received containerd event timestamp - 2025-10-23 19:30:59.624494674 +0000 UTC, namespace - \"k8s.io\", topic - \"/tasks/exit\""
time="2025-10-23T19:30:59.626301912Z" level=debug msg="TaskExit event in podsandbox handler container_id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" pid:203121 exit_status:137 exited_at:{seconds:1761247859 nanos:624467824}"
>>> If EventMonitor handles task exit well, it will close ttrpc
connection and then waitSandboxExit could encounter ttrpc-closed error
time="2025-10-23T19:30:59.688031150Z" level=error msg="failed to delete task" error="ttrpc: closed" id=22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf
```
If both task.Delete calls fail but the shim has already been shut down, it
could trigger a new task.Exit event sent by cleanupAfterDeadShim. This would
result in three events in the EventMonitor's backoff queue, which is unnecessary
and could cause confusion due to duplicate events.
The worst-case scenario caused by two concurrent task.Delete calls is a shim
leak. The timeline for this scenario is as follows:
| Timestamp | Component | Action | Result |
| ------ | ----------- | -------- | -------- |
| T1 | EventMonitor | Sends `task.Delete` | Marked as Req-1 |
| T2 | waitSandboxExit | Sends `task.Delete` | Marked as Req-2 |
| T3 | containerd-shim | Handles Req-2 | Container transitions from stopped to deleted |
| T4 | containerd-shim | Handles Req-1 | Fails - container already deleted<br>Returns error: `cannot delete a deleted process: not found` |
| T5 | EventMonitor | Receives `not found` error | - |
| T6 | EventMonitor | Sends `shim.Shutdown` request | No-op (active container record still exists) |
| T7 | EventMonitor | Closes ttrpc connection | Clean container state dir |
| T8 | containerd-shim | Handles Req-2 | Removes container record from memory |
| T9 | waitSandboxExit | Receives error | Error: `ttrpc: closed` |
| T10 | waitSandboxExit | Sends `shim.Shutdown` request | Fails (connection already closed) |
| T11 | waitSandboxExit | Closes ttrpc connection | No-op (already closed) |
The containerd-shim is still running because shim.Shutdown was sent at T6
before T8. Because container's state dir is deleted at T7, it's unable to clean
it up after containerd restarted.
We should avoid concurrent task.Delete calls here.
I also add subcommand - shutdown - in `ctr shim` for debug.
Fixed: #12344
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 2042e805b8)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
The CDI device injection spec opt was mistakenly added to the OCI
package which brought in an unintended dependency on CDI and its
transitive dependencies.
Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit e20f7f4a24)
Signed-off-by: Derek McGowan <derek@mcg.dev>
Fix issue 11228
`ctr images import --all-platforms` w/o `--local` was failing due to
`unable to initialize unpacker: no unpack platforms defined` error.
W/ `--local`, it unpacks the layers for the strict-default platform.
Now `ctr images import --all-platforms` w/o `--local` unpacks the layers
for the non-strict default platform.
This behavior still differs from `--local`.
i.e., on an arm64 host, arm/v{5,6,7} layers are unpacked too.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Prior to this commit, `/etc/containerd/config.toml` with no version
was parsed as version 3.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
See for rationale the Pull Request description. Added unit test to
demonstrate the difference of this change.
Signed-off-by: Ray Burgemeestre <rayb@nvidia.com>
Uses the new github.com/containerd/errdefs/pkg module which is intended
to hold less stable utility functions separately from the stable
github.com/containerd/errdefs error types.
Includes temporary update to hcsshim until a release is cut there
Signed-off-by: Derek McGowan <derek@mcg.dev>
Makes the pprof server a plugin and also gates by the `shim_tracing`
build tag (like otel is).
With this change, `net/http` is no longer a dependency in the shim.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This is needed so we can build the runc shim without grpc as a
transative dependency.
With this change the runc shim binary went from 14MB to 11MB.
The RSS from an idle shim went from about 17MB to 14MB (back around
where it was in in 1.7).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
pkg/oci is a general utility package with dependency chains that are
uneccessary for the shim.
The shim only actually used it for a convenience function for reading
an oci spec file.
Instead of pulling in those deps just re-implement that internally in
the shim command.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This adds trace context propagation over the grpc/ttrpc calls to a shim.
It also adds the otlp plugin to the runc shim so that it will send
traces to the configured tracer (which is inherited from containerd's
config).
It doesn't look like this is adding any real overhead to the runc shim's
memory usage, however it does add 2MB to the binary size.
As such this is gated by a build tag `shim_tracing`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This commit rewrites and simplifies a lot of this logic to reduce it's
complexity, and also handle the case where the container doesn't have
it's own pid-namespace, which means that we're not guaranteed to receive
the init exit last.
This is achieved by replacing `s.pendingExecs` with `s.runningExecs`,
for which both (previously) pending and de facto running execs are
considered.
The new exit handling logic can be summed up by:
- when we receive an init exit, stash it it in `s.containerInitExit`,
and if a container's init process has exited, refuse new execs.
- (if the container does not have it's own pidns) kill all running
processes (if the container has a private pid-namespace, then all
processes will be dead already).
- wait for the container's running exec count (which includes execs
which have been started but might still early exit) to get to 0.
- publish the stashed away init exit.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
The runc task state machine prevents execs from being created after the
init process has exited, but there are no guards against starting a
created exec after the init process has exited. That leaves a small
window for starting an exec to race our handling of the init process
exiting. Normally this is not an issue in practice: the kernel will
atomically kill all processes in a PID namespace when its "init" process
terminates, and will not allow new processes to fork(2) into the PID
namespace afterwards. Therefore the racing exec is guaranteed by the
kernel to not be running after the init process terminates. On the other
hand, when the container does not have a private PID namespace (i.e. the
container's init process is not the "init" process of the container's
PID namespace), the kernel does not automatically kill other container
processes on init exit and will happily allow runc to start an exec
process at any time. It is the runc shim's responsibility to clean up
the container when the init process exits in this situation by killing
all the container's remaining processes. Block execs from being started
after the container's init process has exited to prevent the processes
from leaking, and to avoid violating the task service's assumption that
an exec can be running iff the init process is also running.
Signed-off-by: Cory Snider <csnider@mirantis.com>
It's not true that `s.mu` needs to be held when calling
`handleProcessExit`, and indeed hasn't been the case for a
while – see 892dc54bd2.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
The main reason is to improve the comment about pidfd in Go 1.23+.
While at it:
- avoid slice manipulation as we only need count;
- avoid repeating "/proc/self/fd".
Updates: #10345.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Update the local content plugin to register itself in a consistent way
as other plugins. This also allows the separate package to define its
own configuration more cleanly.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Commit 8437c567d8 migrated the use of the
userns package to the github.com/moby/sys/user module.
After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).
This patch migrates our code to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When running the test on Ubuntu focal (kernel version 5.4), the
symlink for pidfd is anon_inode:[pidfd].
Updates: #10345
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
Old shims do not implement containerd.task.v3.Task, but it can be
useful to use a new ctr with an older shim especially during upgrade
scenarios.
Signed-off-by: Samuel Karp <samuelkarp@google.com>
The v2 shim interface supports grouping, so a single shim can manage
multiple tasks. Prior to this change, the `shim state` command could
only query the state of the primary task (task that shares the same ID
as the shim).
Signed-off-by: Samuel Karp <samuelkarp@google.com>
There are a couple directories that get created under the default
state directory ("/run/containerd") even when containerd is configured
to use a different location for its state directory. Create the default
state directory even if containerd is configured to use a different
state directory location. This ensure pkg/shim and pkg/fifo won't create
the default state directory with incorrect permissions when calling
os.MkdirAll for their respective subdirectories.
Signed-off-by: Erikson Tung <etung@netflix.com>
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].
This patch deprecates the containerd fork of that package, and adds it as
an alias for the moby/sys/user/userns package.
[1]: https://github.com/opencontainers/runc/commit/3778ae603c706494fd1e2c2faf83b406e38d687d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This PR ignores a new pidfd file descriptor that is introduced in
gotip (future 1.23) and should not be considered when detecting fd leaks.
Fixes#10345
Signed-off-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
Maksim An