Previously, to address issue #11708, PR #11793 changed containerd to always
invoke the shim binary to establish shim connections, rather than reusing the
sandbox shim. However, this change did not ensure that the Shutdown API was
called to stop the shim process.
Starting with containerd v2.0.0, the Shutdown API is only invoked for sandbox
containers (when container.SandboxID is empty). This approach works for
groupable shims, where multiple containers share a single socket address and
only require a single Shutdown call. However, for non-groupable shims, each
container requires its own Shutdown call during cleanup to avoid leaking shim
processes.
Additionally, PR #11793 introduced a corner case during upgrades:
- T1: An old container-shim-runc-v2 (<=v1.7.X) is running for pod A.
- T2: containerd is upgraded to v2.X.Y.
- T3: A new container A-C1 is created in pod A using the new shim-runc-v2 binary.
- T4: bootstrap.json indicates version:3 protocol, but it is downgraded to version:2 in memory.
- T5: containerd is restarted.
- T6: containerd fails to connect to A-C1.
- T7: The A-C1 container is left in EXITED status in the CRI plugin.
To address this, ensure that loadShimTask downgrades to version:2 if necessary,
and always invoke the Shutdown API for each non-groupable shim during cleanup to
prevent resource leaks and handle upgrade scenarios correctly.
(Introduced by #11793)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
the `cleanup.Background` utility was introduced in f606c4eba7,
at which time the project used go1.19, and Go's stdlib context did not yet
have [`context.WithoutCancel`], which was introduced in go1.21.
This patch replaces `cleanup.Background` for `context.WithoutCancel`, which
is near-identical, and part of go stdlib;
`cleanup.Background`:
type clearCancel struct {
context.Context
}
func (cc clearCancel) Deadline() (deadline time.Time, ok bool) {
return
}
func (cc clearCancel) Done() <-chan struct{} {
return nil
}
func (cc clearCancel) Err() error {
return nil
}
// Background creates a new context which clears out the parent errors
func Background(ctx context.Context) context.Context {
return clearCancel{ctx}
}
`context.WithoutCancel`:
// WithoutCancel returns a derived context that points to the parent context
// and is not canceled when parent is canceled.
// The returned context returns no Deadline or Err, and its Done channel is nil.
// Calling [Cause] on the returned context returns nil.
func WithoutCancel(parent Context) Context {
if parent == nil {
panic("cannot create context from nil parent")
}
return withoutCancelCtx{parent}
}
type withoutCancelCtx struct {
c Context
}
func (withoutCancelCtx) Deadline() (deadline time.Time, ok bool) {
return
}
func (withoutCancelCtx) Done() <-chan struct{} {
return nil
}
func (withoutCancelCtx) Err() error {
return nil
}
func (c withoutCancelCtx) Value(key any) any {
return value(c, key)
}
func (c withoutCancelCtx) String() string {
return contextName(c.c) + ".WithoutCancel"
}
[`context.WithoutCancel`]: https://pkg.go.dev/context@go1.21.0#WithoutCancel
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I noticed that the "cleaning up after shim disconnected" was logged as a
warning, but I couldn't find a reason why it was considered a warning (
and didn't see an error attached to the log);
INFO[2025-05-07T08:57:56.576773500Z] ignoring event container=589c33c88591a9936cb8fd64453e4b0c1357ae2f79d975c34645ff8f0aa10485 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
INFO[2025-05-07T08:57:56.576938958Z] shim disconnected id=589c33c88591a9936cb8fd64453e4b0c1357ae2f79d975c34645ff8f0aa10485 namespace=moby
WARN[2025-05-07T08:57:56.577356208Z] cleaning up after shim disconnected id=589c33c88591a9936cb8fd64453e4b0c1357ae2f79d975c34645ff8f0aa10485 namespace=moby
INFO[2025-05-07T08:57:56.577393417Z] cleaning up dead shim namespace=moby
I think this must've been set as a warning by acccident, so changing
this log to an "info" log as it seems part of the normal flow or operations.
While updating, also updating related logs, such as the "cleaning up dead shim"
log, to include the id, so that these logs can more easily be correlated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This implements container restore as described in:
https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone
For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh
The code changes are based on changes I have done in Podman around 2018
and CRI-O around 2020.
The history behind restoring container via CRI/Kubernetes probably
requires some explanation. The initial proposal to bring
checkpoint/restore to Kubernetes was looking at pod checkpoint and
restoring and the corresponding CRI changes.
https://github.com/kubernetes-sigs/cri-tools/pull/662https://github.com/kubernetes/kubernetes/pull/97194
After discussing this topic for about two years another approach was
implemented as described in KEP-2008:
https://github.com/kubernetes/enhancements/issues/2008
"Forensic Container Checkpointing" allowed us to separate checkpointing
from restoring. For the "Forensic Container Checkpointing" it is enough
to create a checkpoint of the container. Restoring is not necessary as
the analysis of the checkpoint archive can happen without restoring the
container.
While thinking about a way to restore a container it was by coincidence
that we started to look into restoring containers in Kubernetes via
Create and Start. The way it was done in CRI-O is to figure out during
Create if the container image is a checkpoint image and if that is true
we are using another code path. The same was implemented now with this
change in containerd.
With this change it is possible to restore the container from a
checkpoint tar archive that is created during checkpointing via CRI.
To restore a container via Kubernetes we convert the tar archive to an
OCI image as described in the kubernetes.io blog post from above. Using
this OCI image it is possible to restore a container in Kubernetes.
At this point I think it should be doable to restore containers in
CRI-O and containerd no matter if they have been created by containerd or
CRI-O. The biggest difference is the container metadata and that can
be adapted during restore.
Open items:
* It is not clear to me why restoring a container in containerd goes
through task/Create(). But as the restore code already exists this
change extended the existing code path to restore a container in
task/Create() to also restore a container through the CRI via
Create and Start.
* Automatic image pulling. containerd does not pull images
automatically if created via the CRI. There is an option in
crictl to pull images before starting, but that uses the CRI
image pull interface. It is still a separate pull and create
operation. Restoring containers from an OCI image is a bit
different. The checkpoint OCI image does not include the base
image, but just a reference to the image (NAME@DIGEST).
Using crictl with pulling will enable the pulling of the
checkpoint image, but not of the base image the checkpoint is
based on. So during preparation of the checkpoint containerd
will automatically pull the base image, but I was not able how
to pull an image blockingly in containerd. So there is a for
loop waiting for the container image to appear in the internal
store. I think this probably can be implemented better.
Anyway, this is a first step towards container restored in Kubernetes
when using containerd.
Signed-off-by: Adrian Reber <areber@redhat.com>
Previously, PluginInfo was called with task options as the primary
value, resulting in opts.BinaryName being omitted. Consequently, the
containerd-shim-runc-v2 fell back to the system's runc binary in the
PATH rather than the explicitly specified one. This change inverts the
option fallback by preferring runtime options over task options,
ensuring the correct binary is used for the PluginInfo request.
Closes: https://github.com/containerd/containerd/issues/11169
Signed-off-by: Jose Fernandez <josef@netflix.com>
Reviewed-by: Erikson Tung <etung@netflix.com>
Uses the new github.com/containerd/errdefs/pkg module which is intended
to hold less stable utility functions separately from the stable
github.com/containerd/errdefs error types.
Includes temporary update to hcsshim until a release is cut there
Signed-off-by: Derek McGowan <derek@mcg.dev>
This adds trace context propagation over the grpc/ttrpc calls to a shim.
It also adds the otlp plugin to the runc shim so that it will send
traces to the configured tracer (which is inherited from containerd's
config).
It doesn't look like this is adding any real overhead to the runc shim's
memory usage, however it does add 2MB to the binary size.
As such this is gated by a build tag `shim_tracing`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
After these changes, in order to add Darwin bind-mount implementation, one only needs:
* Adjust HasBindMounts definition in mount.go
* Provide implementation in mount_darwin.go
There was no consensus on adding dependency on bindfs, that seems to be the only working solution for bind-mounts on Darwin as of today, in https://github.com/containerd/containerd/pull/8789, that's why the actual implementation is not added in current PR.
As a bonus, Linux FUSE-related code was moved to a separate file and possibly could be reused on FreeBSD, though this needs testing.
Signed-off-by: Marat Radchenko <marat@slonopotamus.org>
This commit gets rid of the TODO by moving the check to use the
pluginInfo() infrastructure.
The check is only enforced for shims that return info that can be read
as type runtime.Features. For shims that don't provide that, we just
ignore it, as those shims might not be affected by this.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Allow the api to stay at the same v1 go package name and keep using a
1.x version number. This indicates the API is still at 1.x and allows
sharing proto types with containerd 1.6 and 1.7 releases.
Signed-off-by: Derek McGowan <derek@mcg.dev>
The new `PlunginInfo()` call can be used for instrospecting the details
of the runtime plugin.
```console
$ ctr plugins inspect-runtime --runtime=io.containerd.runc.v2 --runc-binary=runc
{
"Name": "io.containerd.runc.v2",
"Version": {
"Version": "v2.0.0-beta.0-XX-gXXXXXXXXX.m",
"Revision": "v2.0.0-beta.0-XX-gXXXXXXXXX.m"
},
"Options": {
"binary_name": "runc"
},
"Features": {
"ociVersionMin": "1.0.0",
"ociVersionMax": "1.1.0-rc.2",
...,
},
"Annotations": null
}
```
The shim binary has to support `-info` flag, see `runtime/v2/README.md`
Replaces PR 8509 (`api/services/task: add RuntimeInfo()`)
Co-authored-by: Derek McGowan <derek@mcg.dev>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
