To make task.Delete retriable, this patch caches the delete result in
memory after the shim successfully deletes the task. If a later shutdown
attempt fails, a subsequent Delete retry can complete cleanup and return the
cached exit information.
If containerd restarts after deleting the task, the in-memory cache is lost,
and containerd will publish TaskExit and TaskDelete events again to
subscribers. Currently, the exit status returned by shim-delete is always
137. In a follow-up, containerd-shim-runc-v2 should store the actual exit
status in the state directory, so that shim-delete can reuse it instead of
returning a hard-coded value.
Implements: #8981
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Kata's EROFS snapshotter shim set containerd.io/runtime-allow-mounts
before the MountCapabilities bootstrap extension existed
(https://github.com/kata-containers/kata-containers/pull/12763), so
removing the annotation outright would break it and anyone else who
adopted it early with no migration window.
Rather than keep the annotation as a first-class, permanently
supported mechanism, confine falling back to it in
task_mounts_deprecated.go: when a shim's bootstrap result carries no
MountCapabilities extension, its runtime name is checked against the
annotation as a migration path, with the result cached per runtime
name for the life of the daemon, same as the removed mechanism did.
Two runtimes known to never set it, runc.v2 and runhcs.v1, are
skipped outright rather than queried.
The extension stays authoritative whenever a shim provides it: this
fallback only runs in its absence, and is confined to one file so it
can be deleted outright once known early adopters have migrated.
Signed-off-by: Derek McGowan <derek@mcg.dev>
A shim declared the mount types it performs itself through the annotation
removed in an earlier commit, discovered by invoking the shim binary a
second time with -info. Read them instead from the MountCapabilities
extension a shim advertises in its bootstrap result, which is already
available from the shim start that Create() performs anyway.
This requires starting the shim before activating the task's mounts.
Starting a shim does not need the rootfs; only the task.Create call
consumes it. Reordering means a failure after the shim has started must
now tear the shim down, so the teardown that was inlined in the
task.Create error path becomes a deferred cleanup covering the whole
window, reusing the existing cleanupShimTask helper. That defer is
registered after the mount deactivation defer so that the shim, which
may still be using the mounts, is stopped first.
Extract the mount activation logic into a new taskMountController in
task_mounts.go, rather than growing it further inline in Create(): it
already has its own state transitions (owned vs. reused activation,
NotImplemented, AlreadyExists) independent of shim startup, and this
keeps that reasoning testable without spawning a shim.
Also fix two latent nil pointer dereferences: Create() and Delete() both
called the mount.Manager interface directly with no nil check, which
panics whenever the mount manager plugin is not registered.
taskMountController.Activate and Deactivate treat a nil manager as
nothing to do.
Signed-off-by: Derek McGowan <derek@mcg.dev>
A shim declared the mount types it performs itself through a comma
separated string in the generic annotations map of its RuntimeInfo,
discovered by invoking the shim binary a second time with -info. The
values had to reproduce by hand the "<transform>/*" sentinel the mount
manager constructs internally, there was nowhere to put anything but a
name, and it was keyed by runtime name rather than shim instance, so it
could not reflect anything about the environment a particular shim is
running in.
Remove it along with loadShimInfo, shimInfo and the shimInfos cache,
and the runc.v2/runhcs.v1 shortcut that only existed to avoid paying
for it on the default runtimes. This is a breaking change targeted at
2.4; known usage is limited to Nerdbox, which can adopt the shim
capability protocol added in the following commits after this merges.
getRuntimeInfo itself stays: it is also used by PluginInfo and
validateRuntimeFeatures.
Signed-off-by: Derek McGowan <derek@mcg.dev>
TaskService.Shutdown may be invoked multiple times. A grouped shim
itself does not terminate while it still has active tasks. It terminates only
after receiving a Shutdown request when no active tasks remain.
The shim manager should invoke TaskService.Shutdown after deleting every task,
regardless of whether the task represents sandbox or not. This ensures that the
shim receives Shutdown when the last active task is deleted.
In CRI, the pause container uses oom_score_adj=-998, making it unlikely to be
OOM-killed. However, it may still exit or be killed before the other containers
in the sandbox. SandboxService.Shutdown and TaskService.Shutdown are independent
APIs. Sandbox service should make sure there is no active containers
before shutdown. So, for the task service, we should invoke Shutdown for
each task.
Removing the supportSandboxAPIVersion check does not change behavior for shims
using task API versions below v3. Those shims were already forced down the
non-sandboxed path and received Shutdown after every task deletion. The behavior
change applies only to sandboxed tasks using task API v3, which now receive
Shutdown after deletion as well.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
loadShim runs during runtime plugin initialization and issued its task
RPCs on the caller's context, which carries no deadline. A shim that
still owns its socket but has stopped serving, for example one leaked by
a previous containerd whose shutdown timed out, accepts the connection
and then never answers. containerd then never finishes starting and its
API socket is never created.
Give loadShim a single loadTimeout budget covering the whole load. Nested
timeouts can only shorten a deadline, so the load stays bounded however
many calls it makes.
Reaping a shim we give up on runs on its own cleanup budget, detached
from the caller so that a cancelled load cannot leave the shim behind. A
failed delete returns before shutting the shim down and closing its
client, so both are done here, and the failure is now propagated: delete
removes the bundle only when it succeeds, and returning an error lets
loadShims remove it, so a shim that cannot be reaped is no longer
reloaded on every subsequent start.
Co-authored-by: Harshal Patel <106813066+HarshalPatel1972@users.noreply.github.com>
Signed-off-by: Austin Vazquez <austin.vazquez@docker.com>
Correctly handle cases where the mount activation still exists:
- If activation is fully activate, then just return already exists and
allow the caller to return error or call Info to continue.
- If activation is stale or incomplete due to crash during activation,
overwrite the identifier and cleanup the incomplete activation during
activate.
Signed-off-by: Derek McGowan <derek@mcg.dev>
- Create /var/lib/containerd with 0o700 (was: 0o711).
- Create config.TempDir with 0o700 (was: 0o711).
- Create /run/containerd/io.containerd.grpc.v1.cri with 0o700 (was: 0o755).
- Create /run/containerd/io.containerd.sandbox.controller.v1.shim with 0o700 (was: 0o711).
- Leave /run/containerd and /run/containerd/io.containerd.runtime.v2.task created with 0o711,
as required by userns-remapped containers.
/run/containerd/io.containerd.runtime.v2.task/<NS>/<ID> is created with:
- 0o700 for non-userns-remapped containers
- 0o710 for userns-remapped containers with the remapped root group as the owner group.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Allow task manager to fetch info on runtimes at startup.
Use this info to configure whether the runtime allows formatted mounts.
This info could also be used in the future to enforce policy such as
requiring a pre-known set of runtimes or specific runtime properties.
Signed-off-by: Derek McGowan <derek@mcg.dev>
the `cleanup.Background` utility was introduced in f606c4eba7,
at which time the project used go1.19, and Go's stdlib context did not yet
have [`context.WithoutCancel`], which was introduced in go1.21.
This patch replaces `cleanup.Background` for `context.WithoutCancel`, which
is near-identical, and part of go stdlib;
`cleanup.Background`:
type clearCancel struct {
context.Context
}
func (cc clearCancel) Deadline() (deadline time.Time, ok bool) {
return
}
func (cc clearCancel) Done() <-chan struct{} {
return nil
}
func (cc clearCancel) Err() error {
return nil
}
// Background creates a new context which clears out the parent errors
func Background(ctx context.Context) context.Context {
return clearCancel{ctx}
}
`context.WithoutCancel`:
// WithoutCancel returns a derived context that points to the parent context
// and is not canceled when parent is canceled.
// The returned context returns no Deadline or Err, and its Done channel is nil.
// Calling [Cause] on the returned context returns nil.
func WithoutCancel(parent Context) Context {
if parent == nil {
panic("cannot create context from nil parent")
}
return withoutCancelCtx{parent}
}
type withoutCancelCtx struct {
c Context
}
func (withoutCancelCtx) Deadline() (deadline time.Time, ok bool) {
return
}
func (withoutCancelCtx) Done() <-chan struct{} {
return nil
}
func (withoutCancelCtx) Err() error {
return nil
}
func (c withoutCancelCtx) Value(key any) any {
return value(c, key)
}
func (c withoutCancelCtx) String() string {
return contextName(c.c) + ".WithoutCancel"
}
[`context.WithoutCancel`]: https://pkg.go.dev/context@go1.21.0#WithoutCancel
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously, PluginInfo was called with task options as the primary
value, resulting in opts.BinaryName being omitted. Consequently, the
containerd-shim-runc-v2 fell back to the system's runc binary in the
PATH rather than the explicitly specified one. This change inverts the
option fallback by preferring runtime options over task options,
ensuring the correct binary is used for the PluginInfo request.
Closes: https://github.com/containerd/containerd/issues/11169
Signed-off-by: Jose Fernandez <josef@netflix.com>
Reviewed-by: Erikson Tung <etung@netflix.com>
This commit gets rid of the TODO by moving the check to use the
pluginInfo() infrastructure.
The check is only enforced for shims that return info that can be read
as type runtime.Features. For shims that don't provide that, we just
ignore it, as those shims might not be affected by this.
Signed-off-by: Rodrigo Campos <rodrigoca@microsoft.com>
Allow the api to stay at the same v1 go package name and keep using a
1.x version number. This indicates the API is still at 1.x and allows
sharing proto types with containerd 1.6 and 1.7 releases.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Wei Fu