Wait for stdout and stderr copying to complete after validating process exit so deferred process deletion does not cancel readers with output still in flight.
Signed-off-by: Chris Ayoub <cayoub@openai.com>
The layer content cache was a single directory, so every source of
pre-converted blobs had to be merged into one tree. A shared read-only fleet
cache, a host-local cache, and a per-image warm cache could not coexist.
Replace layer_content_cache with layer_content_caches, a list. Each directory
is checked in order and the first hit is staged into the snapshot; a layer
found in none of them falls back to the normal download-and-convert path.
Cache directories are no longer required to exist at startup. A missing one is
indistinguishable from an empty one at lookup time (both are simply a miss),
and it may well be mounted after the daemon starts, so the only check left is
that each path is absolute -- a relative one would be symlinked into the
snapshot dir and dangle.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
ctr resolves named users and groups for run and task exec in the
client process. Since EROFS fsview support moved behind registration,
the ctr application does not link the handler and falls back to a
temporary host mount.
Register the handler with the ctr application so EROFS snapshots can
be inspected directly. Non-EROFS mounts continue through the existing
handlers unchanged.
Signed-off-by: Chris Ayoub <cayoub@openai.com>
Adds a new --scrub-logs flag to ctr run on Windows. This flag enables
the ScrubLogs option for the io.containerd.runhcs.v1 shim.
Assisted-by: gemini-cli
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add `ctr images build-erofs-cache <image> <cache_dir>`, which reads an
already-pulled image's layers from the content store, converts each into a
directly-mountable erofs blob, and writes them under
<cache_dir>/<algorithm>/<hex>.erofs keyed by the layer's diffID. This
populates the directory the erofs snapshotter's layer_content_cache reads
on pull; because the key is the source diffID, layers shared across images
converge on one blob. No converted image is produced.
Extract the per-layer uncompress + mkfs.erofs step out of LayerConvertFunc
into an exported ConvertLayerToErofs so the image converter and the cache
builder share a single conversion path.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Mark converted EROFS manifests with the erofs OS feature and cover
feature-aware manifest selection and unpack routing for erofs images.
Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>
The 'content fetch' command description has been unchanged since it was
first added over 8 years ago. It described the feature as "experimental"
with "few leaps to make this work", which no longer reflects reality.
Update the help text to accurately describe what the command does: it
pulls image content into the content store without unpacking, and can
be used to pre-fetch images.
Fixes#11754
Signed-off-by: Crawford Ma <crawfordxx@users.noreply.github.com>
This adds GPU vendor auto-detection from CDI specs instead of hardcoding nvidia.com.
This allows the --gpus flag to work with both NVIDIA and AMD GPUs by detecting the vendor from available CDI spec files.
Signed-off-by: Shiv Tyagi <Shiv.Tyagi@amd.com>
Remove the confusing `(default: false)` that's printed.
Before this:
ctr --help | tail -n2
--help, -h Show help (default: false)
--version, -v Print the version (default: false)
After this:
ctr --help | tail -n2
--help, -h Show help
--version, -v Print the version
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change maps ctr --gpus requests to CDI device requests.
This is done by mapping --gpus ID to a nvidia.com/gpu=ID device
request.
This removes the dependence on the nvidia-container-cli and instead
uses existing CDI specifications for nvidia devices if available on
the system.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
We have individual goroutine for each sandbox container. If there is any
error in handler, that goroutine will put event in that backoff queue.
So we don't need event subscriber for podsandbox. Otherwise, there will
be two goroutines to cleanup sandbox container.
```
>>>> From EventMonitor
time="2025-10-23T19:30:59.626254404Z" level=debug msg="Received containerd event timestamp - 2025-10-23 19:30:59.624494674 +0000 UTC, namespace - \"k8s.io\", topic - \"/tasks/exit\""
time="2025-10-23T19:30:59.626301912Z" level=debug msg="TaskExit event in podsandbox handler container_id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" pid:203121 exit_status:137 exited_at:{seconds:1761247859 nanos:624467824}"
>>> If EventMonitor handles task exit well, it will close ttrpc
connection and then waitSandboxExit could encounter ttrpc-closed error
time="2025-10-23T19:30:59.688031150Z" level=error msg="failed to delete task" error="ttrpc: closed" id=22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf
```
If both task.Delete calls fail but the shim has already been shut down, it
could trigger a new task.Exit event sent by cleanupAfterDeadShim. This would
result in three events in the EventMonitor's backoff queue, which is unnecessary
and could cause confusion due to duplicate events.
The worst-case scenario caused by two concurrent task.Delete calls is a shim
leak. The timeline for this scenario is as follows:
| Timestamp | Component | Action | Result |
| ------ | ----------- | -------- | -------- |
| T1 | EventMonitor | Sends `task.Delete` | Marked as Req-1 |
| T2 | waitSandboxExit | Sends `task.Delete` | Marked as Req-2 |
| T3 | containerd-shim | Handles Req-2 | Container transitions from stopped to deleted |
| T4 | containerd-shim | Handles Req-1 | Fails - container already deleted<br>Returns error: `cannot delete a deleted process: not found` |
| T5 | EventMonitor | Receives `not found` error | - |
| T6 | EventMonitor | Sends `shim.Shutdown` request | No-op (active container record still exists) |
| T7 | EventMonitor | Closes ttrpc connection | Clean container state dir |
| T8 | containerd-shim | Handles Req-2 | Removes container record from memory |
| T9 | waitSandboxExit | Receives error | Error: `ttrpc: closed` |
| T10 | waitSandboxExit | Sends `shim.Shutdown` request | Fails (connection already closed) |
| T11 | waitSandboxExit | Closes ttrpc connection | No-op (already closed) |
The containerd-shim is still running because shim.Shutdown was sent at T6
before T8. Because container's state dir is deleted at T7, it's unable to clean
it up after containerd restarted.
We should avoid concurrent task.Delete calls here.
I also add subcommand - shutdown - in `ctr shim` for debug.
Fixed: #12344
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Currently run will generate invalid specifications on non-Linux Unix
platforms. Linux should be the default container platform for run when
the platform is not explicitly provided.
Signed-off-by: Derek McGowan <derek@mcg.dev>
snapshotter.Mounts() maybe get empty Options for different snapshot service.
Empty Options will produce invalid mount command from printMounts:
```
$ ctr -n flintlock snapshot --snapshotter devmapper mount /mnt flintlock/flintlock/demo-2/01K24ZRN9EFAVQVNGXQS26BYVG/root
mount -t ext4 /dev/mapper/fc-dev-thinpool-snap-19 /mnt -o
$ cmd=$(ctr -n flintlock snapshot --snapshotter devmapper mount /mnt flintlock/flintlock/demo-2/01K24ZRN9EFAVQVNGXQS26BYVG/root)
$ $cmd
mount: option requires an argument -- 'o'
Try 'mount --help' for more information.
```
Signed-off-by: Chenyang Yan <memory.yancy@gmail.com>
The default computed address follows the format used by the
io.containerd.runc.v2 shim, but there is no requirement that third-party
shims follow the same format. The new --shim-address option allows
debugging of third-party shims with a different address format.
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Schema 1 (`application/vnd.docker.distribution.manifest.v1+prettyjws`) has been
officially deprecated since containerd v1.7 (PR 6884), and disabled since v2.0 (PR 9765).
Users who have been seeing warnings like `conversion from schema 1 images is deprecated`
now have to rebuild the image with Schema 2 or OCI.
Schema 2 was introduced in Docker 1.10 (Feb 2016), so most users should have been already
using Schema 2 or OCI.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The CDI device injection spec opt was mistakenly added to the OCI
package which brought in an unintended dependency on CDI and its
transitive dependencies.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Chris Ayoub