urfave/cli/v3 breaks out of its flag parsing loop when it encounters a
"-" positional argument without appending the remaining arguments,
causing "ctr images export - <image>" to drop all image references and
flags following "-".
When <out> is "-", recover the un-truncated arguments from the parent
"images" command lineage and re-parse the tail starting at "-" with a
sentinel placeholder so trailing image references and flags are
preserved.
This workaround can be removed after the bug is fixed upstream:
* https://github.com/urfave/cli/issues/2418
* https://github.com/urfave/cli/pull/2419
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
urfave/cli/v3 scans for flags across all positional arguments by
default, which causes flags meant for the container process to be
parsed as ctr flags.
Configure StopOnNthArg to stop flag parsing before the container
command and its arguments:
- "ctr run" and "ctr containers create": after Image|RootFS and
CONTAINER (2 args)
- "ctr tasks exec": after CONTAINER (1 arg)
Fixes#14208
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
In urfave/cli/v3, child commands do not inherit
DisableSliceFlagSeparator from the root command, unlike in
urfave/cli/v2. When containerd migrated to v3 in commit f72829f36,
subcommands inadvertently reverted to splitting slice flags on commas,
breaking flags that expect comma-separated values such as
--mount type=bind,src=/tmp,dst=/host,options=rbind:ro.
Explicitly set DisableSliceFlagSeparator on all cli.Command structs in
cmd/ctr/commands, restoring the behavior where slice flags are not
split on commas.
Fixes#14181
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
urfave/cli/v3 no longer passes the context through the Command; pass
it explicitly to prepare for the v3 migration.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
urfave/cli/v3 no longer passes the context through the Command; pass
it explicitly to prepare for the v3 migration.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Although urfave/cli v2 recommends constructing apps with cli.NewApp,
App.Run initializes unset fields through App.Setup, making the zero
value usable. Use struct literals to define the application fields
directly without changing behavior.
This prepares for urfave/cli v3, which removes cli.NewApp and replaces
cli.App with cli.Command.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In urfave/cli/v3, cli.Context is replaced with cli.Command, making cmd
a more logical name. Rename the vars ahead of time to make the v3
migration a more mechanical change.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows explicitly setting up the context inside main, and makes the
transition to urfave/cli/v3 more transparent.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Leave the Value field unset for slice flags without default values. The CLI
initializes these flags with an empty value when they are applied, making
the explicit initializations unnecessary.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `Commands` type is removed in v3 and already identical to `[]*Command`
in v2; remove its uses to ease the transition to v3.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Wait for stdout and stderr copying to complete after validating process exit so deferred process deletion does not cancel readers with output still in flight.
Signed-off-by: Chris Ayoub <cayoub@openai.com>
The layer content cache was a single directory, so every source of
pre-converted blobs had to be merged into one tree. A shared read-only fleet
cache, a host-local cache, and a per-image warm cache could not coexist.
Replace layer_content_cache with layer_content_caches, a list. Each directory
is checked in order and the first hit is staged into the snapshot; a layer
found in none of them falls back to the normal download-and-convert path.
Cache directories are no longer required to exist at startup. A missing one is
indistinguishable from an empty one at lookup time (both are simply a miss),
and it may well be mounted after the daemon starts, so the only check left is
that each path is absolute -- a relative one would be symlinked into the
snapshot dir and dangle.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
ctr resolves named users and groups for run and task exec in the
client process. Since EROFS fsview support moved behind registration,
the ctr application does not link the handler and falls back to a
temporary host mount.
Register the handler with the ctr application so EROFS snapshots can
be inspected directly. Non-EROFS mounts continue through the existing
handlers unchanged.
Signed-off-by: Chris Ayoub <cayoub@openai.com>
Adds a new --scrub-logs flag to ctr run on Windows. This flag enables
the ScrubLogs option for the io.containerd.runhcs.v1 shim.
Assisted-by: gemini-cli
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Add a --dmverity flag to `ctr images build-erofs-cache`. When set, each cached
erofs blob is dm-verity formatted (the hash tree is appended in place) and a
.dmverity sidecar is written alongside it. This is required when the erofs
snapshotter runs with dmverity_mode=on, which rejects cache hits that lack a
sidecar; without it such layers would have to be formatted out-of-band.
Extract the differ's dm-verity formatting into a shared dmverity.FormatLayer so
the differ and the cache builder share one implementation; the differ's
formatDmverityLayer now delegates to it.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Add `ctr images build-erofs-cache <image> <cache_dir>`, which reads an
already-pulled image's layers from the content store, converts each into a
directly-mountable erofs blob, and writes them under
<cache_dir>/<algorithm>/<hex>.erofs keyed by the layer's diffID. This
populates the directory the erofs snapshotter's layer_content_cache reads
on pull; because the key is the source diffID, layers shared across images
converge on one blob. No converted image is produced.
Extract the per-layer uncompress + mkfs.erofs step out of LayerConvertFunc
into an exported ConvertLayerToErofs so the image converter and the cache
builder share a single conversion path.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Mark converted EROFS manifests with the erofs OS feature and cover
feature-aware manifest selection and unpack routing for erofs images.
Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>
The 'content fetch' command description has been unchanged since it was
first added over 8 years ago. It described the feature as "experimental"
with "few leaps to make this work", which no longer reflects reality.
Update the help text to accurately describe what the command does: it
pulls image content into the content store without unpacking, and can
be used to pre-fetch images.
Fixes#11754
Signed-off-by: Crawford Ma <crawfordxx@users.noreply.github.com>
This adds GPU vendor auto-detection from CDI specs instead of hardcoding nvidia.com.
This allows the --gpus flag to work with both NVIDIA and AMD GPUs by detecting the vendor from available CDI spec files.
Signed-off-by: Shiv Tyagi <Shiv.Tyagi@amd.com>
Remove the confusing `(default: false)` that's printed.
Before this:
ctr --help | tail -n2
--help, -h Show help (default: false)
--version, -v Print the version (default: false)
After this:
ctr --help | tail -n2
--help, -h Show help
--version, -v Print the version
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This change maps ctr --gpus requests to CDI device requests.
This is done by mapping --gpus ID to a nvidia.com/gpu=ID device
request.
This removes the dependence on the nvidia-container-cli and instead
uses existing CDI specifications for nvidia devices if available on
the system.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
We have individual goroutine for each sandbox container. If there is any
error in handler, that goroutine will put event in that backoff queue.
So we don't need event subscriber for podsandbox. Otherwise, there will
be two goroutines to cleanup sandbox container.
```
>>>> From EventMonitor
time="2025-10-23T19:30:59.626254404Z" level=debug msg="Received containerd event timestamp - 2025-10-23 19:30:59.624494674 +0000 UTC, namespace - \"k8s.io\", topic - \"/tasks/exit\""
time="2025-10-23T19:30:59.626301912Z" level=debug msg="TaskExit event in podsandbox handler container_id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" id:\"22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf\" pid:203121 exit_status:137 exited_at:{seconds:1761247859 nanos:624467824}"
>>> If EventMonitor handles task exit well, it will close ttrpc
connection and then waitSandboxExit could encounter ttrpc-closed error
time="2025-10-23T19:30:59.688031150Z" level=error msg="failed to delete task" error="ttrpc: closed" id=22e15114133e4d461ab380654fb76f3e73d3e0323989c422fa17882762979ccf
```
If both task.Delete calls fail but the shim has already been shut down, it
could trigger a new task.Exit event sent by cleanupAfterDeadShim. This would
result in three events in the EventMonitor's backoff queue, which is unnecessary
and could cause confusion due to duplicate events.
The worst-case scenario caused by two concurrent task.Delete calls is a shim
leak. The timeline for this scenario is as follows:
| Timestamp | Component | Action | Result |
| ------ | ----------- | -------- | -------- |
| T1 | EventMonitor | Sends `task.Delete` | Marked as Req-1 |
| T2 | waitSandboxExit | Sends `task.Delete` | Marked as Req-2 |
| T3 | containerd-shim | Handles Req-2 | Container transitions from stopped to deleted |
| T4 | containerd-shim | Handles Req-1 | Fails - container already deleted<br>Returns error: `cannot delete a deleted process: not found` |
| T5 | EventMonitor | Receives `not found` error | - |
| T6 | EventMonitor | Sends `shim.Shutdown` request | No-op (active container record still exists) |
| T7 | EventMonitor | Closes ttrpc connection | Clean container state dir |
| T8 | containerd-shim | Handles Req-2 | Removes container record from memory |
| T9 | waitSandboxExit | Receives error | Error: `ttrpc: closed` |
| T10 | waitSandboxExit | Sends `shim.Shutdown` request | Fails (connection already closed) |
| T11 | waitSandboxExit | Closes ttrpc connection | No-op (already closed) |
The containerd-shim is still running because shim.Shutdown was sent at T6
before T8. Because container's state dir is deleted at T7, it's unable to clean
it up after containerd restarted.
We should avoid concurrent task.Delete calls here.
I also add subcommand - shutdown - in `ctr shim` for debug.
Fixed: #12344
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Currently run will generate invalid specifications on non-Linux Unix
platforms. Linux should be the default container platform for run when
the platform is not explicitly provided.
Signed-off-by: Derek McGowan <derek@mcg.dev>
snapshotter.Mounts() maybe get empty Options for different snapshot service.
Empty Options will produce invalid mount command from printMounts:
```
$ ctr -n flintlock snapshot --snapshotter devmapper mount /mnt flintlock/flintlock/demo-2/01K24ZRN9EFAVQVNGXQS26BYVG/root
mount -t ext4 /dev/mapper/fc-dev-thinpool-snap-19 /mnt -o
$ cmd=$(ctr -n flintlock snapshot --snapshotter devmapper mount /mnt flintlock/flintlock/demo-2/01K24ZRN9EFAVQVNGXQS26BYVG/root)
$ $cmd
mount: option requires an argument -- 'o'
Try 'mount --help' for more information.
```
Signed-off-by: Chenyang Yan <memory.yancy@gmail.com>
Samuel Karp