An index can carry more than one manifest that shares a config. A config
lists uncompressed layer digests, so packaging the same layers with
different compression (which `podman manifest push --add-compression`
does) produces manifests with identical configs and the same config
digest.
Right now, when the unpacker visits a manifest, it stores its layers in
a map keyed by its config's digest, so they can be unpacked later once
the config descriptor is visited. Because that map holds only one
manifest's layers per digest, when more than one manifest shares a
config, only the last visited manifest's layers are stored (and
unpacked), and the others' layers are never fetched. Image pulls still
report success, but a later export or push of the image fails with
NotFound.
Fix this by changing the map to store the layers for all manifests
pointing at a specific config (instead of just one manifest). This means
manifests sharing the same config result in more than one set of layers
being stored for the same config digest. Because manifests that share a
config unpack to the same snapshot chain, we only need to unpack the
first manifest's layers, and fetch the others (so their blobs reach the
content store). Unpack skips fetching a layer whose snapshot already
exists, so once a second manifest shares the config, the first
manifest's layers are also fetched explicitly; a config named by a
single manifest keeps that skip. Note that for indexes with more than
one config-sharing manifest, this will now result in additional download
time and storage for those layers.
Don't store anything for a manifest with no layers; an empty slice
reaching the unpack would fail the pull.
(cherry picked from commit 3c5d9fefd8)
Use the release branch's config types and include the registry test
helper needed by the regression tests.
Assisted-by: Claude Code
Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
When unpacking a layer changeset in applyNaive, encountering an opaque
whiteout (.wh..wh..opq) triggers a directory walk to remove files
belonging to lower layers. Since files unpacked earlier in the current
layer are recorded in unpackedPaths and preserved, interleaved files and
opaque whiteouts in the same directory cause repeated walks over the
same growing directory, leading to superlinear execution time.
Once a directory has been purged of lower-layer files by an opaque
whiteout, all existing and subsequent entries in that directory are
guaranteed to be in unpackedPaths. Tracking directories that have
already processed an opaque whiteout allows subsequent opaque whiteouts
for the same directory to return early, restoring linear unpack time.
Benchmark results (n=1,000):
| Metric | Pre-change | Post-change | Delta |
| :--- | :--- | :--- | :--- |
| Execution Time | 2,578.8 ms/op | 135.3 ms/op | ~19.1x faster (94.8% reduction) |
| Allocated Memory | 200.5 MB/op | 2.3 MB/op | ~86.9x less memory (98.8% reduction) |
| Allocs / op | 2,068,820 allocs/op | 48,060 allocs/op | ~43.0x fewer allocs (97.7% reduction) |
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
(cherry picked from commit e6ca378dcf)
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Image config labels are copied onto the container by both the CRI
plugin (BuildLabels) and the client's WithImageConfigLabels option
used by `ctr run`. Labels in the containerd.io/* namespace are
interpreted by containerd itself and labels in the io.cri-containerd*
namespace are interpreted by the CRI plugin. An image config is not a
trusted source for labels in either namespace.
Skip labels in both reserved namespaces when copying labels from an
image config to a container, and warn about each label skipped: an
image that tries to set them may be attempting to alter containerd
behavior. Oversized image labels are already skipped this way by
the CRI plugin.
Labels set explicitly by clients, for example via `ctr run --label`
or in the CRI request, are unaffected.
Verified with the CRI plugin and with `ctr run` against an image
whose config carries labels like these: the labels are no longer
present on the created container and a warning is logged for each.
Assisted-by: Claude Code
Signed-off-by: Ben Cressey <ben@cressey.org>
Signed-off-by: Samuel Karp <samuelkarp@google.com>
(cherry picked from commit 0ec1af4cae)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Port of 7b05ec421 to the release/2.0 call shape, which reads user
databases via path-based user.ParsePasswdFile* / ParseGroupFile*
helpers rather than the fs.FS-based openUserFile wrapper introduced
later on main.
UserFromPath, GIDFromPath, getSupplementalGroupsFromPath, and the
setAdditionalGids closure in WithAppendAdditionalGroups now open
/etc/passwd or /etc/group via openBoundedUserFile, which stats the
resolved path, refuses anything that is not a regular file, and
returns an io.ReadCloser that caps reads at maxUserFileBytes (10
MiB). The reader is consumed by the user.ParsePasswdFilter /
ParseGroupFilter / ParseGroup variants. All four call sites read
either etc/passwd or etc/group; both are regular files on real
systems, well under the cap.
The cap and the regular-file check together bound parser memory use
when reading user-database files of unexpected shape or size.
Adds tests for the cap and for the non-regular file rejection. The
cap test covers three boundary points: a small pad (trailing entry
parsed), a pad placing the entry's last byte exactly on the cap
(still parsed), and a pad past the cap (read returns an "exceeds"
error). The non-regular test verifies a FIFO at etc/group is
rejected before any read is attempted.
Updates TestWithAppendAdditionalGroupsNoEtcGroup to expect "stat
... no such file or directory" instead of "open ...". The
soft-on-missing behavior of WithAppendAdditionalGroups is unchanged.
(cherry picked from commit 7b05ec421d)
Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
Detect strconv.ErrRange and validate uid/gid bounds to avoid falling back to username/group lookups.
Signed-off-by: LEI WANG <ssst0n3@gmail.com>
(cherry picked from commit 85706b6d44)
Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
UnshareAfterEnterUserns() creates a pidfd via os.StartProcess() with
CLONE_PIDFD but fails to close the file descriptor in any code path,
resulting in a file descriptor leak for every container that uses user
namespace isolation.
The leak occurs because:
- The pidfd is created when PidFD field is set in SysProcAttr
- The original defer block only calls PidfdSendSignal() and
pidfdWaitid()
- No code path calls unix.Close(pidfd) to release the file descriptor
This causes one pidfd leak per container launch when user namespace
isolation is enabled (e.g., Kubernetes pods with hostUsers: false). In
production environments with high container churn, this can exhaust the
system's file descriptor limit.
Fix the leak by adding a defer statement immediately after process
creation that ensures unix.Close(pidfd) is always called, regardless of
which code path is taken. This guarantees cleanup even if the function
returns early due to errors or lack of pidfd support.
This follows the same cleanup pattern already established in
core/mount/mount_idmapped_utils_linux.go:getUsernsFD() which properly
closes its pidfd.
Closes: #12166
Signed-off-by: Jose Fernandez <josef@netflix.com>
[Move SupportsPidFD up to handle dupfd in Go 1.23.{0,1} and simplify backport]
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This function has been moved to prevent an unintended dependency on CDI.
Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit bdc847f1eb)
Signed-off-by: Derek McGowan <derek@mcg.dev>
The CDI device injection spec opt was mistakenly added to the OCI
package which brought in an unintended dependency on CDI and its
transitive dependencies.
Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit e20f7f4a24)
Signed-off-by: Derek McGowan <derek@mcg.dev>
Makes the pprof server a plugin and also gates by the `shim_tracing`
build tag (like otel is).
With this change, `net/http` is no longer a dependency in the shim.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This makes it so we don't need to import otelttrpc unless the shim is
compiled with the `shim_tracing` build tag.
This way otel is no longer compiled into the binary at all unless
`shim_tracing` is set.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This adds trace context propagation over the grpc/ttrpc calls to a shim.
It also adds the otlp plugin to the runc shim so that it will send
traces to the configured tracer (which is inherited from containerd's
config).
It doesn't look like this is adding any real overhead to the runc shim's
memory usage, however it does add 2MB to the binary size.
As such this is gated by a build tag `shim_tracing`
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
After these changes, in order to add Darwin bind-mount implementation, one only needs:
* Adjust HasBindMounts definition in mount.go
* Provide implementation in mount_darwin.go
There was no consensus on adding dependency on bindfs, that seems to be the only working solution for bind-mounts on Darwin as of today, in https://github.com/containerd/containerd/pull/8789, that's why the actual implementation is not added in current PR.
As a bonus, Linux FUSE-related code was moved to a separate file and possibly could be reused on FreeBSD, though this needs testing.
Signed-off-by: Marat Radchenko <marat@slonopotamus.org>
Commit 8437c567d8 migrated the use of the
userns package to the github.com/moby/sys/user module.
After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).
This patch migrates our code to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].
This patch deprecates the containerd fork of that package, and adds it as
an alias for the moby/sys/user/userns package.
[1]: https://github.com/opencontainers/runc/commit/3778ae603c706494fd1e2c2faf83b406e38d687d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
There's a couple spots where we know exactly how large
the destination buffer should be, so pre-size these to
avoid any reallocs to a higher capacity.
Signed-off-by: Danny Canter <danny@dcantah.dev>
This functionality is not directly related to containerd and could move
to external package at some point.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Samuel Karp