When a registry returns 403 Forbidden on a HEAD request (e.g., manifest
resolve or push existence check), the diagnostic error body is lost
because HEAD responses carry no body per HTTP spec. This leaves users
with an opaque "403 Forbidden" message and no actionable guidance.
Add a follow-up GET on HEAD 403 to retrieve the registry's OCI error
body. The existing unexpectedResponseErr machinery already parses the
body into structured errors — it just needs the body to be present. The
fallback lives in a shared withGETErrorBody helper used by both the
pusher and resolver: it only enriches when the GET also returns 403, and
preserves the original HEAD request's method and status while borrowing
just the body, so the resulting error's status and body stay consistent.
Scoped to 403 only because it is rare (CMK key disabled, IP firewall,
RBAC misconfiguration) and its body is highly diagnostic, while other
status codes either already use GET or have bodies that add no value.
Fixes#8969
Signed-off-by: Andrew Au <cshung@gmail.com>
Allow the last host to retry on transient network errors to incrase the
likelihood of the operation succeeding and help reduce flaky tests.
Signed-off-by: Derek McGowan <derek@mcg.dev>
The CRI progress reporter cancels an image pull if it sees no progress
for 5 seconds. It tracks this through active HTTP requests. During
remote fetches, the HTTP response reader is closed via a deferred
call after `content.Copy` completes.
Diagnosis:
`content.Copy` handles both downloading the stream and committing
the writer to the content store. Any delays during the database
commit phase (e.g. from database locks, slow disk syncs, or concurrent
pull deduplication blocks) keep the HTTP connection open. The progress
reporter sees the request is still active (`activeReqs = 1`) but no new
bytes are coming in, leading to a premature timeout cancellation.
Reproduction:
We reproduced this flakiness deterministically on a GCE VM under a
simulated 2 Mbps ingress bandwidth limit using Linux traffic control
ingress policing (`tc filter ... action police rate 2mbit`). Under this
slowness, the download took longer than the progress timeout during the
slow commit phase, triggering context cancellation and failing the
`TestCRIImagePullTimeout/HoldingContentOpenWriterWithLocalPull` test.
Solution:
To fix this, we wrap the HTTP reader in a `closeOnEOFReader` or
`closeOnEOFReadSeeker` before handing it to `content.Copy`. If the
underlying connection reader implements `io.Seeker`, it is dynamically
wrapped in `closeOnEOFReadSeeker` to forward `Seek` operations. This
ensures that O(1) Range seeks are fully preserved during network
resumes or retries. The wrappers automatically close the underlying
network stream as soon as `Read()` returns `io.EOF` (when the download
completes, before the database commit begins). This drops `activeReqs`
to `0` early, freeing the socket and preventing progress timeouts
during commits. A `sync.Once` ensures that subsequent deferred
`Close()` calls do not double-decrement the reporter.
How it was tested:
Verified the fix on a GCE VM under a simulated 2 Mbps ingress
bandwidth limit. Verified seeker safety via standalone logic audits
and trace proofs.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Scary warnings were printed for any mediaType that wasn't mapped;
INFO[2026-03-10T11:39:45.677430346Z] Docker daemon commit=83bca51 containerd-snapshotter=true storage-driver=overlayfs version=29.3.0
...
INFO[2026-03-10T11:39:45.689383471Z] API listen on /var/run/docker.sock
WARN[2026-03-10T11:40:32.382484965Z] reference for unknown type: application/vnd.oci.empty.v1+json digest="sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a" mediatype=application/vnd.oci.empty.v1+json size=2
WARN[2026-03-10T11:40:32.445406215Z] reference for unknown type: application/vnd.oci.empty.v1+json
WARN[2026-03-10T11:40:32.695256132Z] reference for unknown type: application/vnd.dev.sigstore.bundle.v0.3+json digest="sha256:7e8cf55036d9be9d6a0d720a2e78468401cc0e01946dbbcb1cd7622ae854a0a3" mediatype=application/vnd.dev.sigstore.bundle.v0.3+json size=10491
WARN[2026-03-10T11:40:32.695314215Z] reference for unknown type: application/vnd.oci.empty.v1+json digest="sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a" mediatype=application/vnd.oci.empty.v1+json size=2
WARN[2026-03-10T11:40:32.745608673Z] reference for unknown type: application/vnd.dev.sigstore.bundle.v0.3+json
WARN[2026-03-10T11:40:32.745629632Z] reference for unknown type: application/vnd.oci.empty.v1+json
WARN[2026-03-10T11:40:32.845403049Z] reference for unknown type: application/vnd.dev.sigstore.bundle.v0.3+json
WARN[2026-03-10T11:40:32.945182007Z] reference for unknown type: application/vnd.dev.sigstore.bundle.v0.3+json
INFO[2026-03-10T11:40:34.714635924Z] image pulled digest="sha256:37539dd4d60fc70968d164d3850d903a2c56f6402214a1953fbf9fcb81ada731" remote="docker.io/moby/buildkit:latest
These cases are not actionable by the user, and not critical, so let's change
them to a debug-message, but add more context. Also update the GoDoc for this
function to better cover its functionality.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `push` function below assumes that digest and mediatypes are
populated and set. If they aren't, then the requests made are malformed,
attempting to invoke `HEAD /blobs/` (instead of `HEAD /blobs/<digest>`).
Additionally, if we *were* to move past this point, we'd then populate
an empty digest in the query parameter, and even provide invalid HTTP
mediatypes.
However, the `WithDescriptor` `WriterOpt` specifically notes that "Write
does not require any field of desc to be set". It's very easy for the
caller to read this as an optional field, to skip it, and then get
confusing HTTP errors from inside the `push` function.
We can avoid this by explicitly validating that the descriptor is valid
and provide early feedback.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Error strings should not be capitalized (unless beginning with proper
nouns or acronyms) or end with punctuation, per Go style guide and
staticcheck ST1005.
This change lowercases the first letter of error messages in fmt.Errorf
calls throughout the codebase, while preserving uppercase for acronyms
at the start of error strings (e.g., NRI, RDT, CDI, UID, GID, VHD, FUSE).
Also fixes typo 'faild' -> 'failed' in resolver_test.go.
Changes:
- core/metadata: target.Digest, target.Size, target.MediaType
- core/remotes/docker: fetcher.Seek, fetcherByDigest
- integration: various test error messages
- internal/cri/server: containerConfig
- plugins/snapshots: erofs
Fixes#12011
Signed-off-by: Aadhar Agarwal <aadagarwal@microsoft.com>
This reverts commit 4bf1705a88.
This caused issues for downstream clients who were wrapping the type
contingent on it implementing io.ReaderAt.
Consequently this is causing headaches due to increased round trips with
the remote.
Meanwhile I only added this as a convenience for implementing
content.Provider in the remote which can done even without the original
change, just... less conveniently.
In hindsight, this was just a bad change as it has a rather wide impact
and the actual implementation isn't an optimized ReaderAt.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
As the same instance of a map is used in context and mutated
directly, this leads to a situation where:
- Calling WithMediaTypeKeyPrefix from parallel goroutines where
the context was based on the same base context can trigger a panic.
- A subcontext calling WithMediaTypeKeyPrefix changes the value for
another context when they both originate from the same base context.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
If a host is configured that's different to the registry host namespace (e.g. a mirror), then
containerd will append the registry host namespace to requests as a query parameter called `ns`.
Set this query parameter on push requests for proxy mirror support.
Signed-off-by: Will Jordan <will.jordan@gmail.com>
Always attempt the fallback on hosts configured with the referrers
capability. Fallback for any non-404 error.
Signed-off-by: Derek McGowan <derek@mcg.dev>
This commit also adds an info-level log message for the case where the
registry doesn't send a digest header since the computed digest is a
valuable part of our content Commit interface.
Signed-off-by: Max Jonas Werner <max@coppersoft.com>
The distribution spec says this about the Docker-Content-Digest
header:
> These headers are OPTIONAL and clients SHOULD NOT depend on them
This commit changes the behaviour of the `dockerPusher` in a way that
makes it more compliant with that requirement in that it allows for it
to be missing from HTTP responses.
Signed-off-by: Max Jonas Werner <max@coppersoft.com>
Maksym Pavlenko