This reverts commit f078cebbd1.
Propagating State deadline errors causes Get to fail and List to omit
tasks when a shim is temporarily unresponsive, such as during slow
cleanup in Delete. A timeout does not mean the task no longer exists,
and omitting it makes container lifecycle management harder for clients
and CRI.
Restore the previous behavior of returning UNKNOWN on State timeouts,
so tasks remain visible and callers can reconcile their state.
Refs: https://github.com/containerd/containerd/pull/14380#pullrequestreview-5689176711
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Init.Delete takes p.mu and holds it across initState.Delete, whose
delete() shuts the terminal console down and waits up to 10s for stdio
to drain, execs `runc delete`, closes the process IO and console and
unmounts the rootfs. Init.Status, ExitStatus and
ExitedAt take the same mutex, so the shim's task.State RPC, which calls
all three, queues behind a Delete. containerd gives State a 2s deadline
(io.containerd.timeout.task.state), so the call fails. Depending on the
containerd version that is logged as "get state for <id>: context
deadline exceeded" and the task reported as UNKNOWN, or the error is
propagated, making Get fail and List drop the task. execProcess has the
identical shape with e.mu.
This is easy to hit when the unmount is slow. On a node with tens of GB
of dirty page cache, umount of the overlayfs rootfs calls
sync_filesystem() on the upper filesystem and was measured at 17.6s with
12.9 GB dirty (0.22s with 1.0 GB). That node logged 4,661 State timeouts
across 284 containers over a window of heavy writes.
This change lets State answer while Delete runs. Delete now releases mu
for the slow cleanup, and every method that is not a plain reader still
waits for the cleanup to end:
- Init and execProcess get a `deleting` channel, read and written only
with mu held. delete() is entered and left with mu held; it sets
`deleting`, unlocks, runs the old body unchanged, and a deferred
relock clears `deleting` and closes the channel, whether the cleanup
succeeded, failed or panicked.
- A new lock() takes mu and, while `deleting` is set, unlocks, waits for
the channel and locks again. Every method that
changes the process (Init and exec alike) now calls lock() instead of
mu.Lock(), so the cleanup is still exclusive with all of them, exactly
as when Delete held mu throughout. They then run
against deletedState if Delete succeeded, or the previous state if it
failed, which is what they saw before.
- Status, ExitStatus and ExitedAt keep taking mu directly and so no
longer wait for the cleanup. They report the state from before the
Delete, since nothing is changed until it ends: a created process
stays created with a zero exit status and exit time, and a stopped
one keeps its real exit status and exit time. There is no
created -> stopped -> created flicker if the Delete fails.
The state machine (init_state.go, exec_state.go, deleted_state.go) is
unchanged, and there is no second lock: readers still take mu, so Status
during an in-flight `runc start` still waits and returns "running",
which CRI recovery relies on (a task seen as created is killed).
Behavior change: State/Status/ExitStatus/ExitedAt no longer wait for the
cleanup part of Delete and report the state from before the Delete. If
the Delete fails nothing changes, and an exit that arrived meanwhile is
applied afterwards, as it was on main. One consequence: while the
Delete of a created task with a non-zero pid runs, a second plain
task.Delete from the client (no WithProcessKill) sees "created" and
fails at once with "task must be stopped before deletion", where it
used to wait for the first Delete, see "stopped" and get NotFound.
Everything else is unchanged: other calls wait for Delete as before, a
second Delete waits and gets NotFound, exit events wait, and the task
service still removes the process only after Delete returns.
Because mu is now released inside delete(), any new method that changes
a process must take mu through lock(); this is documented on mu.
This does not make Delete faster. It still takes as long as before, and
the dirty page cache that makes umount slow is a separate problem.
Timeouts are unchanged.
Tests: Delete is blocked inside an io.Closer; State must return and
report the pre-Delete state, mutating methods must wait, and readers
must take mu.
Assisted-by: Claude Code (Claude Sonnet 5.5)
Signed-off-by: Md Tahsin Rahman <tahsin8243@gmail.com>
(cherry picked from commit a71f15225d)
The test tarball from test/build.sh is built on the kubekins image and
run on other distros. A dynamic runc built on Debian trixie needs
GLIBC_2.38 and fails on AL2023 (kubernetes/kubernetes#142987).
Build with `make static` like the runc release binaries. EL has no
libseccomp-static, so the Fedora/EL VMs keep the dynamic build via
RUNC_DYNAMIC=1.
Signed-off-by: Davanum Srinivas <davanum@gmail.com>
(cherry picked from commit ef4b3a3314)
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
With 1.7 gone the matrix was left testing only 2.0, which is neither the
current LTS nor the release immediately preceding main. RELEASES.md
promises two paths into a given release: a sequential minor upgrade and a
sequential LTS upgrade. For main those are 2.4 and 2.3 respectively, so
test both.
Switch the download to the static release tarball. Release binaries are
built on Ubuntu 22.04, and the dynamically linked containerd from 2.3 on
needs GLIBC_2.32 and GLIBC_2.34. The EL8 hosts kept in the VM
integration matrix for their old glibc only have 2.28, so the previous
release never started there and every case timed out waiting for it. The
2.0 binaries needed nothing newer than GLIBC_2.4, which is why the old
matrix did not hit this. The static build has no libc dependency, and
its shim is the same binary the dynamic tarball already shipped.
Also bump the previous-release config to version 3. Version 2 only
mattered while 1.x was in the matrix, and every release now under test
writes version 3 by default.
kill-shim-before-delete-task needs its expected exit status picked per
version. Once the shim has been killed, containerd reports whatever
`<shim> delete` returns, and manager.Stop prefers the exit status
recorded in the bundle, falling back to a synthesized SIGKILL status when
there is no record. Recording landed in v2.4.0, so a bundle left by 2.3
reports 128+SIGKILL while one left by 2.4 reports the status the task
actually exited with. The old matrix predated recording entirely and only
ever covered the fallback; keying the expectation off the previous
version covers both branches.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
(cherry picked from commit e4e83b3af8)
(release/2.4 tests only 2.3, which is both its preceding minor and its preceding LTS release)
Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
containerd 2.4 is released, and per the documented upgrade path in
RELEASES.md a direct upgrade is only supported between sequential minor
releases or between sequential LTS releases. 1.7 (LTS) -> 2.3 (LTS) is
the last supported jump out of 1.7, so main no longer needs to test it.
Remove the 1.7 arm of TestUpgrade along with the cases that only ever
ran for it:
- recover-ungroupable-shim, which needed the runcv1 runtime handler and
the pre-existing-config variant of the test harness.
- should-address-shim-version-mismatches, which covered the task v2
bootstrap.json downgrade done for 1.7 shims.
- TestIssue10467, which pinned v1.7.20 to verify the sandbox bucket
migration.
With those gone, oneSevenCtrdConfig, the usingExistingConfig plumbing
and the runtime handler parameter are all unused, so drop them too.
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
(cherry picked from commit 35105fc2f0)
Signed-off-by: Chris Henzie <chrishenzie@gmail.com>
The pusher creates the upload request body as an io.Pipe. request.do
attaches the pipe reader to the request before calling Authorize. If
Authorize fails, do returns before client.Do, so the reader is never
handed to the HTTP client, which would have closed it, and nothing else
closes it.
A concurrent pushWriter.Write then blocks forever in pipe.Write. It only
reads the error reported by the request goroutine after the pipe is
closed, and it takes no context, so cancelling the caller does not help.
Callers such as content.Copy never return, and anything they hold, for
example a concurrency slot, is never released.
Close the body when authorization fails so the blocked Write returns the
authorization error. Transient errors keep going through the existing
retry and content.ErrReset handling.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
These releases include 1 security fix following the security policy:
- crypto/x509: mitigation for security concerns with a specific Certificate Authority
We're blocking trust in a small set of intermediate certificates due to specific security concerns around their operation.
Verify will now filter out chains which contain these intermediates, preventing them from being returned to the caller.
We highly recommend immediately upgrading to this release.
This is CVE-2026-111943 and Go issue https://go.dev/issue/82345.
release notes: https://go.dev/doc/devel/release#go1.27.3
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
(cherry picked from commit 9bacf37b79)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Restore support for -1 as an unknown-size sentinel for backward
compatibility. Normalize it to 0, the documented value for unknown
sizes in the Writer interface.
Other negative sizes remain invalid, and known-size copies retain
their existing bounds checks.
Updates 91f62bdd77
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The size comes from the local content store, but validate it as a
defense-in-depth before passing it to content.Copy.
Updates 91f62bdd77
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Push depended on content.Copy to invalidate negative sizes. Check for invalid
descriptor sizes in push itself, without relying on the validation in Copy.
Updates 91f62bdd77
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
WriteBlob expects the size of the content to be known in advance, but
currently relies on Copy to validate the descriptor size.
Check for invalid descriptor sizes in WriteBlob itself, without relying
on the validation in Copy.
Updates 91f62bdd77
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
pullProgressReporter samples activeReqs and totalBytesRead on a periodic
ticker. A response with an empty body, such as a manifest Resolve HEAD
request, does not increment totalBytesRead. In addition, an activeReqs
transition to zero and back to one between ticker ticks is not observed
by the ticker. In both cases, lastSeenTimestamp is not updated and a
subsequent request can be cancelled before it has run for timeout.
Record the timestamp of the most recent request start, response header
receipt, or body byte read in pullRequestReporter, and advance
lastSeenTimestamp from that timestamp on each tick.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Pin workflow runners to ubuntu-24.04 to avoid deprecation warnings and
breaking changes when ubuntu-latest migrates to Ubuntu 26.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
(cherry picked from commit f0e5cfae13)
Signed-off-by: Samuel Karp <samuelkarp@google.com>
limactl start creates the default instance directory before downloading
the guest image. Delete the stopped instance before retrying so the
second limactl start invocation does not fail on an existing instance.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
systemd-tmpfiles-clean.timer runs 15 minutes after boot and removes
empty directories under /tmp with mtimes older than 10 days. Some
integration tests place containerd's root in t.TempDir() and unpack
image layers with older directory timestamps under /tmp. If the timer
fires during extraction, systemd-tmpfiles removes newly created empty
directories before the final directory timestamp pass.
Assisted-by: Antigravity
Signed-off-by: Samuel Karp <samuelkarp@google.com>
Commit records a snapshot in the metadata store, which syncs its own
file, while layer.erofs and the snapshot directory may still be in the
page cache. A power loss shortly after a pull can then leave a committed
snapshot whose blob is corrupt or whose directory is missing. A later
pull does nothing because the chain ID exists, and running the image
fails to mount the layer.
With fsync_on_commit = true, Commit fsyncs everything in the snapshot
directory, the directory itself and snapshots/ before the snapshot is
recorded. Blobs served from a layer content cache are not synced. The
option is off by default and lives in the snapshotter because the
differ's SyncFs does not reach Commit, where the walking differ's blob
is created.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Mykhaylo Sul <mykhaylo.sul@oss.qualcomm.com>
These releases include 15 security fixes following the security policy:
- net/http: HTTP/2 server crash due to HPACK encoder race
HTTP/2 servers could end up crashing due to inadvertently
modifying its HPACK encoder concurrently. This happens because the
server modifies the HPACK encoder from two goroutines without
synchronization: one uses the encoder to encode a HEADERS frame as part
of a response sent to a client and the other modifies the encoder's
table size when handling a SETTINGS frame containing
SETTINGS_HEADER_TABLE_SIZE that a client sends. A malicious client can
repeatedly send a request while changing the header table size to crash
the server.
Fix this issue by not applying SETTINGS_HEADER_TABLE_SIZE immediately.
Instead, buffer any SETTINGS_HEADER_TABLE_SIZE received, and only apply
the new value prior to the next time the server writes a frame.
Thanks to RyotaK (https://ryotak.net) of GMO Flatt Security Inc. for reporting this issue.
This is CVE-2026-97032 and Go issue https://go.dev/issue/81867.
- net/http: HTTP/2 server memory exhaustion due to Trailer headers
When "Trailer" headers are sent by a client, the HTTP server internally
uses the header values to populate the Request.Trailer map passed to the
server handler. Because Request.Trailer is a map, each entry incurs
memory overhead. For HTTP/2 servers, a malicious client can exploit this
by sending a "Trailer" header that declares a large number of fields,
causing the server to allocate a disproportionate amount of memory while
bypassing Server.MaxHeaderValueCount and Server.MaxHeaderBytes limits.
This exploit is not applicable for HTTP/1 servers, which do not support
multiplexing a large number of requests over one TCP connection, and
whose Server.MaxHeaderBytes are calculated differently.
Server.MaxHeaderValueCount and Server.MaxHeaderBytes limits are now
applied towards the trailer fields declared in "Trailer" headers.
Thanks to RyotaK (https://ryotak.net) of GMO Flatt Security Inc. for reporting this issue.
This is CVE-2026-78659 and Go issue https://go.dev/issue/81857.
- crypto/tls: reject malformed ECH outer extension references
Multiple ECH outer extension references are not
permitted under RFC 9849; previously, a client
could send a well-crafted packet that could
trigger memory exhaustion in the server process
by specifying multiple references.
We now reject these as malformed and curb the
memory amplification vector as a result.
This is CVE-2026-97031 and Go issue https://go.dev/issue/81855.
- cmd/go: checksum bypass for golang.org/fips140
Previously, a user operating inside of a malicious
Go project that defines a bogus golang.org/fips140
and operates a malicious GOMODPROXY the user chooses
to connect to can serve an arbitrary module in its
place.
We now unpack the trusted ziphash for the bundled
golang.org/fips140 module and construct its entry
in the GOMODCACHE such that it can be verified by
the toolchain.
This is CVE-2026-94444 and Go issue https://go.dev/issue/81833.
- cmd/go: checksum database bypass for golang.org/toolchain
Previously, a user operating inside of a malicious
Go project that defines a bogus golang.org/toolchain
go.sum entry and operates a malicious GOMODPROXY the
user chooses to use can bypass the intended checksum.
We now ensure that golang.org/toolchain always goes
to the network for the canonical checksum.
This is CVE-2026-94447 and Go issue https://go.dev/issue/81834.
- html/template: reset context tracking on consecutive template expressions
When a JavaScript template literal contains
consecutive expressions, the context tracking
state was not properly reset upon entering a
new expression.
We now ensure that template-literal expression
entries correctly reset context variables so all
subsequent regular expression literals are
accurately recognized and escaped.
This is CVE-2026-94448 and Go issue https://go.dev/issue/81821.
- html/template: recognize yield as regexp preceder keyword
A trusted template author may have previously
written a valid template wherein the use of
the yield keyword would not be correctly
escaped.
We now ensure that valid keyword uses are
escaped and non-keyword uses are not escaped.
This is CVE-2026-97030 and Go issue https://go.dev/issue/81823.
- net/textproto, mime/multipart: memory limit bypass when parsing MIME headers
Parsing a multipart form could bypass memory limits and read an
arbitrarily long line into memory when the remaining limit at the
start of a part was less than 400 bytes.
Multipart form memory limits are now properly enforced in this situation.
Thanks to Jakub Ciolek (https://ciolek.dev) for reporting this issue.
This is CVE-2026-94440 and Go issue https://go.dev/issue/81741.
- net/http: HTTP/1 client connection desynchronization after CONNECT rejection
When http.Transport sends an HTTP/1 CONNECT request with a non-empty
Request.Body, it writes the body directly to the connection without
framing after the request headers. If the server rejects the CONNECT
request with a non-2xx keep-alive response, Transport returns the
connection to the idle pool. Because CONNECT requests do not have a
request body, the server may interpret the trailing body bytes as a
subsequent pipelined HTTP/1.1 request on the connection, leaving the
pooled connection desynchronized and causing the next caller that reuses
it to read the response to the injected request. In reverse proxies
(including httputil.ReverseProxy) that forward CONNECT requests through
a shared Transport, this can lead to cross-user response poisoning.
The HTTP/1 transport now closes a connection after sending a CONNECT
request, regardless of the response status.
In addition, ReverseProxy now rejects incoming CONNECT requests
with a 405 Method Not Allowed response. ReverseProxy has never handled
CONNECT requests in a useful fashion (it does not convert the
connection into a bidirectional tunnel), so we do not expect this
change to negatively affect any current users.
Thanks to Xclow3n (Rajat Raghav) for reporting this issue.
This is CVE-2026-56866 and Go issue https://go.dev/issue/81740.
- net/http: HTTP/1 server connection desynchronization after 2xx CONNECT response
When an HTTP server handler sent a 2xx response to an HTTP/1 CONNECT request
and returned without hijacking the connection, the server improperly continued
to read and serve requests from the connection. Since a 2xx response to an
HTTP/1 CONNECT converts the connection into a tunnel, the server should not
treat the connection as continuing to contain HTTP.
The impact of this misbehavior is mostly limited to potential request smuggling,
where an intermediate proxy considers the data on the connection to be tunneled
and the server considers it to be HTTP.
The HTTP/1 server now always closes a connection after responding to a CONNECT
request, regardless of the response status.
Thanks to Jakub Ciolek (https://ciolek.dev) for reporting this issue.
This is CVE-2026-94439 and Go issue https://go.dev/issue/81744.
- net/http: excessive CPU consumption from repeated initial window changes
A malicious HTTP/2 peer could cause excessive CPU consumption in the
client or server by opening a large number of streams and then sending
many small SETTINGS frames containing SETTINGS_INITIAL_WINDOW_SIZE values.
The HTTP/2 client and server now efficiently handle changes to the
initial window size (O(1) rather than O(number of streams)).
Thanks to Jakub Ciolek (https://ciolek.dev) for reporting this issue.
This is CVE-2026-78669 and Go issue https://go.dev/issue/81742.
- net/http: HTTP/2 transport accepts malformed framing-related headers
Historically, we have been rather lax about malformed framing-related
headers in our HTTP/2 implementation, as they cannot interfere with
HTTP/2 framing. However, this makes it possible for our HTTP/2
implementation to forward responses containing such headers to an HTTP/1
client when acting as a reverse proxy. If the HTTP/1 client also does
not behave strictly enough, this can result in response smuggling.
We now delete malformed framing-related headers when received by our
HTTP/2 transport, so they will not be forwarded to a potentially
vulnerable HTTP/1 client.
Thanks to TJ Barton for reporting this issue.
This is CVE-2026-78660 and Go issue https://go.dev/issue/81115.
- os: Root.Mkdir(All) can follow junctions out of the root on Windows
On Windows, when the target of Root.Mkdir or Root.MkdirAll was a junction
pointing to an empty location, the operation would create a directory at
the junction target even when that target was located outside the root.
This only applies to operations where the last path component is a
junction (path/to/junction, but not path/junction/target).
Root.Mkdir and Root.MkdirAll now correctly avoid resolving junctions.
Thanks to Daniele Ballarini for reporting this issue.
This is CVE-2026-56857 and Go issue https://go.dev/issue/81739.
- net/http: lack of limit on size of parsed Range headers
When parsing a Range header containing a large number of small ranges,
FileServer(FS), ServeContent, and ServeFile(FS) could consume an excessive
amount of CPU.
These functions now ignore Range headers containing more than 200 ranges.
The limit is controlled by the new httpservecontentmaxranges=
GODEBUG setting. Setting GODEBUG=httpservecontentmaxranges=0
disables the limit.
Thanks to Jakub Ciolek (https://ciolek.dev) for reporting this issue.
This is CVE-2026-78667 and Go issue https://go.dev/issue/81858.
- net/http: double flow control refund on HTTP/2 server streams
The HTTP/2 server could refund connection-level flow control twice for the same data:
Once when a client resets a stream (refunding data for any sent-but-unread portion
of the stream), and again when a request handler reads the buffered data.
A malicious client could exploit this to bypass the configured connection-level
flow control limit (MaxReceiveBufferPerConnection). Total buffered data is still
limited by the concurrent stream limit and stream-level flow control.
The HTTP/2 server now waits to refund connection-level flow control for reset
streams until after the request handler is complete.
Thanks to Ali Sherif (https://www.linkedin.com/in/ali-sherif-13812b276/) for reporting this issue.
This is CVE-2026-78663 and Go issue https://go.dev/issue/81743.
release notes: https://go.dev/doc/devel/release#go1.27.2
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit e4ad708faf)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Maksym Pavlenko