The HEAD request in resolveMetadataRef is an optimization that servers
are not required to support. Its error is already ignored and the code
falls through to the GET, which is retried.
Retrying the HEAD adds nothing, since the GET covers the same failures,
and costs up to ~7s of backoff before the GET even starts.
Signed-off-by: Adam Stokes <adam.stokes@docker.com>
Independent local content stores only coordinate writers within each
store instance. Concurrent cache exports to the same directory can open
the same ingest files, causing rename failures or incomplete writes.
Acquire a filesystem lock per ingest reference before opening a writer
or aborting an ingest. Release it when the writer closes or commits,
including error paths, and report contention as unavailable so existing
retry logic can wait. Preserve references so interrupted uploads remain
resumable, and retain lock files to avoid races with replacement inodes.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
resolveMetadata stored its result into the resolver cache through
release(v) where v was a *Metadata. On a failed resolve or a cache
hit it set the pointer to nil, but passing a typed nil pointer as an
interface stores a non-nil value, so a nil *Metadata was cached for
that remote#ref. A later resolve of the same key read it back and
dereferenced nil, crashing buildkitd. Because the cache is shared
across the jobs on a solve state, an untrusted Dockerfile or frontend
could crash the daemon, and one build could poison another's.
Store an untyped nil instead, matching the HTTP source, so only a
successfully resolved Metadata is ever cached.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 4dd98355a71d79bdf93bd3456fda3a64847ce1de)
(cherry picked from commit c1399f7266310397f5615f4751147f886afff6d2)
(cherry picked from commit 16c6856432)
Verify that a permissive host Git config cannot enable file-backed
submodules in a remote build context, with and without keeping .git.
Confirm that an unrestricted Git checkout can fetch the same submodule
so the test checks protocol enforcement rather than an invalid fixture.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Release checked the wrong error variable, so a failing result release
was swallowed and the proxy was left unreleased.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
When a job merged an edge into another, mergeTo appended the dependency
cache keys to the target edge's secondaryExporters on the scheduler
goroutine. Cache export reads the same slice through the exporter of a
result loaded from that edge, and runs outside the scheduler, e.g. for
build history provenance or cache export while another build is still
running and merging into the shared edge.
Guard secondaryExporters with a mutex and let ExportTo iterate over a
copy, so nested exports run without holding the lock.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Wait for initial frontend data before starting ServeConn so
container startup does not consume the HTTP/2 preface timeout.
Close pipe descriptors on cancellation to unblock the wait.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The HTTP source issues every request with a bare client.Do and returns
the error unchanged, so one dropped connection fails the whole solve.
A solve can hold thousands of independent HTTP sources, each fetched
over its own connection and with little reuse between them, so this is
likely rather than rare on a lossy path. A 2000-URL build reproduces it
within seconds, consistently:
failed to load cache key: Get "http://...": EOF
Registry fetches already retry this class of error through retryhandler.
Use the same helper here.
Only the request is retried. A failure part way through reading the
response body is not covered, and status-code handling is unchanged.
Signed-off-by: Adam Stokes <adam.stokes@docker.com>
- cpu: add amd64 v2/v3/v4 feature bits
- darwin: expose setpriority PRIO_DARWIN_* constants
- unix/darwin: add missing pwritev test case on darwin/arm64
- unix/darwin: add O_EXEC and O_SEARCH on darwin
- unix/darwin: pass fcntl data as unsafe.Pointer in FcntlF{lock,store} on darwin
- unix/linux: don't warn about closed stderr from child process
- unix/linux: unify ppc64x syscall wrappers
- unix: add listns syscall introduced in Linux 6.19
- unix: add Mmsghdr for linux
- unix: add PrctlArg5Ptr
- unix: align the getdents buffer
- unix: extend z/OS support
- unix: keep generated types compilable with current cgo
- unix: pass xattr data as unsafe.Pointer in BSD xattr syscalls
- unix: provide linux/sparc64 assembly for syscalls
- unix: return syscall error in *listxattrNS on BSD
- unix: update glibc to 2.44
- unix: update linux to v7.2
- unix: use the stat64 family and fix Stat_t on linux/sparc64
- windows: drop arm
full diff: https://github.com/golang/sys/compare/v0.48.0...v0.49.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When a build has several exporters, runExporters runs them in parallel
and each one calls the same provenance attestation ContentFunc. Every
call finalized the shared ProvenanceCreator predicate (FinishedOn,
Layers, SysUsage, CompatibilityVersion) while another exporter could be
JSON-encoding it, which the race detector reports as a data race.
Add PredicateJSON, which finalizes and encodes the predicate under a
per-creator mutex and returns the encoded bytes, so no caller keeps a
reference to the shared predicate. Make Predicate unexported so all
callers go through the locked path.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The resize and signal goroutines in runProcess both assigned the err
variable of the enclosing function, so concurrent Resize and Signal
calls on a process raced, and either goroutine could log the other's
error. Use a local error in each goroutine.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
resultProxy.loadResult added the refs of a failed exec to errResults
without holding rp.mu, and borrowed the ExecError so its finalizer would
not release them. If the proxy was released while the load was still
running, for example by the gateway forwarder discarding a lazy ref
whose ReadFile was evaluating it, Release had already run and the refs
were never released. They stayed in use until the daemon restarted, and
prune could not remove them.
Check rp.released under rp.mu after the load and release the refs
right away if the proxy is already gone.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
require.Contains on a map[*Job]struct{} compares the job with every key
using reflect.DeepEqual, which dereferences other jobs and reads their
live progress state while its goroutines run. This raced under -race.
Check membership with a map lookup instead.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
progressWriter.Close set done while other goroutines could still be
writing to the writer, for example a shared vertex writing its snapshot
into the progress of a job that is finishing. Write read the flag
without synchronization. Make done an atomic.Bool.
Add a test that closes a writer while it is being written to; it
reports the race under -race without the fix.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The ExecError of a failed exec is cached by the shared op and returned
to every build that shares the vertex. Each resultProxy then set
OwnerBorrowed on the same error concurrently, which races. Add
ExecError.Borrow that sets the flag under a mutex and use it from
loadResult, Release and the finalizer. Ref ownership is unchanged.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
On the containerd worker the snapshotter is a gRPC client. If a build
is cancelled while finalize is committing a lazy ref, containerd can
apply the commit while the client returns Canceled. finalize then
deleted the lease protecting the committed snapshot and kept
equalMutable, so the record pointed at a mutable snapshot that no
longer existed. Every later build using the record failed with "failed
to stat active key during commit: snapshot ... does not exist" until
the daemon restarted.
Run finalize without cancellation so the commit result is not lost. If
Commit still returns an error, check whether the target snapshot was
committed and, if so, complete the finalize. Keep the lease when that
cannot be determined.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Tõnis Tiigi