Add a FinalizeExport RPC to pass the combined exporter response back to
the session provider after all exports complete. This lets clients run
completion work while the build's gateway references remain available.
Keep the callback optional for compatibility with existing session
exporters. Ignore unsupported finalization, propagate callback failures,
and support providers that only register a finalization callback.
Signed-off-by: Alberto Garcia Hierro <alberto.hierro@docker.com>
Render parsed source identifiers back to their canonical SourceOp form before
source policy evaluation. This lets Git subdir cleanup use the existing source
parser and avoids policy-specific Git parsing.
Add String methods for source identifiers and cover them with unit tests, plus
a client integration regression for canonical Git subdir policy matching.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Bump patch-level dependencies while leaving
packageurl-go unchanged. This updates AWS config/credentials,
smithy-go, klauspost/compress, and sigstore-go, with required
companion bumps from their module constraints.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Anchor rm paths before splitting them so parent traversal is normalized
relative to the fileop root before the final path component is appended.
Add regression coverage for parent traversal attempts and preserve removal of
terminal symlinks without following them.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit cc8b70d02fb890880e79abd44b9a0ad8b7e15d5b)
(cherry picked from commit 3916124aa4)
Reject negative output indexes on LLB input edges while decoding
the definition. This prevents malformed client-provided graphs
from carrying a negative solver edge index into execution.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
(cherry picked from commit c3de4fd617413d6e4b4ecfcd7842a15ef5e2b66d)
(cherry picked from commit 5aefc4a61f)
Malicious or malformed LLB definitions could reference negative or
out-of-range op input indices, or omit required diff lower/upper
inputs, causing the daemon to panic on a slice index or nil pointer
dereference during load and cache-key computation.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 2177ce16335f943e82458958902f9f21f6993d8d)
(cherry picked from commit ed777881d1)
The regression test's comments described the earlier consumer-side fix
(probeResultCache reading a completed dep's result keys). The fix that
landed instead makes processExecReq append an executed edge's committed
keys to edgeState.keys on completion. Reword the mechanism references
accordingly; scenarios and assertions are unchanged and still
discriminate: the HeldCompleteBaseMisses subtests fail without the
append, and the ignore-cache subtest fails without its IgnoreCache
guard.
Also use t.Context() instead of context.TODO() and drop t.Parallel().
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
An edge that completes by executing never adds its committed cache keys
to edgeState.keys: the pre-execution cache query found nothing (that is
why it executed) and Save only attaches the keys to the result. A
consumer subscribing afterwards therefore receives zero keys, finds no
probeable match, latches noCacheMatchPossible and re-executes even
though its own cached record is intact.
Normally such an edge dies with its build; the next build recreates it,
re-queries the now-populated cache and delivers the key normally. But
when a concurrent build keeps the shared edge referenced in
Solver.actives, a later build reuses it already complete, inherits the
empty key list and re-runs steps that should have been cache hits
(moby/buildkit#4674 family).
Append the committed result keys to edgeState.keys on completion so an
executed edge advertises its keys the same way a cache-hit edge does.
Skipped for cache loads, whose record key is already present, and for
ignore-cache edges, so that a --no-cache dependency does not let
consumers match records through a step that was forced to re-run.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
TestCacheMissStaleCompleteSharedDep reproduces the concurrency-triggered
miss in-process (no docker): a producer builds step->base fresh, a holder
job keeps only the shared base alive, the producer is discarded, then a
consumer builds a fresh step over the same (already-complete) base. On
pristine code the consumer re-executes (execCallCount=1) instead of
reusing the cached result; with the probeResultCache fix it reuses (0).
Verified to discriminate: fails on the pre-fix tree, passes with fafe01622.
Signed-off-by: Felix de Souza <fdesouza@palantir.com>
Diagnose the concurrency-triggered cross-build miss: build 2/13 never
queries because its shared base dependency is reused already-complete
(pinned in Solver.actives by a concurrent build) and hands over an
edgeState with zero cache keys, so probeCache/Query is never called and
noCacheMatchPossible latches the dep to complete-only.
Two probes, keyed by vertex digest:
- checkDepMatchPossible: WARN when noCacheMatchPossible flips, logging the
dep's state/keys/keyMap and whether its result still carries a cache key
(dep_result_keys>0 with dep_keymap==0 = the starvation).
- processDepReq: DEBUG on each cache-slow/complete dep delivery, logging
edgeState keys vs result keys, so the fatal complete/0-key delivery in the
concurrent run contrasts with the healthy cache-slow/keyed delivery in the
serial run.
Keeps the earlier probeCache-skip WARN as a proven-negative.
Signed-off-by: Felix de Souza <fdesouza@palantir.com>
bolt documents the array freelist as potentially degrading significantly
for large fragmented databases, while the hashmap freelist is faster in
almost all circumstances. From the [DB.FreelistType] GoDoc:
// FreelistType sets the backend freelist type. There are two options. Array which is simple but endures
// dramatic performance degradation if database is large and fragmentation in freelist is common.
// The alternative one is using hashmap, it is faster in almost all circumstances
// but it doesn't guarantee that it offers the smallest page id available. In normal case it is safe.
// The default type is array
FreelistType FreelistType
While the default is still `FreelistArrayType`, the package shows that
the intent is to make `FreelistMapType` the default in future;
https://pkg.go.dev/go.etcd.io/bbolt@v1.5.0#pkg-constants
// TODO(ahrtr): eventually we should (step by step)
// 1. default to `FreelistMapType`;
// 2. remove the `FreelistArrayType`, do not export `FreelistMapType`
// and remove field `FreelistType' from both `DB` and `Options`;
This keeps the on-disk format unchanged and only changes the in-memory
freelist implementation used after opening the DB.
[DB.FreelistType]: https://pkg.go.dev/go.etcd.io/bbolt@v1.5.0#DB.FreelistType
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Change solver state locking to RWMutex and use read locks for read-only
snapshots of jobs, resolver caches, cache managers, and metadata.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Snapshot the state's job set under state.mu before walking job values.
This matches the lock used by job attach and discard paths and avoids
concurrent map iteration when op resolution loads shared job values.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Ref and rootfs helpers may read paths supplied by frontends or build outputs.
Reject non-regular files before reading them into memory so device and FIFO
paths cannot feed no-EOF streams into parsers or bundle injection logic.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Resolving a username to uid/gid read /etc/passwd and /etc/group via
os.Open with no upper bound, letting a crafted image force unbounded
memory use during user resolution. Cap reads at 10MiB and reject
non-regular files in both the OCI executor and the chown user resolver.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 83cfc1e0ea1dcf8816f259ee6720b8694ab874e5)
The github.com/mitchellh/hashstructure/v2 module was archived, and
there's a maintained fork in the gohugoio org.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Keep exec network modes limited to sandbox, host, and none, and pass proxy
network configuration separately through solve and executor runtime state.
Proxy execs now use bridge-style egress by default, host egress only for host
network mode with entitlement, and no proxy for none mode. Add integration
coverage for bridge, host, and none proxy behavior across OCI and containerd
workers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add a passthrough LLB op that builds all inputs and exposes selected
inputs as outputs. Add State.Requires as a client shortcut for requiring
dependencies without returning their snapshots.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Updating to the lowest release that includes [protobuf@dfab275], which
removed use of the github.com/golang/protobuf/ptypes/timestamp.Timestamp
type alias (deprecated).
[protobuf@dfab275]: dfab275eca
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a proxyNetwork TOML setting and --proxy-network daemon flag to enable
exec proxy enforcement for every build. Wire the default through controller
and solver setup while preserving per-build enablement.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Apply proxy network as an explicit LLB mutation before digest recompute,
while keeping runtime load options such as platform normalization applied
when creating vertices.
This preserves distinct cache keys for proxy-network builds without
breaking gateway warning and source-map lookups that use the original LLB
digests from the frontend.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Record each proxied exec request and print a redacted method and URL list in
the exec progress logs after the process completes.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Route proxy network policy checks through the existing source policy evaluator so
session metadata, deny messages, and URL converts use the same path as LLB
sources. Keep proxy-specific request rewriting in the proxy provider.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Record successful GET responses through the exec proxy as provenance
materials and report incomplete material coverage as a typed solve error.
Thread proxy policy and capture state through typed executor/network options.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add a build request option that rewrites default exec networking to an
internal proxy network while preserving explicit none networking.
Route HTTP and HTTPS traffic through a BuildKit-owned proxy namespace, enforce
source policy checks for proxied requests, and inject a temporary CA into Linux
rootfs trust bundles for HTTPS interception.
Share namespace pooling between CNI and proxy providers, and cover proxy mode
with unit and integration tests.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add compatibility-version 30 for the new OCI media type default while keeping
version 20 pinned to the historical image exporter behavior through v0.31.x.
Track omitted oci-mediatypes separately from explicit false, add v30
compatibility goldens, and document the supported compatibility versions.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
FileOpSolver.getInput wrapped each input cache.ImmutableRef in a fresh
*workerRefResult without cloning, so ExecError.Inputs shared the same
*immutableRef entry with the caller. A release on either side cascaded
into equalMutable.release on a ref the other side still owned, surfacing
as "snapshot does not exist" on a later Finalize -- same shape as the
prior failed-exec output ref fix.
Clone the ref so ExecError owns its own counted ref.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
When an ExecOp exec fails, ExecOp.Exec commits each output mutable
snapshot to an immutable ref, appends it to the results returned to the
solver, AND stores it (via the error-decoration defer) into the
ExecError. The error path stored the SAME *workerRefResult into
ExecError.Mounts that was already handed to the solver, so a single
committed output ref was owned through two independent channels but
backed by only one counted cache ref.
When the error owner is released (e.g. gateway/solver error cleanup),
that single release drains the shared refcount to zero, cascading into
equalMutable.release and deleting the mutable record's lease. The active
snapshot is protected only by that lease, so a subsequent containerd
metadata GC collects it. The surviving solver-side owner is then used as
the upper parent of a Diff, whose Finalize -> Snapshotter.Commit runs on
the already-collected active snapshot, producing:
failed to finalize upper parent during diff: failed to commit <active>
to <final> during finalize: failed to stat active key during commit:
snapshot <active> does not exist: not found
The fix is twofold:
- exec.go: clone the result (res.Clone()) before embedding it in
ExecError.Mounts so the error owner has its own counted ref.
- worker/result.go: make workerRefResult.Clone produce an independent
*WorkerRef (and thus an independent cache ref) instead of sharing the
embedded one and orphaning the original.
Both are required: the call-site clone is what stops the over-release,
and the Clone-method fix is what makes that clone actually independent
(otherwise it aliases the shared *WorkerRef and leaks the original ref).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
RefManager.Commit nilled m.mr unconditionally, so Mount.Release became
a no-op on the failure path while the underlying cache.MutableRef was
never released. Release on failure before clearing m.mr.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The error defer's active-commit loop wrapped the commit error into the
outer err but skipped releasing active.Ref via `continue`. The mutable
was neither put into execMounts nor released, leaking on each failed
Commit.
Release before continue.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
ExecOp.Exec's post-exec commit loop returned (nil, err) when
mutable.Commit failed mid-loop, leaving the already-committed outputs
in the named `results` return clobbered to nil with no holder -- a
permanent leak.
Release them inline instead. An internal commit failure is not a
user-facing exec error, so they don't belong in ExecError.Mounts
either.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Clone the existing slow-cache input result when wrapping errors instead of
creating a separate shared owner. This keeps SlowCacheError.Result tied to
the same release chain and avoids releasing the input while the error still
holds it.
Update the containerd cleanup check to ignore empty leases while still
failing on leases that retain resources, and add a deterministic regression.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
