Render parsed source identifiers back to their canonical SourceOp form before
source policy evaluation. This lets Git subdir cleanup use the existing source
parser and avoids policy-specific Git parsing.
Add String methods for source identifiers and cover them with unit tests, plus
a client integration regression for canonical Git subdir policy matching.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Use the resolver-cache ref ID directly when loading a cached HTTP snapshot.
This avoids dereferencing missing metadata for cache entries found during a
concurrent solve.
Reset the unauthorized checksum race request counter per solve iteration so
each retry applies the intended first-request delay.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Move the HTTP snapshot resolver-cache lookup into a helper so the lock
release is deferred after acquisition. This keeps invalid cache entries
and digest mismatch errors from returning while the cache lock is held.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Avoid storing typed nil HTTP metadata in the resolver cache after failed
metadata resolution. Also make resolver cache release functions consistently
callable when no error is returned.
Add a client integration test covering concurrent same-URL HTTP sources with
and without checksum when the server returns 401.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add solve-wide compatibility-version support for image and oci
exports, with historical goldens and release compatibility tests.
Backfill version 10 for v0.13-v0.14 git artifact behavior, keep
version 20 as current, and reject unsupported zstd on v10.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Open the snapshot mount as an os.Root and perform file reads
through root-relative APIs in verifySignature and
computeChecksumResponse, consistent with the write path.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Move safeFileName from source/http to source/util/pathutil
and apply it to the containerblob source as well. Harden
containerblob/pull.go to use os.OpenRoot for file writes,
preventing path traversal via crafted filenames.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 3d6e587655d72c343f6fdc7268480a900ba45b0c)
Open the snapshot mount as an os.Root and perform file write/chown/chtimes
through root-relative APIs to keep operations constrained to the mount root.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit d568881c97278d87e4f6f01a1f8a67ad807152bb)
Add safeFileName and route all getFileName sources through it.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 9d117af5ab1e1032f75658884384328fea440843)
Add detached PGP verification for HTTP sources during metadata resolution
and expose LLB options/caps/attrs for signature validation.
Extract shared OpenPGP verification/parsing logic into util/pgpsign and
reuse it from git signing, plus add integration and source-level tests.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
These requests allow computing additional checksum algorithms
for the payload of HTTP sources.
Optionally suffix can be passed that is added to payload. This
is needed to make validation of PGP signatures possible remotely.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
In (*httpSourceHandler).resolveMetadata method there is
a potential resp.Body leak in case when response status code is not
valid (< 200 or >= 400). Also resp.Body is not closed before exiting
if status code is okay.
This commit adds resp.Body closing before exiting from resolveMetadata
method.
Signed-off-by: Mikhail Dmitrichenko <m.dmitrichenko222@gmail.com>
New ResolverCache interface in JobContext allows build jobs to
memorize and synchronize accesses to mutable remote resources.
This is to make sure that when multiple parts of the same build
job, or build job and source metadata resolver access the same
remote resources, it remains the same for the duration of the
single build request, even if data happens to change on the remote side.
Fix such a possible case in the HTTP source. Even if the server
now returns completely different data, if the same URL was accessed
once for the ongoing build, then the initial contents are always
used until the build completes.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Fixes issue where HTTP result can be released after metadata resolve
and rest of the build might pull new (potentially different) data in again.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Convert usages of `github.com/docker/docker/pkg/idtools` to
`github.com/moby/sys/user` in order to break the dependency between
buildkit and docker.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Support additional request header fields for HTTP sources, "Accept" and
"User-Agent" as a start.
The "Accept" header is required in some cases and some servers may vary
the response body based on the header value.
The "User-Agent" header may be useful in custom frontends and
potentially other cases.
- llb: Add `llb.Header` and `HTTPInfo.Header` to allow `client/llb`
users to set these header fields on HTTP sources. The argument to
`llb.Header` is a struct to effectively limit header fields to a
subset.
- llb: Define and flag new `source.http.header` capability when
`llb.Header` is used.
- solver: Define new `http.header.` source attribute prefix. Giving
each header field its own attribute (opposed to JSON encoding the
header struct) will allow source policy to make assertions on
individual header fields.
- source/http: Parse `http.header.` attributes into a sorted slice and
include them in cache key digest.
- source/http: Set request headers accordingly.
Signed-off-by: Dan Duvall <dduvall@wikimedia.org>
Support authentication for HTTP sources.
- llb: Define general `llb.AuthOption` interface composed of
`HTTPOption` and `GitOption`.
- llb: Refactor `llb.AuthHeaderSecret` to return an `llb.AuthOption` so
it may be used with both `llb.Git` and `llb.HTTP`.
- llb: Define `HTTPInfo.AuthHeaderSecret`.
- llb: Define and flag new `source.http.auth` capability when
`HTTPInfo.AuthHeaderSecret` is set.
- solver: Define new `http.auth` source attribute.
- source/http: If an `http.auth` attribute is specified, resolve a
secret named by its value and set the "Authorization" request header.
Signed-off-by: Dan Duvall <dduvall@wikimedia.org>
Currently the lookup from previous local etags was based on
filename. This leads to possibility where (misbehaving) server
may reuse the same eTag for different URLs. While using only
the filename might theoretically create more cache matches when
the same file is used via multiple URLs, I think was accidental
mistake and not intentional.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
There is a possibility to get a digest mismatch error
if the metadata for previous download does not point to
a valid reference anymore.
To mitigate this, check that ref that etag points to
is still valid before using it.
Additionally `.cacheKey` property was not previously
set in the cases where old reference was reused. This
caused a case where even if the download needed to be
performed again, it always failed validation, even if
the digest had not actually changed since previous download.
There is still a small possibility that gc/prune request
will delete the downloaded record in between cachemap and
exec call and that the contents changes in the server
at that exact time. To fix that case we would need to
modify cachemap so that it can keep hold of references
until build is complete.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
We shouldn't use the cachemount root, we should actually properly
use the worker's specified root which is propagated from the config.
Signed-off-by: Justin Chadwell <me@jedevc.com>
On a build with no-cache, cache mounts were not pruned correctly
if the mount was on top of another ref. This also appeared in
Dockerfile when mode/uid/gid was set because implicit parent
ref is created in these cases in order to change the permissions
of a subdir that is used as a cache mount base.
Because it is not possible to know ahead of time what ref
will become the parent of cache mount during build, all cache
mounts matching the ID that have a parent will be pruned.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This enables 3 tests that had been previously skipped
due to the lack of Windows support initially.
The remaining ones in that group, see #4485, will need
some code modifications to be enabled.
Signed-off-by: Anthony Nandaa <profnandaa@gmail.com>
Sources are a pretty neat extension point, except there are a few code
paths that hard-code against each type. This moves code around and
adjusts interfaces so that Source implementations are self-contained and
merely need to be registered with the source.Manager.
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
The go http library will normally implicitly set an Accept-Encoding of
gzip to requests and then transparently decompress the response body.
However, if the Accept-Encoding header is explicitly set by the caller
it will no longer transparently decompress the body. This was causing
HTTP LLB sources to have unexpectedly compressed contents.
The fix here just unsets the Accept-Encoding header after the HEAD
request. Another possible fix would be to do our own gzip decompression
after reading the body if it was gzipped, but this approach seemed
slightly simpler.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
When server returns a weak etag, the prefix can
cause a mismatch on direct comparison with old value.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
