Support importing git sources from OCI or registry-backed bundle blobs
and exporting resolved checkouts as single-file git bundles.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Allow git sources to fetch a pinned commit without resolving the ref
against the remote tip, while preserving cache keys for canonical
branch refs and covering the behavior with tests.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Normalize Git subdir fragments and validate checkout subdir components
so each segment must be a real directory, preventing traversal and symlink escapes.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 8c994eb561a2646b35352e5663afecd225306214)
Add git.mtime=commit option that normalizes all file, symlink,
and directory mtimes in a git snapshot to the resolved commit
timestamp. This enables reproducible builds from git sources.
When SOURCE_DATE_EPOCH is set in the Dockerfile frontend, the
git context automatically uses commit-time mtimes. The URL
query parameter ?mtime=commit|checkout can override this.
New LLB attr (git.mtime) and capability (source.git.mtime) are
registered as experimental. Cache keys include the mtime policy
so that commit-time and checkout-time snapshots are distinct.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Image blob source in LLB allows addressing a single blob
from a container image registry. The difference from the image
source is that image source needs to point to a manifest that
internally points to an array of layer blobs that are all extracted
on top of each other to form a root FS. Contrary, image blob
points to a single blob that is not extracted but downloaded
as a single file into an empty snapshot, similarily how
the HTTP source works.
The main use case for this source is to pin snapshots of
HTTP URLs, upload the downloaded blob into container registry,
and then use a source policy to map a HTTP URL (whose content
might be changed) to the copy of the source as image blob
to ensure immutability.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add detached PGP verification for HTTP sources during metadata resolution
and expose LLB options/caps/attrs for signature validation.
Extract shared OpenPGP verification/parsing logic into util/pgpsign and
reuse it from git signing, plus add integration and source-level tests.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
ResolveImageConfig was changed to ResolveSourceMetadata long time
ago for cross-source implementation but the worker implementation
was still using old method name with conversions.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This allows images to be pulled by tag and then
checked against the digest. If digest is added directly
to the image reference, then tag is ignored.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Support additional request header fields for HTTP sources, "Accept" and
"User-Agent" as a start.
The "Accept" header is required in some cases and some servers may vary
the response body based on the header value.
The "User-Agent" header may be useful in custom frontends and
potentially other cases.
- llb: Add `llb.Header` and `HTTPInfo.Header` to allow `client/llb`
users to set these header fields on HTTP sources. The argument to
`llb.Header` is a struct to effectively limit header fields to a
subset.
- llb: Define and flag new `source.http.header` capability when
`llb.Header` is used.
- solver: Define new `http.header.` source attribute prefix. Giving
each header field its own attribute (opposed to JSON encoding the
header struct) will allow source policy to make assertions on
individual header fields.
- source/http: Parse `http.header.` attributes into a sorted slice and
include them in cache key digest.
- source/http: Set request headers accordingly.
Signed-off-by: Dan Duvall <dduvall@wikimedia.org>
Support authentication for HTTP sources.
- llb: Define general `llb.AuthOption` interface composed of
`HTTPOption` and `GitOption`.
- llb: Refactor `llb.AuthHeaderSecret` to return an `llb.AuthOption` so
it may be used with both `llb.Git` and `llb.HTTP`.
- llb: Define `HTTPInfo.AuthHeaderSecret`.
- llb: Define and flag new `source.http.auth` capability when
`HTTPInfo.AuthHeaderSecret` is set.
- solver: Define new `http.auth` source attribute.
- source/http: If an `http.auth` attribute is specified, resolve a
secret named by its value and set the "Authorization" request header.
Signed-off-by: Dan Duvall <dduvall@wikimedia.org>
Calling marshal changes the internal state of the op, for example
addCap() helper adds capability constraints. These can race with
same map being read by another Marshal call. Locking the Marshal
function itself also makes sure that the cache is not recomputed
in this case.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This fixes a problem with the new protobuf marshaling with the standard
library. LLB digests are now forced into deterministic marshaling to
ensure they produce the same digest when marshaled multiple times.
In addition, the marshal cache has also been fixed to work in
multi-threaded frontends with multiple different constraints.
Previously, if an LLB vertex was used in multiple goroutines and
marshaled concurrently, the cache would be broken. This could cause
certain problems when a specific node was used multiple times in the
same LLB tree.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This is more versatile function that works for any source,
not just images.
It can be used together with a policy that switches
between input and output source as well as for adding
additional metadata for other sources in the future.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This resolves a regression introduced in
50e75e3565. In this previous patch, I'd
incorrectly assumed that scp-like URLs can express a subset of
"standard"-URLs and so we can always safely convert them for
consistency. This isn't true - the URL "git@example.com:foo" should be
resolved to the home directory of the host, however, the converted URL
"ssh://git@example.com/foo" will be resolved to the root of the host.
To resolve this, we need to not perform this conversion. However, we
also need preserve the behaviour of firm distinction between SCP and
normal URL types (so as to keep proper port parsing).
To do this, we add a new GitURL type to the gitutil package. This new
type contains all useful fields shared in common between the standard
libraries url package and our custom scp-style url parsing package. This
keeps the previous property of a single clean interface to all GitURLs,
while also ensuring that we preserve the original URL to pass to the Git
CLI (making sure we strip fragments out, which are used as
buildkit-level metadata).
As a side-effect of this, the client-side calling code for parsing
git urls is simplified (so we don't have to do fragment wrangling at
every call point).
Signed-off-by: Justin Chadwell <me@jedevc.com>
The "reference" package was moved to a separate module, which was extracted
from b9b19409cf
Also updating docker/docker, which also switched to this new module;
vendor: github.com/docker/docker 032797ea4bcb (v25.0.0-dev)
full diff: afd4805278...032797ea4b
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this change, ResolveImageConfig was unaware of source policies.
This means that:
1. Images for denied sources may be resolved
2. Image configs may get pulled for sources that are later converted to
a different image
The update makes it so the image resolver first runs a given ref through
the source policy and uses any mutated ref for the actual resolve
(instead of the original ref).
It also returns the mutated ref so it can be used correctly by the
frontend (e.g. don't want to do llb.Image(oldRef@resolvedDigest)).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This is just a first round of "would have been useful to me" sort of
documentation and examples for the LLB client API.
This is not in any way exhaustive.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This breaking api change refactors the LLB API to prevent reference
mangling and demangling throughout OCI access. Once the session and
store IDs have been determined in the dockerfile frontend, we keep them
the same, and attach them as additional properties.
This has the additional effect of making the actual reference used in
the image resolution arbitrary, since we only parse and access the
digest. The rest of the name can be selected to optimize for log
readability.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We don't need multiple caps for a single feature - the caps that these
were copied by were addded over time, we don't immediately need all of
them to start with, only the main feature one is initially required.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Instead of using custom parsing mechansisms for references in
oci-layout, we use containerd's reference.Parse or docker distribution's
reference.Parse (depending on where we do the parsing, and what's
consistent with the file where it's already done). These operations are
neater than manually parsing, and have hopefully more consistent error
messages, and better handling of labels (for if/when those are
introduced).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Adds a new `WithLayerLimit` option to `llb.Image`
only pulls specified number of layers instead of
full image.
This can be used in combination with DiffOp/MergeOp
to pull any subset of layers from an image in any order.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
- the git protocol detection is required by buildx, and should reside in
a seperate exported gitutil package.
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
