This breaking api change refactors the LLB API to prevent reference
mangling and demangling throughout OCI access. Once the session and
store IDs have been determined in the dockerfile frontend, we keep them
the same, and attach them as additional properties.
This has the additional effect of making the actual reference used in
the image resolution arbitrary, since we only parse and access the
digest. The rest of the name can be selected to optimize for log
readability.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Instead of using custom parsing mechansisms for references in
oci-layout, we use containerd's reference.Parse or docker distribution's
reference.Parse (depending on where we do the parsing, and what's
consistent with the file where it's already done). These operations are
neater than manually parsing, and have hopefully more consistent error
messages, and better handling of labels (for if/when those are
introduced).
Signed-off-by: Justin Chadwell <me@jedevc.com>
This mirrors the structure of the names for the local cache directory,
as well as the names for the oci exporter (when using a content store).
This ensures that we cannot encounter name collisions (intentionally or
unintentionally).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Change how provenance information is captured from builds.
While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.
All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.
Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Prevent git commands we run from reading the user or system
configuration, or cloning submodules from the local filesystem.
Signed-off-by: Cory Snider <csnider@mirantis.com>
In the scenario with no session id, then the oci-layout resolver would
still attempt to load a caller with the empty session id. This
inevitably failed, and would fallback to any caller, but this would take
5 seconds to fail with the configured timeout.
In a fresh pull of an OCI image context, this could take up to 15
seconds, as 3 separate calls to the relevant functions would be made.
This patch fixes the issue by correctly identifying this case, and
directly falling through to any caller. Additionally, if a session id is
present, it will always be loaded with no fallback available. To do
this consistently, the helper methods are refactored into a more
consistent withCaller function.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Adds a new `WithLayerLimit` option to `llb.Image`
only pulls specified number of layers instead of
full image.
This can be used in combination with DiffOp/MergeOp
to pull any subset of layers from an image in any order.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
When server returns a weak etag, the prefix can
cause a mismatch on direct comparison with old value.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.
The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This fixes errors such as:
error: cannot update ref 'refs/heads/v2.30.0': trying to write non-commit object 2d9685d47a7e516281aa093bf0cddc8aafa72448 to branch 'refs/heads/v2.30.0'
which occur when cloning a tag rather than branch.
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
If the ref is invalid we are seeing a panic from `umask-git` because
the error type is not always a unix.WaitStatus:
```
#1 0.227 fatal: Not a valid object name 000111222333444555666777888999aaabbbcccd^{commit}
#1 0.229 panic: interface conversion: interface {} is syscall.WaitStatus, not unix.WaitStatus
#1 0.229
#1 0.229 goroutine 1 [running]:
#1 0.229 github.com/moby/buildkit/source/git.gitMain()
#1 0.229 /src/source/git/gitsource_unix.go:66 +0x27d
#1 0.229 github.com/docker/docker/pkg/reexec.Init(...)
#1 0.229 /src/vendor/github.com/docker/docker/pkg/reexec/reexec.go:26
#1 0.229 main.init.0()
#1 0.229 /src/cmd/buildkitd/main.go:76 +0xf6
#1 0.633 fatal: reference is not a tree: 000111222333444555666777888999aaabbbcccd
#1 0.635 panic: interface conversion: interface {} is syscall.WaitStatus, not unix.WaitStatus
#1 0.635
#1 0.635 goroutine 1 [running]:
#1 0.635 github.com/moby/buildkit/source/git.gitMain()
#1 0.635 /src/source/git/gitsource_unix.go:66 +0x27d
#1 0.635 github.com/docker/docker/pkg/reexec.Init(...)
#1 0.635 /src/vendor/github.com/docker/docker/pkg/reexec/reexec.go:26
#1 0.635 main.init.0()
#1 0.635 /src/cmd/buildkitd/main.go:76 +0xf6
```
This is from trying to solve:
```
llb.Git("https://github.com/moby/buildkit.git", "000111222333444555666777888999aaabbbcccd")
```
Signed-off-by: coryb <cbennett@netflix.com>
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This is mostly just preparation for merge-op. The existing
Extract method is updated to be usable for unlazying any type of refs
rather than just lazy blobs. The way views are created is simplified and
centralized in one location.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
The Parent method will no longer make sense with forthcoming Merge and
Diff support as refs will become capable of having multiple parents. It
was also only ever used externally to get the full chain of refs for
each layer in the ref's chain.
The newly added LayerChain method replaces Parents with a method that
just returns a slice of refs for each layer in the ref's chain. This
will work more seamlessly with Merge and Diff (in which case it returns
the "flattened" ancestors of the ref) in addition to being a bit easier
to use for the exiting cases anyways.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
won't make sense soon due to other potential changes. For example,
there can now be multiple blobs associated with a ref (for different
compression types), so the fact that you could access the "Blob"
field from the Info method on Ref incorrectly implied there was just
a single blob for the ref. This is on top of the fact that there is
no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
Previously, many parts of the code outside the cache package could
obtain the bolt storage item for any ref and read/write it directly.
This made it hard to understand what fields are used and when. Now,
the Metadata method has been removed from the Ref interface and
replaced with getters+setters for metadata fields we want to expose
outside the package, which makes it much easier to track and
understand. Similar changes have been made to the metadata search
interface.
3. Use a consistent getter+setter interface for metadata, replacing
the mix of interfaces like Metadata(), Size(), Info() and other
inconsistencies.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This is a safer alternative until we figure out why
http.Transport based limiting fails.
Some connections like cache export/import do not have a
domain key atm and these connections use global pool.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in
Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>
It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource
Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>
Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked.
Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>
