vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Switch to use github.com/containerd/platforms module, because containerd's
platforms package has moved to a separate module. This allows updating the
platforms parsing independent of the containerd module itself.
The package in containerd is deprecated, but kept as an alias to provide
compatibility between codebases.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Backlink walking should not be needed to determine layers for
current build and should be safe to skip. This improves performance
of provenance creation for certain builds.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This is more versatile function that works for any source,
not just images.
It can be used together with a policy that switches
between input and output source as well as for adding
additional metadata for other sources in the future.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Running interactive container APIs was done by giving
the gateway implementation access to worker controller
directly, but it should be passed with a build job instead.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 0971dffaab93d91e51af984b44c745b35b3c5b4d)
This is possible with llb.Scratch in a multi-platform build. We were
accidentally discarding the nil entries in Refs.
Instead, we just skip over nil refs. This isn't ideal - we *should* be
able generate provenance for a nil ref. However, 1. this isn't handled
for llb.Scratch as a single-platform result, and 2. this is tricky,
since the ResultProxy is nil at this point (so there's no ID to match
on). This should be ok for a quick fix, but we can come back and fix
this later.
Signed-off-by: Justin Chadwell <me@jedevc.com>
When a step in the dockerfile is a dependency of multiple other steps in
the dockerfile, the provenance attestation would record the layer chain
for that step multiple times even with the same layer chain.
This is because the provenance attestation reuses the exporter mechanic
and the exporter mechanic would need to visit this same step multiple
times to produce the appropriate cache entries.
Since these duplicate layer chains aren't intentional, this modifies the
provenance attestation capture to detect these duplicates and remove
them.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
ResolveImageConfig can be called concurrently - for example, by
dockerfile2llb during conversion, we loop through each stage and resolve
the base image for that stage.
In the case that two calls to ResolveImageConfig finish at roughly the
same time, we can hit an edge case where we attempt to modify the
bridge's image records at the same time.
To fix this, we just need to use the bridge's mutex to prevent
concurrent access here.
This should fix the following stack trace found in CI:
sandbox.go:144: goroutine 1079 [running]:
sandbox.go:144: github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).ResolveImageConfig(0xc000431e00, {0x1c2b040?, 0xc0008e5b30?}, {0xc00094ba00?, 0xc0003728f0?}, {0x0, 0xc0006cb580, {0x19ba868, 0x7}, {0xc0008f7500, ...}, ...})
sandbox.go:144: /src/solver/llbsolver/provenance.go:139 +0x1fb
sandbox.go:144: github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.func3.1()
sandbox.go:144: /src/frontend/dockerfile/dockerfile2llb/convert.go:405 +0x5fe
sandbox.go:144: golang.org/x/sync/errgroup.(*Group).Go.func1()
sandbox.go:144: /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
sandbox.go:144: created by golang.org/x/sync/errgroup.(*Group).Go
sandbox.go:144: /src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestIntegration/TestNoCache/worker=oci-rootless/frontend=builtin (4.45s)
No other explanation for this failure makes sense - `b` cannot be `nil`
at this point, since a call to `b.llbBridge.ResolveImageConfig` has just
succeeded (also because that would be very strange).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Sources are a pretty neat extension point, except there are a few code
paths that hard-code against each type. This moves code around and
adjusts interfaces so that Source implementations are self-contained and
merely need to be registered with the source.Manager.
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
Before this change, ResolveImageConfig was unaware of source policies.
This means that:
1. Images for denied sources may be resolved
2. Image configs may get pulled for sources that are later converted to
a different image
The update makes it so the image resolver first runs a given ref through
the source policy and uses any mutated ref for the actual resolve
(instead of the original ref).
It also returns the mutated ref so it can be used correctly by the
frontend (e.g. don't want to do llb.Image(oldRef@resolvedDigest)).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The data structures used by llbsolver shouldn't be bundled into the same
package as the actual creation of the build config.
As it was, it wasn't possible to use the data structures (in tests,
external libraries using buildkit, etc) without pulling in more of
buildkit than was probably intended.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This breaking api change refactors the LLB API to prevent reference
mangling and demangling throughout OCI access. Once the session and
store IDs have been determined in the dockerfile frontend, we keep them
the same, and attach them as additional properties.
This has the additional effect of making the actual reference used in
the image resolution arbitrary, since we only parse and access the
digest. The rest of the name can be selected to optimize for log
readability.
Signed-off-by: Justin Chadwell <me@jedevc.com>
It is unsafe to make direct modifications in the
Annotations map of the Remote descriptor. The same
map inside the Descriptor could be used in other
components.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Change how provenance information is captured from builds.
While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.
All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.
Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
