When recomputing digests happens, the metadata is lost. This previously
happened only with sources that were affected by source policies so it
wasn't noticed, but now any LLB Op that is rewritten is affected since
the gogo protobuf switch will cause all digests to be rewritten if the
frontend and buildkit are using different versions.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This ensures different valid protobuf serializations that are sent by
frontends will be rewritten into digests that are normalized for the
buildkit solver.
The most recent example of this is that older frontends would generate
protobuf with gogo and the newer buildkit is using the google protobuf
library. These produce different serializations and cause the solver to
think that identical operations are actually different.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This fixes a problem with the new protobuf marshaling with the standard
library. LLB digests are now forced into deterministic marshaling to
ensure they produce the same digest when marshaled multiple times.
In addition, the marshal cache has also been fixed to work in
multi-threaded frontends with multiple different constraints.
Previously, if an LLB vertex was used in multiple goroutines and
marshaled concurrently, the cache would be broken. This could cause
certain problems when a specific node was used multiple times in the
same LLB tree.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
On a build with no-cache, cache mounts were not pruned correctly
if the mount was on top of another ref. This also appeared in
Dockerfile when mode/uid/gid was set because implicit parent
ref is created in these cases in order to change the permissions
of a subdir that is used as a cache mount base.
Because it is not possible to know ahead of time what ref
will become the parent of cache mount during build, all cache
mounts matching the ID that have a parent will be pruned.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Switch to use github.com/containerd/platforms module, because containerd's
platforms package has moved to a separate module. This allows updating the
platforms parsing independent of the containerd module itself.
The package in containerd is deprecated, but kept as an alias to provide
compatibility between codebases.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is more versatile function that works for any source,
not just images.
It can be used together with a policy that switches
between input and output source as well as for adding
additional metadata for other sources in the future.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Ensure interactive calls validate same conditions that
the build requests do. Refactor of the build side is to ensure
we use the same validation function for both cases. There
was no validation issue with the LLB validation.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit d1970522d7145be5f4a1f1a028b1910bb527126c)
Trivially created by looking for every reference to .Variant and adding
OSVersion and OSFeatures, except the ones related to the string
representation of a Platform instance.
I then went through and ensured every assignment of OSFeatures that
might leak out, i.e., not local-only or for marhsalling purposes, uses
the append-to-nil idiom to avoid sharing the slice storage and allowing
accidental mutation after-the-fact.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Sources are a pretty neat extension point, except there are a few code
paths that hard-code against each type. This moves code around and
adjusts interfaces so that Source implementations are self-contained and
merely need to be registered with the source.Manager.
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
Before this, in the case where nothing was mutated the visited memo
would never be updated, thus causing exponential complexity.
Now the memo is updated even when nothing is mutated, just setting old
and new to be the same digest.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Updates the field names for source policies to be a little less
confusing: `Source` -> `Selector`, `Destination` -> `Updates`.
Also collapse `Type` into the `Identifier` which matches how the rest of
buildkit works and greatly simplifies matching.. and generally makes it
so the policy engine doesn't need to care about schemes.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This turns the closure into a proper recursive funciton.
It also ensures there isn't a chance of infinite recursion due to
changes to op inputs.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
fix
Alternative to PR 2816 ("dockerfile: support Dockerfile.pin for pinning sources")
This version is implemented on the llbsolver side and agnostic to the LLB frontends.
See `solver/llbsolver/vertex.go:loadLLB()`.
See `docs/build-repro.md` for the usage.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Change how provenance information is captured from builds.
While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.
All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.
Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.
The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
The `defaultPlatform` got on the build system include the variant for arm v6, v7 etc.
This is an important information for us to find the matched platform, but this information
is missed since the code doesn't include variant when assemble the platform information.
And since "V8" variant for arm64 is normalized to an empty string, so we need to take care
of this as well.
Signed-off-by: Dave Chen <dave.chen@arm.com>
