With the switch from gogo, the `oneof` fields no longer have their
`json:"name"` fields emitted. Protobuf doesn't formally support these
fields and tells users to use the `protojson` package, but we didn't and
the official format is incompatible with the current format we use.
This adds some custom marshaling code to the already existant custom
unmarshaling code to ensure these fields are marshaled with the correct
casing.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This ensures different valid protobuf serializations that are sent by
frontends will be rewritten into digests that are normalized for the
buildkit solver.
The most recent example of this is that older frontends would generate
protobuf with gogo and the newer buildkit is using the google protobuf
library. These produce different serializations and cause the solver to
think that identical operations are actually different.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
The relative paths option for protoc generators doesn't work well when
it comes to dependencies. This simplifies the code generation to avoid
using `go generate` and to use one global command for protoc generation.
This is similar to https://github.com/docker/buildx/pull/2713 since the
same problems with code generation occur here too.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This allows LLB-directed content-based cache enablement for each mount.
Some mounts may not be explicitly unabled (because it would be unsafe) -
for these cases we explicitly error out.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Higher-level clients should be able to check server-side caps to ensure
that when they attempt to use multiple exporters, that the feature is
actually supported.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This modifies how build steps are unmarshaled from JSON into the
provenance attestation. The current method doesn't correctly handle
protobuf attributes that are used with `oneof`.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Trivially created by looking for every reference to .Variant and adding
OSVersion and OSFeatures, except the ones related to the string
representation of a Platform instance.
I then went through and ensured every assignment of OSFeatures that
might leak out, i.e., not local-only or for marhsalling purposes, uses
the append-to-nil idiom to avoid sharing the slice storage and allowing
accidental mutation after-the-fact.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Alternative to PR 2816 ("dockerfile: support Dockerfile.pin for pinning sources")
This version is implemented on the llbsolver side and agnostic to the LLB frontends.
See `solver/llbsolver/vertex.go:loadLLB()`.
See `docs/build-repro.md` for the usage.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This breaking api change refactors the LLB API to prevent reference
mangling and demangling throughout OCI access. Once the session and
store IDs have been determined in the dockerfile frontend, we keep them
the same, and attach them as additional properties.
This has the additional effect of making the actual reference used in
the image resolution arbitrary, since we only parse and access the
digest. The rest of the name can be selected to optimize for log
readability.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We don't need multiple caps for a single feature - the caps that these
were copied by were addded over time, we don't immediately need all of
them to start with, only the main feature one is initially required.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This allows a frontend to request a specific for stubs removal.
By default, if not specified, this will revert to the previous
behaviour. New gateway clients however will set the property to the
desired recursive removal mode.
This property needs to be set for both components that call the
executor: for ExecOp, as well as for the StartContainer API.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Allows reproducible timestamps for layer and image timestamps.
Implemented as a frontend-opt because in the future same
option could be detected by frontend for custom behavior
and same value should also apply timestamps for FileOps.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This adds experimental support for Azure Blob Storage based remote cache to buildkit. For usage instructions please refer to the updated Readme. We have tried to keep it similar to the S3 based implementation while using equivalent Azure Blob Storage specific semantics where appropriate.
This also adds end-to-end tests to exercise the Azure Blob Storage based cache using the [Azurite emulator](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite).
Co-authored-by: Amr Mahdi <amrh@microsoft.com>
Co-authored-by: Pranav Pandit <pranavp@microsoft.com>
Signed-off-by: Pranav Pandit <pranavp@microsoft.com>
This will eventually clients to query the server to confirm if
attestations are supported. However, for now, we can keep it disabled
until the gateway api stabilizes.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Adds a new `WithLayerLimit` option to `llb.Image`
only pulls specified number of layers instead of
full image.
This can be used in combination with DiffOp/MergeOp
to pull any subset of layers from an image in any order.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.
The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this change, if the path provided to FileOp.Rm was a symlink then
the target of the symlink would be removed instead of the symlink
itself. Now, the symlink will be removed instead. However, any symlinks
present in the parent dirs of the specified path will still be resolved
before calling os.Remove; this change only results in the base of the
specified path not being followed.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
