Add a build request option that rewrites default exec networking to an
internal proxy network while preserving explicit none networking.
Route HTTP and HTTPS traffic through a BuildKit-owned proxy namespace, enforce
source policy checks for proxied requests, and inject a temporary CA into Linux
rootfs trust bundles for HTTPS interception.
Share namespace pooling between CNI and proxy providers, and cover proxy mode
with unit and integration tests.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Add `reset=true` attribute to the local cache exporter that removes
unreferenced blobs from the cache directory after export, preventing
unbounded growth.
Signed-off-by: Jiří Moravčík <jiri.moravcik@gmail.com>
Add solve-wide compatibility-version support for image and oci
exports, with historical goldens and release compatibility tests.
Backfill version 10 for v0.13-v0.14 git artifact behavior, keep
version 20 as current, and reject unsupported zstd on v10.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Make sure that the cache importer doesn't fail the build
if there is issue with the source, only if there is issue
with parameters.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Instead of using a fixed 3-second timeout after solve completion,
use an inactivity-based timeout that waits 5 seconds since the last
received status message. This prevents premature stream closure when
status messages are still being delivered through slow connections
(e.g., proxied environments).
Also ignore context.Canceled errors in the status goroutine to avoid
failing builds when the inactivity timeout triggers - the build itself
succeeded, we just may have missed some final status messages.
Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>
Add support for dynamic source policies via client session.
Client session can allow or deny specific source or
ask additional metadata information via sourcemetaresolver if
that is needed to make the decision.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Devices can be marked as "automatically allowed" by TOML config
or by the CDI spec of specific file via annotation.
Device that is is not "automatically allowed" needs to be allowed
by the build request by passing entitlement. For example a Dockerfile
may not use a device without use invoking the build permitting it.
--allow device grants access to any device.
--allow device=kind|name grants access to specific device.
--allow device=kind|name,alias=kind|name allows mapping kind to
a specific device or one device to another. Alias is the name requested
by the build and device is the actual device that is being enabled.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This seems to be completely unused.
I believe it is remnant of pre-buildkit session implementation
and was used for either logging of some transfer reuse.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
We can derive exporter ids from their place in the exporter array in a
SolveRequest - this removes the need to manually generate and handle
multiple sets of IDs.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch adds support for multiple exporters at the control API, and
propogates the resulting required changes through the client and the
solver.
A few notable changes:
- Each exporter instance now has an associated identifier
- Build records in the build history now have multiple possible
descriptors to built content
- Exporter responses are all merged together (like we currently do with
multiple cache exporters). We likely will need to revisit this design
later, since now cache exporters do not line up one-to-one with
exporters.
For backwards compatability, new clients will continue to produce
requests that contain the now deprecated exporter fields, as well as the
new ones. New servers will attempt to use deprecated fields if they
are present.
Co-authored-by: a-palchikov <deemok@gmail.com>
Co-authored-by: fahed dorgaa <fahed.dorgaa@gmail.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch adds multi-plexing to the local file transfer protocol (from
server to client). This is implementation-wise similar to the
multiplexing from the containerd content store transfer protocol, using
a GRPC header to select the appropriate target.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This completes propogating the fsutil.FS abstraction into the SolveOpt,
deprecating the old LocalDirs.
Since this is entirely a golang-level abstraction, we could potentially
investigate just removing the old LocalDirs directly.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch modifies the function signature of the FSSync provider to
take an fsutil.FS instead of a simple raw path resolved to the client's
root filesystem.
Internally, we were already creating an fsutil.FS to Send to the
buildkit server, however, this abstraction didn't reach the session
attachable parameters, so we couldn't provide our own custom FS
implementation.
The rationale behind this change is to allow providing more abstract
custom filesystem implementations to a BuildKit client. This way, we can
start to build from filesystems that might not be on disk - for example,
we could use our Static filesystem implementation in tests to prevent
creating lots of temporary directories, or we could use our Merge
filesystem implementation to allow easily creating variants of a single
context.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This allows some of the client processing code to abstract more over the
details of how the ociindex package works, and how it loads tags - the
calling code only need to know that it's being pointed at a containerd
content store directory.
Importantly, this allows for reusability, so we can use this same code
more easily in buildx for oci-layouts.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Alternative to PR 2816 ("dockerfile: support Dockerfile.pin for pinning sources")
This version is implemented on the llbsolver side and agnostic to the LLB frontends.
See `solver/llbsolver/vertex.go:loadLLB()`.
See `docs/build-repro.md` for the usage.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This mirrors the structure of the names for the local cache directory,
as well as the names for the oci exporter (when using a content store).
This ensures that we cannot encounter name collisions (intentionally or
unintentionally).
Signed-off-by: Justin Chadwell <me@jedevc.com>
This feature adds support for specifying unpack=true to the oci exporter
options to unpack the resulting result for the client.
To do this, we setup a content store on the client, and forward it
through to the server, which can then copy the exported data into the
content store.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Changes the filesync attachable to accept an interface instead of a
static allowlist of dirs. This way a single session can support syncing
directories not known ahead of time.
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
Allow overriding the tag field for the local cache, which enables
"scoping" cache exports, to allow multiple outputs to a single location.
This was previously enabled only for importing cache, but exporting
cache was never implemented.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This fixes what appears to be a bug from an old refactor to allow
shared sessions:
ef58b61d83
Without this, the session cannot really be shared effectively.
Signed-off-by: coryb <cbennett@netflix.com>
This also fixes the bug where registry cache options with a missing ref
would be silently discarded by the server-side component that turns the
legacy settings into the new settings.
Signed-off-by: Jonny Stoten <jonny.stoten@docker.com>
