When GracefulStop is called gRPC waits for current requests to finish
before closing. While this is generally the behavior we want, it is
not always same for the History.Listen endpoint. That endpoint is
usually open even if buildkit is not actively processing any builds,
because client may be waiting for new events.
The new logic is that if GracefulStop will happen, history will
close active listeners if there are no active builds. If there are
active builds then active listeners will be closed after all the
active builds have completed their finalizers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Current implementation based on leases.SynchronousDelete only works
with the containerd worker and is ignored otherwise. This means that
blobs referenced by history records were left on disc until the
periodic background GC was initialized later.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Use golang.org/x/exp/trace to implement an trace recorder that saves the trace
to a circular buffer and can be retrieved at any time.
Debug endpoints have been added under /debug/flight to start and stop the trace
as well as to set its period.
Due to golang.org/x/exp/trace, the minimum go version has been bumped to 1.22
Signed-off-by: Alberto Garcia Hierro <damaso.hierro@docker.com>
Client can send a finalize update to build record that
will complete saving the traces and block until the record
has been updated. If no request is sent then the traces will be
sent after a 3 second timeout as before.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The solver has a Close method to shutdown the scheduler, which releases
a goroutine. We should call it on shutdown.
While in the area, we can also close the sysSampler.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We can derive exporter ids from their place in the exporter array in a
SolveRequest - this removes the need to manually generate and handle
multiple sets of IDs.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch adds support for multiple exporters at the control API, and
propogates the resulting required changes through the client and the
solver.
A few notable changes:
- Each exporter instance now has an associated identifier
- Build records in the build history now have multiple possible
descriptors to built content
- Exporter responses are all merged together (like we currently do with
multiple cache exporters). We likely will need to revisit this design
later, since now cache exporters do not line up one-to-one with
exporters.
For backwards compatability, new clients will continue to produce
requests that contain the now deprecated exporter fields, as well as the
new ones. New servers will attempt to use deprecated fields if they
are present.
Co-authored-by: a-palchikov <deemok@gmail.com>
Co-authored-by: fahed dorgaa <fahed.dorgaa@gmail.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
The "reference" package was moved to a separate module, which was extracted
from b9b19409cf
Also updating docker/docker, which also switched to this new module;
vendor: github.com/docker/docker 032797ea4bcb (v25.0.0-dev)
full diff: afd4805278...032797ea4b
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Migrate history objects to separate namespace to holding
reference to a blob does not interfer with the GC labels
held for same blobs by the containerd image store.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
When error appears on sending to network, channel
still needs to be read empty to ensure that the other
side does not get blocked on sending to the channel.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This makes it possible for callers to customize the full CacheManager
used by buildkitd without having to rewrite/fork the whole control
package.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Status and ListenHistory API return events with one-sided timestamp
for events that have not been completed yet. Because client and
server may not have exactly the same time the client can't really
be sure what these values mean without knowing the current server
time as well.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Alternative to PR 2816 ("dockerfile: support Dockerfile.pin for pinning sources")
This version is implemented on the llbsolver side and agnostic to the LLB frontends.
See `solver/llbsolver/vertex.go:loadLLB()`.
See `docs/build-repro.md` for the usage.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This reverts commit 6f21d6b403.
We don't need the exporter opts for multiplatform, they were a hack to
attempt to preserve the semantics of converting between Ref->Refs
freely (both buildkit-side and frontend-side).
Signed-off-by: Justin Chadwell <me@jedevc.com>
We no longer need to (or should) force refs conversion.
A single Ref is distinctly different from a Refs map with a single
element - they express the user-specified difference between whether
multi-platform exports should be used or not. It's important to preserve
this sematic, especially on the side of the frontend, where it can't be
recovered if lost.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This is required now that the correspondence between platforms and refs
has been broken. Without it, there's no way to detect between the
following cases from the output that they emit:
buildctl build --opt platform=linux/amd64 --opt attest:sbom
buildctl build --opt platform=linux/amd64 --opt attest:sbom --opt build-arg:BUILDKIT_MULTI_PLATFORM=true
We'd like the former to produce a flat file structure through the local
exporter, and the latter to produce an explicitly nested file structure
through the same exporter. However, both of these produce Refs instead
of a singular Ref, so we can't just look at the Result to know which
one.
Similar to how we handle the SOURCE_DATE_EPOCH build-arg, we can handle
the multi-platform args at the control API boundary, setting the
explicit option multi-platform in the exporter if it is set. In the
future, we can guide users away from the BUILDKIT_MULTI_PLATFORM args
entirely, and towards the multi-platform exporter option if we want.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Previously, we attempted to gracefully resolve duplicate cache export
options, however, we should explicitly error - a client has made an
error if the same options appear twice. If we really want
de-duplication, then clients such as buildx should implement it there.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Change how provenance information is captured from builds.
While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.
All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.
Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
