When GracefulStop is called gRPC waits for current requests to finish
before closing. While this is generally the behavior we want, it is
not always same for the History.Listen endpoint. That endpoint is
usually open even if buildkit is not actively processing any builds,
because client may be waiting for new events.
The new logic is that if GracefulStop will happen, history will
close active listeners if there are no active builds. If there are
active builds then active listeners will be closed after all the
active builds have completed their finalizers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Current implementation based on leases.SynchronousDelete only works
with the containerd worker and is ignored otherwise. This means that
blobs referenced by history records were left on disc until the
periodic background GC was initialized later.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Client can send a finalize update to build record that
will complete saving the traces and block until the record
has been updated. If no request is sent then the traces will be
sent after a 3 second timeout as before.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Update to containerd 1.7.18, which now migrated to the errdefs module. The
existing errdefs package is now an alias for the module, and should no longer
be used directly.
This patch:
- updates the containerd dependency: https://github.com/containerd/containerd/compare/v1.7.17...v1.7.18
- replaces uses of the old package in favor of the new module
- adds a linter check to prevent accidental re-introduction of the old package
- adds a linter check to enforce using an alias, to prevent accidental use
of the errdefs package in BuildKit or Moby.
- adds a linter check to prevent using the "log" package, which was also
migrated to a separate module.
There are still some uses of the old package in (indirect) dependencies,
which should go away over time.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
While debugging around, I found some instances of os.ErrNotExist
appearing in the logs from the history API - however, no stack traces
were available, since they're not wrapped.
This patch just adds wrapping for them, with some additional debbuging
info.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch adds support for multiple exporters at the control API, and
propogates the resulting required changes through the client and the
solver.
A few notable changes:
- Each exporter instance now has an associated identifier
- Build records in the build history now have multiple possible
descriptors to built content
- Exporter responses are all merged together (like we currently do with
multiple cache exporters). We likely will need to revisit this design
later, since now cache exporters do not line up one-to-one with
exporters.
For backwards compatability, new clients will continue to produce
requests that contain the now deprecated exporter fields, as well as the
new ones. New servers will attempt to use deprecated fields if they
are present.
Co-authored-by: a-palchikov <deemok@gmail.com>
Co-authored-by: fahed dorgaa <fahed.dorgaa@gmail.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
When a record is marked for deletion the event is set to nil
but we don't check if it's nil when sending events which could
cause a panic.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
The events for currently active builds were sent through
pubsub channel instead of directly to the current request,
like it was done for completed builds for example.
This meant that if there are more active builds running
than the pubsub channel buffer (32) the sends will block.
Because the history API mutex is held in this process it
will eventually block the requests for builds that try to
update their history records.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
When moving blobs to history namespace also move the blobs
the labels point to, but don't move actual image layers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The temporary lease must be passed via the context for the call to
w.Commit() to prevent the cleanup of metadata resources before they are
fully referenced.
This fixes https://github.com/moby/buildkit/issues/3921
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
Migrate history objects to separate namespace to holding
reference to a blob does not interfer with the GC labels
held for same blobs by the containerd image store.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Previously this caused event for active builds to be
resent to all client who are monitoring when new
client makes a request.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
