When recomputing digests happens, the metadata is lost. This previously
happened only with sources that were affected by source policies so it
wasn't noticed, but now any LLB Op that is rewritten is affected since
the gogo protobuf switch will cause all digests to be rewritten if the
frontend and buildkit are using different versions.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
In this case the current stack trace points to the line
where the context was created. Instead the stack should be
captured when the defer is running so the return path to
the defer call is also part of the stack.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
With the switch from gogo, the `oneof` fields no longer have their
`json:"name"` fields emitted. Protobuf doesn't formally support these
fields and tells users to use the `protojson` package, but we didn't and
the official format is incompatible with the current format we use.
This adds some custom marshaling code to the already existant custom
unmarshaling code to ensure these fields are marshaled with the correct
casing.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
When GracefulStop is called gRPC waits for current requests to finish
before closing. While this is generally the behavior we want, it is
not always same for the History.Listen endpoint. That endpoint is
usually open even if buildkit is not actively processing any builds,
because client may be waiting for new events.
The new logic is that if GracefulStop will happen, history will
close active listeners if there are no active builds. If there are
active builds then active listeners will be closed after all the
active builds have completed their finalizers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This ensures different valid protobuf serializations that are sent by
frontends will be rewritten into digests that are normalized for the
buildkit solver.
The most recent example of this is that older frontends would generate
protobuf with gogo and the newer buildkit is using the google protobuf
library. These produce different serializations and cause the solver to
think that identical operations are actually different.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Current implementation based on leases.SynchronousDelete only works
with the containerd worker and is ignored otherwise. This means that
blobs referenced by history records were left on disc until the
periodic background GC was initialized later.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Many of the functions in the solver that operated on the edges were very
large which obscured their usage and the overall flow of the scheduler.
This change refactors those functions into separate smaller functions to
make it easier to follow the overall flow of the scheduler.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
The relative paths option for protoc generators doesn't work well when
it comes to dependencies. This simplifies the code generation to avoid
using `go generate` and to use one global command for protoc generation.
This is similar to https://github.com/docker/buildx/pull/2713 since the
same problems with code generation occur here too.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This fixes a problem with the new protobuf marshaling with the standard
library. LLB digests are now forced into deterministic marshaling to
ensure they produce the same digest when marshaled multiple times.
In addition, the marshal cache has also been fixed to work in
multi-threaded frontends with multiple different constraints.
Previously, if an LLB vertex was used in multiple goroutines and
marshaled concurrently, the cache would be broken. This could cause
certain problems when a specific node was used multiple times in the
same LLB tree.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
We shouldn't use the cachemount root, we should actually properly
use the worker's specified root which is propagated from the config.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Move scheduler debug statements to their own functions and in their own
file so that scheduler debug statements don't make the scheduler logic
more complex to follow. Some of the scheduler logs can be quite long and
can make it difficult to follow the code logic. This changes these log
statements to `debugSchedulerXXX` where `XXX` is the message that would
be printed.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
If error is returned from `Save()`, `rck` is nil and
causes nil dereference panic on accessing `TraceFields` when
printing debug logs.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This updates the pipe library to use generics for the request payload
and the status value. This allows the solver to put in explicit types
rather than rely on type casting from interfaces which helps with type
safety and understandability.
The status value used by the solver uses the `any` type instead of an
explicit type because the `unpark` method takes a generic list of pipes
and the different pipes have different result types. We can likely
change this in the future or create a discriminated union for the
types that can be used in this package. That is left for future work
because at least the request payload is typed now.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
On a build with no-cache, cache mounts were not pruned correctly
if the mount was on top of another ref. This also appeared in
Dockerfile when mode/uid/gid was set because implicit parent
ref is created in these cases in order to change the permissions
of a subdir that is used as a cache mount base.
Because it is not possible to know ahead of time what ref
will become the parent of cache mount during build, all cache
mounts matching the ID that have a parent will be pruned.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Secret, SSH and Tmpfs mounts never have an input vertex
and zero value for input should not be considered and input
at index 0 .
Currently, if for example secret mount had a input=0, it
caused content based checksum from first input(rootfs),
that could take quite a lot of time when image was big
and had lots of files. The computation result was cached
but if there was a cache invalidation in previous commands
it caused checksum to recomputed again for the command with
the secret mount.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Commit 4b85f11164 migrated the use of the
userns package to the github.com/moby/sys/user module.
After further discussion with maintainers, it was decided to move the
userns package to a separate module, as it has no direct relation with
"user" operations (other than having "user" in its name).
This patch migrates our code to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The userns package in libcontainer was integrated into the moby/sys/user
module at commit [3778ae603c706494fd1e2c2faf83b406e38d687d][1].
The userns package is used in many places, and currently either depends
on runc/libcontainer, or on containerd, both of which have a complex
dependency tree. This patch is part of a series of patches to unify the
implementations, and to migrate toward that implementation to simplify
the dependency tree.
[1]: 3778ae603c
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
