vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Secret, SSH and Tmpfs mounts never have an input vertex
and zero value for input should not be considered and input
at index 0 .
Currently, if for example secret mount had a input=0, it
caused content based checksum from first input(rootfs),
that could take quite a lot of time when image was big
and had lots of files. The computation result was cached
but if there was a cache invalidation in previous commands
it caused checksum to recomputed again for the command with
the secret mount.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Switch to use github.com/containerd/platforms module, because containerd's
platforms package has moved to a separate module. This allows updating the
platforms parsing independent of the containerd module itself.
The package in containerd is deprecated, but kept as an alias to provide
compatibility between codebases.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These test all of the new behavior:
- Checks for old default no content cache
- Checks for old read-only and no-output allowed content cache
- Checks for new root selector allowed content cache
- Checks for new caller options that allow enabling/disabling it
Signed-off-by: Justin Chadwell <me@jedevc.com>
This allows LLB-directed content-based cache enablement for each mount.
Some mounts may not be explicitly unabled (because it would be unsafe) -
for these cases we explicitly error out.
Signed-off-by: Justin Chadwell <me@jedevc.com>
These mounts are actually safe, as suggested by Erik on slack:
> Is it correct that this wouldn’t be a problem in the case where the
> selector of the mount is just “/“? Because then there’s no “hidden”
> files.
>
> If so, maybe there’s a path to enabling content cache for rw mounts
> that are from “/“. Then you could also get the same end behavior by
> not using selectors and instead always copying the subdir you want to
> mount to scratch.
This patch adds a check for this case, and explicitly enables
content-based cache for these cases.
Co-authored-by: Erik Sipsma <erik@sipsma.dev>
Signed-off-by: Justin Chadwell <me@jedevc.com>
This refactors the content-based cache to be that little bit tidier. In
addition to adding comments that explain *why* we're even bothering,
this restructures the code to avoid being unclear.
To explain the changes in a little more detail (since it's not
abundantly clear why this translation is valid), the initial condition
looks like:
if (!m.Readonly || m.Dest == pb.RootMount) && m.Output != -1 {
deps[m.Input].NoContentBasedHash = true
We can apply De Morgan's law recursively to invert the condition and the
result:
deps[m.Input].NoContentBasedHash = true
if (m.Readonly && m.Dest != pb.RootMount) || m.Output == -1 {
deps[m.Input].NoContentBasedHash = false
With all the juggling of NoContentBasedCache, we invert the variable
name to be ContentBasedCache (and invert everywhere it's used as well):
if (m.Readonly && m.Dest != pb.RootMount) || m.Output == -1 {
deps[m.Input].ContentBasedHash = true
This reads a bit easier, but now we split this into two separate
branches for readability (and so we can comment each one in more detail
separately):
if m.Readonly && m.Dest != pb.RootMount {
deps[m.Input].ContentBasedHash = true
}
if m.Output == -1 {
deps[m.Input].ContentBasedHash = true
}
While this has been the behavior for ages, I think it makes sense to
deliberately this behavior slightly. It doesn't make sense to me that we
should only disallow read-only root mounts, but no-output root mounts
are allowed - there's no reason these shouldn't behave identically, by
splitting these out:
if m.Readonly {
deps[m.Input].ContentBasedHash = true
}
if m.Output == -1 {
deps[m.Input].ContentBasedHash = true
}
if m.Dest == pb.RootMount {
deps[m.Input].ContentBasedHash = false
}
There is a small chance that this is a breaking change for some users,
however, 1. SkipOutput (-1) is very rare and not often used in the wild
(except for dockerfiles, where it's only used for non-root mounts), and 2.
will only cause a cache miss.
Signed-off-by: Justin Chadwell <me@jedevc.com>
A cache ID should not have any impact on whether or not a step should be
re-run any more than the content of that cache does (or rather,
doesn't).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
In these cases, we can just call digest.Encoded(), instead of needing to
play around with removing the prefix.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This updates #4270 to add an integration test and also merge some of the
logic for how the selectors are created. Now, `toSelectors` will perform
the root path detection instead of some custom logic in `getMountDeps`.
`dedupePaths` has also been updated to check if the number of paths is 1
or less so it can avoid an allocation when the function is a no-op.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Trivially created by looking for every reference to .Variant and adding
OSVersion and OSFeatures, except the ones related to the string
representation of a Platform instance.
I then went through and ensured every assignment of OSFeatures that
might leak out, i.e., not local-only or for marhsalling purposes, uses
the append-to-nil idiom to avoid sharing the slice storage and allowing
accidental mutation after-the-fact.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
This migrates uses of github.com/opencontainers/runc/libcontainer/user
to the new github.com/moby/sys/user module, which was extracted from
runc at commit [opencontainers/runc@a3a0ec4].
This is the initial release of the module, which is a straight copy, but
some changes may be made in the next release (such as fixing camel-casing
in some fields and functions (Uid -> UID).
[opencontainers/runc@a3a0ec4]: a3a0ec48c4
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Correctly set the content based selector when multiple bind mounts refer
to the same source. Previously, a selector that referred to the root
filesystem would be ignored. This is because a blank selector refers to
the root filesystem.
When two bind mounts referred to the same dependency, one mount would
add a selector while the other would be skipped. This caused the cache
key to be only computed based on the more narrow filesystem which caused
erroneous cache hits.
Now, the creation of the selector includes the root filesystem for
consideration. It fills in `/` as the selector and then removes it later
so that we don't narrow the selection in an invalid way.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
The digest of the merge/diff ops' CacheMap
would be json strings like:
```
{"Type":"buildkit.merge.v0","Merge":{"inputs":[{"input":0},{"input":1}]}}
```
rather than a sha256.
Signed-off-by: Chris Goller <goller@gmail.com>
Sources are a pretty neat extension point, except there are a few code
paths that hard-code against each type. This moves code around and
adjusts interfaces so that Source implementations are self-contained and
merely need to be registered with the source.Manager.
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
This change properly handles paths on different platforms. In short, this
change checks the target platform we're building an image for and applies
normalization steps to make sure the file paths are valid. This makes buildkit
properly handle paths on both *nix systems and on Windows.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
FileOpSolver keeps internal state about the vertex that
has been loaded into it and should not be reused.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This allows a frontend to request a specific for stubs removal.
By default, if not specified, this will revert to the previous
behaviour. New gateway clients however will set the property to the
desired recursive removal mode.
This property needs to be set for both components that call the
executor: for ExecOp, as well as for the StartContainer API.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Change how provenance information is captured from builds.
While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.
All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.
Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This patch connects the raw protobuf attestations through to the gateway
clients and servers, ensuring that the results are properly passed
through correctly, and that the appropriate refs are correctly cleaned
up.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This centralizes the location where ProgressKey gets set, which works
because it only needs information about the vertex, nothing op-specific.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, the worker's FromRemote method only partially intialized
progress controllers, leaving out the vertex digest. This meant that
only status updates would be sent and vertex start/stops would not be
sent.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.
The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
