vtproto is an extra protobuf compiler that generates special methods
suffixed with `VT` that create typed and unrolled marshal and unmarshal
functions similar to gogo that can be used for performance sensitive
code. These extensions are optional for code to use but buildkit uses
them.
A codec is also included to utilize vtproto for grpc code. If the
package `github.com/moby/buildkit/util/grpcutil/encoding/proto` is
imported then vtproto will be used if it exists and otherwise it will
use the standard marshaling and unmarshaling methods.
This codec has an important difference from the default codec. The
default codec will always reset messages before unmarshaling. In most
cases, this is unnecessary and is only relevant for `RecvMsg` on
streams. In most cases, if we are passing in an existing message to this
method, we want to reuse the buffers. This codec will always merge the
message when unmarshaling instead of resetting the input message.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Remove gogoproto in favor of the standard protobuf compiler. This
removes any nonstandard extensions that were part of gogoproto such as
the custom types.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Secret, SSH and Tmpfs mounts never have an input vertex
and zero value for input should not be considered and input
at index 0 .
Currently, if for example secret mount had a input=0, it
caused content based checksum from first input(rootfs),
that could take quite a lot of time when image was big
and had lots of files. The computation result was cached
but if there was a cache invalidation in previous commands
it caused checksum to recomputed again for the command with
the secret mount.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Switch to use github.com/containerd/platforms module, because containerd's
platforms package has moved to a separate module. This allows updating the
platforms parsing independent of the containerd module itself.
The package in containerd is deprecated, but kept as an alias to provide
compatibility between codebases.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows LLB-directed content-based cache enablement for each mount.
Some mounts may not be explicitly unabled (because it would be unsafe) -
for these cases we explicitly error out.
Signed-off-by: Justin Chadwell <me@jedevc.com>
These mounts are actually safe, as suggested by Erik on slack:
> Is it correct that this wouldn’t be a problem in the case where the
> selector of the mount is just “/“? Because then there’s no “hidden”
> files.
>
> If so, maybe there’s a path to enabling content cache for rw mounts
> that are from “/“. Then you could also get the same end behavior by
> not using selectors and instead always copying the subdir you want to
> mount to scratch.
This patch adds a check for this case, and explicitly enables
content-based cache for these cases.
Co-authored-by: Erik Sipsma <erik@sipsma.dev>
Signed-off-by: Justin Chadwell <me@jedevc.com>
This refactors the content-based cache to be that little bit tidier. In
addition to adding comments that explain *why* we're even bothering,
this restructures the code to avoid being unclear.
To explain the changes in a little more detail (since it's not
abundantly clear why this translation is valid), the initial condition
looks like:
if (!m.Readonly || m.Dest == pb.RootMount) && m.Output != -1 {
deps[m.Input].NoContentBasedHash = true
We can apply De Morgan's law recursively to invert the condition and the
result:
deps[m.Input].NoContentBasedHash = true
if (m.Readonly && m.Dest != pb.RootMount) || m.Output == -1 {
deps[m.Input].NoContentBasedHash = false
With all the juggling of NoContentBasedCache, we invert the variable
name to be ContentBasedCache (and invert everywhere it's used as well):
if (m.Readonly && m.Dest != pb.RootMount) || m.Output == -1 {
deps[m.Input].ContentBasedHash = true
This reads a bit easier, but now we split this into two separate
branches for readability (and so we can comment each one in more detail
separately):
if m.Readonly && m.Dest != pb.RootMount {
deps[m.Input].ContentBasedHash = true
}
if m.Output == -1 {
deps[m.Input].ContentBasedHash = true
}
While this has been the behavior for ages, I think it makes sense to
deliberately this behavior slightly. It doesn't make sense to me that we
should only disallow read-only root mounts, but no-output root mounts
are allowed - there's no reason these shouldn't behave identically, by
splitting these out:
if m.Readonly {
deps[m.Input].ContentBasedHash = true
}
if m.Output == -1 {
deps[m.Input].ContentBasedHash = true
}
if m.Dest == pb.RootMount {
deps[m.Input].ContentBasedHash = false
}
There is a small chance that this is a breaking change for some users,
however, 1. SkipOutput (-1) is very rare and not often used in the wild
(except for dockerfiles, where it's only used for non-root mounts), and 2.
will only cause a cache miss.
Signed-off-by: Justin Chadwell <me@jedevc.com>
A cache ID should not have any impact on whether or not a step should be
re-run any more than the content of that cache does (or rather,
doesn't).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This updates #4270 to add an integration test and also merge some of the
logic for how the selectors are created. Now, `toSelectors` will perform
the root path detection instead of some custom logic in `getMountDeps`.
`dedupePaths` has also been updated to check if the number of paths is 1
or less so it can avoid an allocation when the function is a no-op.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Trivially created by looking for every reference to .Variant and adding
OSVersion and OSFeatures, except the ones related to the string
representation of a Platform instance.
I then went through and ensured every assignment of OSFeatures that
might leak out, i.e., not local-only or for marhsalling purposes, uses
the append-to-nil idiom to avoid sharing the slice storage and allowing
accidental mutation after-the-fact.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Correctly set the content based selector when multiple bind mounts refer
to the same source. Previously, a selector that referred to the root
filesystem would be ignored. This is because a blank selector refers to
the root filesystem.
When two bind mounts referred to the same dependency, one mount would
add a selector while the other would be skipped. This caused the cache
key to be only computed based on the more narrow filesystem which caused
erroneous cache hits.
Now, the creation of the selector includes the root filesystem for
consideration. It fills in `/` as the selector and then removes it later
so that we don't narrow the selection in an invalid way.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This change properly handles paths on different platforms. In short, this
change checks the target platform we're building an image for and applies
normalization steps to make sure the file paths are valid. This makes buildkit
properly handle paths on both *nix systems and on Windows.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
This allows a frontend to request a specific for stubs removal.
By default, if not specified, this will revert to the previous
behaviour. New gateway clients however will set the property to the
desired recursive removal mode.
This property needs to be set for both components that call the
executor: for ExecOp, as well as for the StartContainer API.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Change how provenance information is captured from builds.
While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.
All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.
Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This centralizes the location where ProgressKey gets set, which works
because it only needs information about the vertex, nothing op-specific.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, the worker's FromRemote method only partially intialized
progress controllers, leaving out the vertex digest. This meant that
only status updates would be sent and vertex start/stops would not be
sent.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
won't make sense soon due to other potential changes. For example,
there can now be multiple blobs associated with a ref (for different
compression types), so the fact that you could access the "Blob"
field from the Info method on Ref incorrectly implied there was just
a single blob for the ref. This is on top of the fact that there is
no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
Previously, many parts of the code outside the cache package could
obtain the bolt storage item for any ref and read/write it directly.
This made it hard to understand what fields are used and when. Now,
the Metadata method has been removed from the Ref interface and
replaced with getters+setters for metadata fields we want to expose
outside the package, which makes it much easier to track and
understand. Similar changes have been made to the metadata search
interface.
3. Use a consistent getter+setter interface for metadata, replacing
the mix of interfaces like Metadata(), Size(), Info() and other
inconsistencies.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Relates to a82fff6377/docs/packages.md (proxies)
> (..) the first four of these are the standard built-in build-arg options
> available for `docker build`
> (..) The last, `all_proxy`, is a standard var used for socks proxying. Since
> it is not built into `docker build`, if you want to use it, you will need to
> add the following line to the dockerfile:
>
> ARG all_proxy
Given the we support all other commonly known proxy env-vars by default, it makes
sense to add this one as well.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
A bug in cloneExecOp prevented mounts from being included in the digest
computed for the execOp cache map. This could lead to an exec being
wrongly cached when a different mount was used for a subsequent
execution.
Repro case:
https://gist.github.com/aaronlehmann/cfeaefc028df8127fb85b9b5f9125f2d
In this example, pass2 should generate an empty diff because the /from
and /to mounts are the same busybox image. But before the fix, it uses
the cached result from pass1 (with different mounts) instead.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
Cache mount instances are shared between multiple vertextes/builds
so if one of the cloned instance gets committed reference count
will get corrupted as other parts of the code still see reference as
mountable.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
- Plumb default worker by adding GetDefault() to frontend.WorkerInfos
- To avoid cyclic dependency, refactor frontend.WorkerInfos to worker.Infos
- Refactor gateway.NewContainer to share code with llbsolver/ops/exec.go
Signed-off-by: Edgar Lee <edgarl@netflix.com>
