In these cases, we can just call digest.Encoded(), instead of needing to
play around with removing the prefix.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This updates #4270 to add an integration test and also merge some of the
logic for how the selectors are created. Now, `toSelectors` will perform
the root path detection instead of some custom logic in `getMountDeps`.
`dedupePaths` has also been updated to check if the number of paths is 1
or less so it can avoid an allocation when the function is a no-op.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
When a record is marked for deletion the event is set to nil
but we don't check if it's nil when sending events which could
cause a panic.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
When a step in the dockerfile is a dependency of multiple other steps in
the dockerfile, the provenance attestation would record the layer chain
for that step multiple times even with the same layer chain.
This is because the provenance attestation reuses the exporter mechanic
and the exporter mechanic would need to visit this same step multiple
times to produce the appropriate cache entries.
Since these duplicate layer chains aren't intentional, this modifies the
provenance attestation capture to detect these duplicates and remove
them.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This modifies how build steps are unmarshaled from JSON into the
provenance attestation. The current method doesn't correctly handle
protobuf attributes that are used with `oneof`.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Trivially created by looking for every reference to .Variant and adding
OSVersion and OSFeatures, except the ones related to the string
representation of a Platform instance.
I then went through and ensured every assignment of OSFeatures that
might leak out, i.e., not local-only or for marhsalling purposes, uses
the append-to-nil idiom to avoid sharing the slice storage and allowing
accidental mutation after-the-fact.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
This migrates uses of github.com/opencontainers/runc/libcontainer/user
to the new github.com/moby/sys/user module, which was extracted from
runc at commit [opencontainers/runc@a3a0ec4].
This is the initial release of the module, which is a straight copy, but
some changes may be made in the next release (such as fixing camel-casing
in some fields and functions (Uid -> UID).
[opencontainers/runc@a3a0ec4]: a3a0ec48c4
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The events for currently active builds were sent through
pubsub channel instead of directly to the current request,
like it was done for completed builds for example.
This meant that if there are more active builds running
than the pubsub channel buffer (32) the sends will block.
Because the history API mutex is held in this process it
will eventually block the requests for builds that try to
update their history records.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Correctly set the content based selector when multiple bind mounts refer
to the same source. Previously, a selector that referred to the root
filesystem would be ignored. This is because a blank selector refers to
the root filesystem.
When two bind mounts referred to the same dependency, one mount would
add a selector while the other would be skipped. This caused the cache
key to be only computed based on the more narrow filesystem which caused
erroneous cache hits.
Now, the creation of the selector includes the root filesystem for
consideration. It fills in `/` as the selector and then removes it later
so that we don't narrow the selection in an invalid way.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
The "reference" package was moved to a separate module, which was extracted
from b9b19409cf
Also updating docker/docker, which also switched to this new module;
vendor: github.com/docker/docker 032797ea4bcb (v25.0.0-dev)
full diff: afd4805278...032797ea4b
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The digest of the merge/diff ops' CacheMap
would be json strings like:
```
{"Type":"buildkit.merge.v0","Merge":{"inputs":[{"input":0},{"input":1}]}}
```
rather than a sha256.
Signed-off-by: Chris Goller <goller@gmail.com>
ResolveImageConfig can be called concurrently - for example, by
dockerfile2llb during conversion, we loop through each stage and resolve
the base image for that stage.
In the case that two calls to ResolveImageConfig finish at roughly the
same time, we can hit an edge case where we attempt to modify the
bridge's image records at the same time.
To fix this, we just need to use the bridge's mutex to prevent
concurrent access here.
This should fix the following stack trace found in CI:
sandbox.go:144: goroutine 1079 [running]:
sandbox.go:144: github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).ResolveImageConfig(0xc000431e00, {0x1c2b040?, 0xc0008e5b30?}, {0xc00094ba00?, 0xc0003728f0?}, {0x0, 0xc0006cb580, {0x19ba868, 0x7}, {0xc0008f7500, ...}, ...})
sandbox.go:144: /src/solver/llbsolver/provenance.go:139 +0x1fb
sandbox.go:144: github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.func3.1()
sandbox.go:144: /src/frontend/dockerfile/dockerfile2llb/convert.go:405 +0x5fe
sandbox.go:144: golang.org/x/sync/errgroup.(*Group).Go.func1()
sandbox.go:144: /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
sandbox.go:144: created by golang.org/x/sync/errgroup.(*Group).Go
sandbox.go:144: /src/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
--- FAIL: TestIntegration/TestNoCache/worker=oci-rootless/frontend=builtin (4.45s)
No other explanation for this failure makes sense - `b` cannot be `nil`
at this point, since a call to `b.llbBridge.ResolveImageConfig` has just
succeeded (also because that would be very strange).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Sources are a pretty neat extension point, except there are a few code
paths that hard-code against each type. This moves code around and
adjusts interfaces so that Source implementations are self-contained and
merely need to be registered with the source.Manager.
Signed-off-by: Alex Suraci <suraci.alex@gmail.com>
This adds trace logs to the solver's cache manager to assist debugging
cache misses+hits.
It also replace the LazyStackTrace struct with a function that returns a
stack trace string only if trace level is enabled, an empty string
otherwise. This makes it easier to use with logrus hooks that pass the
fields around to different goroutines without needing to try to "unlazy"
the fields first.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
The os.RemoveAll() call returns nil if the path doesn't exist. When the
rmPath function is called with allowNotFound set to false, it doesn't change the
behaviour of the function.
Change the code so if allowNotFound is set to false, we first check
whether the file exists. If it doesn't exist, return an error.
Add tests for three relevant cases.
Signed-off-by: Jakub Ciolek <jakub@ciolek.dev>
The older of rules in policy matters. Eg. in [DENY *, ALLOW ref]
mixing the order would deny all sources so map can't be used
to deduplicate the rules.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Before this change, ResolveImageConfig was unaware of source policies.
This means that:
1. Images for denied sources may be resolved
2. Image configs may get pulled for sources that are later converted to
a different image
The update makes it so the image resolver first runs a given ref through
the source policy and uses any mutated ref for the actual resolve
(instead of the original ref).
It also returns the mutated ref so it can be used correctly by the
frontend (e.g. don't want to do llb.Image(oldRef@resolvedDigest)).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This change properly handles paths on different platforms. In short, this
change checks the target platform we're building an image for and applies
normalization steps to make sure the file paths are valid. This makes buildkit
properly handle paths on both *nix systems and on Windows.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
When moving blobs to history namespace also move the blobs
the labels point to, but don't move actual image layers.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
If resultProxy has been created but the result has not been
evaluated then it shouldn't show missing provenance error
for that result.
This patch works together with previous one that fixes
error reporting on provenance creation.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The temporary lease must be passed via the context for the call to
w.Commit() to prevent the cleanup of metadata resources before they are
fully referenced.
This fixes https://github.com/moby/buildkit/issues/3921
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
The data structures used by llbsolver shouldn't be bundled into the same
package as the actual creation of the build config.
As it was, it wasn't possible to use the data structures (in tests,
external libraries using buildkit, etc) without pulling in more of
buildkit than was probably intended.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This also allows us to specify that local images should be generated
with exactly the same specification as remote images, but with the
pkg:oci scheme instead of pkg:docker.
Signed-off-by: Justin Chadwell <me@jedevc.com>
