Commit Graph

243 Commits

Author SHA1 Message Date
Erik Sipsma
3bfb9cfc4d Fully initialize progress controller in FromRemote
Before this, the worker's FromRemote method only partially intialized
progress controllers, leaving out the vertex digest. This meant that
only status updates would be sent and vertex start/stops would not be
sent.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-03-04 20:09:26 -08:00
Tõnis Tiigi
0a7d8907ae Merge pull request #2685 from ktock/remotecachecompression
Enable selecting compression on registry/localdir cache export
2022-03-04 08:29:02 -08:00
Kohei Tokunaga
e3f6d7b99e Enable selecting compression on remote cache export
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2022-03-04 16:28:57 +09:00
Edgar Lee
d21254e7f7 Add events for exec op
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2022-03-02 12:05:35 -08:00
Erik Sipsma
a96f9a9f1f progress: use unique id for vertex start/stop
This commit fixes more issues where vertex progress updates were using
the same ID and incorrectly deduplicating with each other. A previous
commit addressed this in solver/jobs.go but the same problem existed
elsewhere.

The update results in a random ID being used for vertex start/stop
events as it is the simplest approach that works everywhere in a
consistent pattern.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-02-24 12:30:41 -08:00
CrazyMax
c393d5c66d Merge pull request #2658 from tonistiigi/inline-cache-progress
show progress when exporting inline cache
2022-02-24 11:33:13 +01:00
Tõnis Tiigi
b0e56cd563 Merge pull request #2654 from crazy-max/buildinfo-contexts
buildinfo: named input contexts support
2022-02-23 12:13:14 -08:00
CrazyMax
2661917e8d buildinfo: handle deps
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-23 10:40:33 +01:00
Kohei Tokunaga
3b7422996a cache: Clean up temporary mount pool on restart
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2022-02-23 16:33:25 +09:00
Tonis Tiigi
6e82a4a89b show progress when exporting inline cache
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-02-22 20:05:05 -08:00
CrazyMax
a3da60d114 fix nil pointer dereference in bridge result
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-18 19:54:25 +01:00
CrazyMax
fd0af43d77 buildinfo: export metadata without exporter
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-16 20:32:57 +01:00
CrazyMax
9cd97fb726 buildinfo: check nil attrs
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-14 22:55:37 +01:00
CrazyMax
cdaafb9303 buildinfo: use metadata to set frontend build sources
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-14 21:55:20 +01:00
CrazyMax
b2e5d1938d buildinfo: set frontend attrs to bridge result
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-14 21:55:20 +01:00
CrazyMax
c29411b41a buildinfo: add build attributes and frontend
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-14 21:55:19 +01:00
CrazyMax
b4e37a867f buildinfo: refactor
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-02-14 21:55:18 +01:00
Tõnis Tiigi
58bac77c86 Merge pull request #2588 from tonistiigi/amd64-variants-support
amd64 variants support
2022-02-14 11:45:56 -08:00
genglu.gl
13665367df Close readclosers returned by DecompressStream
Signed-off-by: genglu.gl <luzigeng32@163.com>
2022-02-11 14:53:14 +08:00
Brian Goff
893bfddbc9 Move non-dist layer conversion to GetRemotes
This just makes sure the logic for the layer conversion is all in one
place and settable by a common option.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2022-02-10 01:12:42 +00:00
Erik Sipsma
0566b9a345 Add support for progress groups.
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.

The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-02-08 11:27:49 -08:00
Erik Sipsma
bb09f3c032 Improve progress output for merge+diff ops.
Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-02-08 11:26:05 -08:00
Erik Sipsma
553d3eafda Use parentheses for merge+diff vertex names.
This is easier to read than quotes, which result in lots of escaping in
progress output.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-02-08 11:26:05 -08:00
Tõnis Tiigi
17e01bac9a Merge pull request #2579 from tonistiigi/secret-env
exec: allow mounting secret environment variables
2022-02-08 10:09:55 -08:00
Tonis Tiigi
cab33b1e31 exporter: support for compression-level
compression-level option can be set on export to
define the preferred speed vs compression ratio. The
value is a number dependent on the compression algorithm.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-02-01 15:21:46 -08:00
Tonis Tiigi
9301b5f2e0 llbsolver: avoid embedded emulators for higher amd64 variants
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-31 11:28:08 -08:00
Tonis Tiigi
e38f34c7d4 archutil: refactor to return structs
Working with strings is error-prone because a platform
can be in multiple string forms and less flexible.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-31 11:28:02 -08:00
Tonis Tiigi
30a0b02479 exec: allow mounting secret environment variables
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-31 11:22:57 -08:00
Tonis Tiigi
ffce8218e3 keep buffer of logs to show on failure when regular logs are clipped
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-24 21:30:46 -08:00
Tonis Tiigi
dc21885891 hack: enable more linters
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-19 12:20:30 -08:00
Tonis Tiigi
082c666bdb exec: check full platform vector when checking for emulator
Instead of doing a direct string comparison we should
use `platforms.Only` so that we can also detect the
variants that are compatible but don’t match directly.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-18 23:54:24 -08:00
Erik Sipsma
1835ef5118 solver: fix nil result handling
Before this, you could return worker ref results from ops that have nil
refs but once they were attempted to be used, various nil exceptions
would get hit. Now, those cases should be handled.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-01-06 11:05:51 -08:00
Erik Sipsma
8c1e411d01 Add DiffOp support to solver and client.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-01-06 11:05:51 -08:00
Tonis Tiigi
872518e334 update warning type definition
Detail is now an array and URL is a separate field.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-12-13 21:38:20 -08:00
Tonis Tiigi
71316c6f29 split warning message into short and detail
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-12-02 18:17:35 -08:00
Tonis Tiigi
7ee783e90c add source mapping support to warnings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-11-30 21:58:37 -08:00
Tonis Tiigi
6cad384e93 support setting warnings from frontends
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-11-24 00:19:43 -08:00
Erik Sipsma
43a74261b3 Remove symlink, not target, in FileOp.Rm.
Before this change, if the path provided to FileOp.Rm was a symlink then
the target of the symlink would be removed instead of the symlink
itself. Now, the symlink will be removed instead. However, any symlinks
present in the parent dirs of the specified path will still be resolved
before calling os.Remove; this change only results in the base of the
specified path not being followed.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-18 12:44:12 -08:00
Erik Sipsma
d73e62f878 Add initial MergeOp implementation.
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-18 11:10:48 -08:00
Erik Sipsma
9321ec2f82 Refactor cache record mount cache.
This is mostly just preparation for merge-op. The existing
Extract method is updated to be usable for unlazying any type of refs
rather than just lazy blobs. The way views are created is simplified and
centralized in one location.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-17 11:02:16 -08:00
Erik Sipsma
2fcce87cf4 Include DescHandler CacheOpts during export.
Before this, descriptor handlers were not included with calls to the
exporter, which then sometimes called LoadRef and failed to get a ref
because it was lazy. This change results in the DescHandlers of the
already loaded refs to get plugged into context so they can be re-used
by the exporter if it needs to load the ref again.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-17 11:02:16 -08:00
CrazyMax
d81e79540b update to github.com/mitchellh/hashstructure v2.0.2
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-11-09 09:56:10 +01:00
Akihiro Suda
0279989d7f Merge pull request #2433 from tonistiigi/gha-export-fix
gha: fix handling removed blobs on reexport
2021-11-01 13:23:57 +09:00
Akihiro Suda
d47b46cf2a Merge pull request #2432 from crazy-max/gofmt
go fmt: add //go:build
2021-10-29 12:32:52 +09:00
Tonis Tiigi
962c287b1b gha: fix handling removed blobs on reexport
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-10-28 20:30:48 -07:00
Tõnis Tiigi
b33357bcd2 Merge pull request #2430 from crazy-max/cgroup-parent
cgroup-parent support
2021-10-28 10:21:03 -07:00
CrazyMax
54b8ff2fc8 go fmt: add //go:build
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-10-28 13:26:43 +02:00
Tõnis Tiigi
4b86211bed Merge pull request #2405 from ktock/cachecompression
Propagate compression options to the inline cache export
2021-10-27 20:58:53 -07:00
CrazyMax
c82ef274bf cgroup-parent support
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-10-27 23:50:20 +02:00
Kohei Tokunaga
f9e0346b34 Propagate compression options to the inline cache export
Co-authored-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-10-22 15:43:32 +09:00