Commit Graph

197 Commits

Author SHA1 Message Date
Justin Chadwell
50c2083304 epoch: refactor frontend attribute parsing
Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-12-08 14:14:51 +00:00
Tonis Tiigi
dbee61670d add build history APIs
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-11-22 23:59:33 -08:00
Justin Chadwell
e8b4f7fc76 test: fix flaky duplicate cache test
Return args order is not guaranteed (since we use a map internally), so
we need to make sure the test does not rely on order of returns.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-11-22 17:56:46 +00:00
Tõnis Tiigi
ae9d0f57c7 Merge pull request #3271 from jedevc/duplicate-exporter
Don't allow duplicate cache exporters
2022-11-21 15:49:33 -08:00
Justin Chadwell
621ab1c0af cache: error on duplicate sets of cache options
Previously, we attempted to gracefully resolve duplicate cache export
options, however, we should explicitly error - a client has made an
error if the same options appear twice. If we really want
de-duplication, then clients such as buildx should implement it there.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-11-16 12:01:35 +00:00
Tonis Tiigi
9acc6d30eb refactor buildinfo into provenance capture
Change how provenance information is captured from builds.

While previously frontend passed the buildinfo
sources with metadata, now all information is captured
through buildkit. A frontend does not need to implement
buildinfo and can't set incorrect/incomplete buildinfo
for a build result.

All LLB operations can now collect as much provenance
info as they like that will be used when making the
attestation. Previously this was limited to a single Pin
value. For example now we also detect secrets and SSH IDs
that the build uses, or if it accesses network, if local
sources are used etc.. The new design makes sure this
can be easily extended in the future.

Provenance capture can now detect builds that do
multiple separate subsolves in sequence. For example,
first subsolve gathers the sources for the build and
second one builds from immutable sources without a
network connection. If first solve does not participate
in final build result it does not end up in provenance.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-11-15 19:37:03 -08:00
Tonis Tiigi
5d5a6b93e0 Add slsa provenance attestation unless disabled
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-11-15 19:37:02 -08:00
a-palchikov
cf45d2842d Add support for multiple cache exports.
Fix cache import/export tests w.r.t inline caching.

Signed-off-by: a-palchikov <deemok@gmail.com>
2022-11-03 14:10:52 +01:00
Justin Chadwell
aacbedbb41 sbom: add post-processor to perform sbom fallback
This adds functionality to perform an SBOM scan if the frontend did not
produce a valid SBOM.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-10-13 18:17:53 +01:00
Tonis Tiigi
3cbee1c618 epoch: reuse epoch exporter key constant
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-10-12 18:09:14 -07:00
Tonis Tiigi
ec0b7b9905 exporter: add SOURCE_DATE_EPOCH support for image exporter
Allows reproducible timestamps for layer and image timestamps.

Implemented as a frontend-opt because in the future same
option could be detected by frontend for custom behavior
and same value should also apply timestamps for FileOps.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-10-12 17:29:01 -07:00
Justin Chadwell
10a334f5bd gateway: add Evaluate API
Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-10-12 07:40:10 +01:00
Aaron Lehmann
f6b002e29e Clean up CNI NS pool on main app shutdown
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2022-09-14 07:32:46 -07:00
Jonny Stoten
3b8298b67a Add context to errors setting up caches
Signed-off-by: Jonny Stoten <jonny.stoten@docker.com>
2022-07-26 12:55:21 +01:00
Tõnis Tiigi
86c33b66e1 Merge pull request #2725 from crazy-max/control-buildkit-version
control: add info service
2022-04-08 16:04:06 -07:00
CrazyMax
d2a757f862 control: add buildkit version to worker record
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-04-08 13:06:22 +02:00
Tõnis Tiigi
642281198b Merge pull request #2786 from thaJeztah/llbsolver_opts
solver/llbsolver: add Opt struct for llbsolver.New()
2022-04-07 08:49:01 -07:00
Sebastiaan van Stijn
aad0cde1c6 solver/llbsolver: add Opt struct for llbsolver.New()
The list of arguments for llbsolver.New() was a bit unwieldy, and
made it unclear what each of the arguments was for, so I wondered
if it would make sense to either define an Opt struct for this, or
to implement functional arguments (which could be an alternative).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-07 11:57:00 +02:00
Sebastiaan van Stijn
095fb7c766 control: remove unused error return from Controller.Register()
I noticed we were not handling errors in the integration in Moby, and looking
at the code in BuildKit, it appears this never returned an error, and no error
handling was present in BuildKit itself.

This function was added in 45bcca5c80, and never
returned an error at the time, so possibly this was anticipating "for future use",
but 5 years have passed, so perhaps that will never happen :)

Coincidentally, this also makes implement the `session.Attachable` interface in:
1c037fd52f/session/session.go (L32-L35)

    // Attachable defines a feature that can be exposed on a session
    type Attachable interface {
        Register(*grpc.Server)
    }

(Not sure if that's a "good" thing or a "bad" thing)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-04-07 11:20:14 +02:00
CrazyMax
859ad497fd control: add info service
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2022-03-22 13:43:31 +01:00
Erik Sipsma
0566b9a345 Add support for progress groups.
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.

The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-02-08 11:27:49 -08:00
Tonis Tiigi
872518e334 update warning type definition
Detail is now an array and URL is a separate field.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-12-13 21:38:20 -08:00
Tonis Tiigi
71316c6f29 split warning message into short and detail
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-12-02 18:17:35 -08:00
Tonis Tiigi
7ee783e90c add source mapping support to warnings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-11-30 21:58:37 -08:00
Tonis Tiigi
6cad384e93 support setting warnings from frontends
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-11-24 00:19:43 -08:00
Tõnis Tiigi
f9cb2b7b4a Merge pull request #2456 from sunchunming/fix_grpc_message_overflow
limit vertex size with 50000 to avoid huge grpc message making client…
2021-11-19 00:01:33 -08:00
sunchunming
bb24010f6e limit log vertex size with 1M to avoid huge grpc message making client buffer overflow
Signed-off-by: sunchunming <sunchunming1@jd.com>
2021-11-19 14:49:31 +08:00
Erik Sipsma
d73e62f878 Add initial MergeOp implementation.
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-18 11:10:48 -08:00
Tonis Tiigi
161b581e71 solver: increase timeout for job registration
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-09-18 21:28:45 -07:00
Tonis Tiigi
5cfe388b88 control: fix 64bit alignment for buildcount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-08-20 10:39:19 -07:00
Koichi Shiraishi
14a1d6e543 all: remove duplicate imports
Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
2021-08-11 08:28:30 +09:00
Morlay
18b49fd7dc refactor to use util/bklog instead of using logurs directly
Signed-off-by: Morlay <morlay.null@gmail.com>
2021-07-13 11:42:31 +08:00
Tonis Tiigi
1d0597e85a handle unconfigured spans without errors
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-22 08:27:35 -07:00
Tonis Tiigi
d8fc4e15f9 enable collecting traces via control api
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-20 21:57:35 -07:00
Tonis Tiigi
e09d89bc95 control: split status messages if they grow big
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-10-26 14:51:47 -07:00
Cory Bennett
355e937e15 update gateway to add ability to run and exec into containers
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2020-08-31 21:13:59 +00:00
Tonis Tiigi
c8190b1518 lint: enable golint
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:24 -07:00
Tonis Tiigi
2e9987ad16 session: track sessions with a group construct
Avoid hidden session passing and allow one session to drop when
multiple builds share a vertex.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-05 00:15:25 -07:00
Tonis Tiigi
d1458a6587 update supported platforms without restart
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-26 08:15:42 -08:00
Edgar Lee
5ec7bc7713 Change FrontendInputs in backend API definitions to *pb.Definition
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-02-24 14:38:01 -08:00
Edgar Lee
7e0f923181 Implement CapFrontendInput to pass llb.States to frontends
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-02-24 14:38:01 -08:00
Tonis Tiigi
8ad218804e correct int64 alignment
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-05 17:20:23 -07:00
Tonis Tiigi
42cd080667 imageutil: keep config and manifest lease for time based cache
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 19:18:56 -07:00
msg
78a8e643f6 Add missing locks around broadcasts
Signed-off-by: Mark Gordon <msg555@gmail.com>
2019-05-10 18:27:37 -04:00
Kunal Kushwaha
a2bbb5ff39 security entitlement support
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2019-03-27 13:57:03 +09:00
Akihiro Suda
26010a4864 decouple SessionManager from Worker
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-02-23 22:55:17 +09:00
Akihiro Suda
6e7617e889 support local cache exporter and importer
Export:

  $ buildctl build ... --export-cache type=local,store=/path/to/output-dir

Import:

  $ buildctl build ... --import-cache type=local,store=/path/to/input-dir

Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`

Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
Ian Campbell
b4454a169f implement gateway.Reference.StatFile method
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:45 +01:00
Ian Campbell
b30ee91a53 implement gateway.Reference.ReadDir method
The alternative is a helper container running e.g. find, capturing the output
to a scratch to be read back (with ref.ReadFile) and parsed.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:44 +01:00
Akihiro Suda
bf571a519e update Go to 1.11
For consistency with Moby (moby/moby#37358)

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-09-15 14:25:11 +09:00