Commit Graph

95 Commits

Author SHA1 Message Date
Erik Sipsma
0566b9a345 Add support for progress groups.
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.

The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2022-02-08 11:27:49 -08:00
Tonis Tiigi
dc21885891 hack: enable more linters
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-01-19 12:20:30 -08:00
Erik Sipsma
d73e62f878 Add initial MergeOp implementation.
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-18 11:10:48 -08:00
Erik Sipsma
03ed0548ef cache: Replace Parent method with LayerChain.
The Parent method will no longer make sense with forthcoming Merge and
Diff support as refs will become capable of having multiple parents. It
was also only ever used externally to get the full chain of refs for
each layer in the ref's chain.

The newly added LayerChain method replaces Parents with a method that
just returns a slice of refs for each layer in the ref's chain. This
will work more seamlessly with Merge and Diff (in which case it returns
the "flattened" ancestors of the ref) in addition to being a bit easier
to use for the exiting cases anyways.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-11-17 11:02:16 -08:00
CrazyMax
5fcc94454f Generate and embed build sources
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-09-18 01:39:35 +02:00
Erik Sipsma
a9f1980ebb Refactor cache metadata interface.
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
   won't make sense soon due to other potential changes. For example,
   there can now be multiple blobs associated with a ref (for different
   compression types), so the fact that you could access the "Blob"
   field from the Info method on Ref incorrectly implied there was just
   a single blob for the ref. This is on top of the fact that there is
   no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
   Previously, many parts of the code outside the cache package could
   obtain the bolt storage item for any ref and read/write it directly.
   This made it hard to understand what fields are used and when. Now,
   the Metadata method has been removed from the Ref interface and
   replaced with getters+setters for metadata fields we want to expose
   outside the package, which makes it much easier to track and
   understand. Similar changes have been made to the metadata search
   interface.
3. Use a consistent getter+setter interface for metadata, replacing
   the mix of interfaces like Metadata(), Size(), Info() and other
   inconsistencies.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-08-25 19:15:09 +00:00
Kohei Tokunaga
f8d30d567e Add estargz compression type
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-08-24 14:10:09 +09:00
Erik Sipsma
4237175ac7 pull: use resolvemode in flightcontrol key.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-08-16 17:26:36 +00:00
Koichi Shiraishi
671ae38fb7 all: unify the specs-go package import alias to ocispecs
ocispecs means "O"pen "C"ontainer "I"nitiative image-spec/"specs"-go/v1
                      opencontainers          /image-spec/specs-go/v1

Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
2021-08-11 08:29:09 +09:00
coryb
de0019bf35 fix dropped pull progress output due to canceled context
fixes #2248

Signed-off-by: coryb <cbennett@netflix.com>
2021-07-14 09:11:44 -07:00
Tonis Tiigi
a558ac4cce add per domain semaphore to limit concurrent connections
This is a safer alternative until we figure out why
http.Transport based limiting fails.

Some connections like cache export/import do not have a
domain key atm and these connections use global pool.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-07-13 14:08:03 -07:00
Cory Bennett
b1d441b175 [#2112] progress.FromContext returns a writer factory
this allows progress.Controller to manage the writer lifecycle

Signed-off-by: Cory Bennett <cbennett@netflix.com>
2021-06-28 18:58:29 +00:00
Cory Bennett
1da8b671c4 [#2112] progress.Controller should own the progress.Writer to prevent leaks
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2021-06-28 18:56:45 +00:00
Tonis Tiigi
e46cfab9fb fix invalid cancellation errors on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-01 00:27:31 -08:00
Tonis Tiigi
e3b05289d8 add session injection to remote loading
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-02 22:20:44 -08:00
Tonis Tiigi
6a3d6e983b source: fix puller error caching on cancellation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-10-27 13:21:49 -07:00
ktock
a9ba84fd5e Limit size of additional label for avoiding preparation failure
In containerd, there is a size limit for label size (4096 chars).
If an image has many layers (> (4096-43)/72 > 55),
`containerd.io/snapshot/remote/stargz.layers` will hit the limit of
label size and the remote snapshot preparation will fail.
This commit fixes this by limiting the size of the label.

Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-09-11 11:40:56 +09:00
ktock
c975424deb Enable to use remote snapshots for refs
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-08-24 08:44:00 +09:00
Tonis Tiigi
f1cd79bf65 resolver: add better pooling and custom authenticator
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-08-13 17:27:11 -07:00
Erik Sipsma
1b30fd146b cache: Remove ImageRef from DescHandlers
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Erik Sipsma
926ca1804c pull: de-pointer non-optional platform field
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Erik Sipsma
55cbd19dec Add support for lazily-pulled blobs in cache manager.
This allows the layers of images to only be pulled if/once they are actually
required.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Tonis Tiigi
05013a663b pull: fix session updating on resolver
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-05 00:15:33 -07:00
Tonis Tiigi
214aa5dbcf pull: allow separate sessions for different parts of pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-05 00:15:33 -07:00
Tonis Tiigi
2e9987ad16 session: track sessions with a group construct
Avoid hidden session passing and allow one session to drop when
multiple builds share a vertex.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-05 00:15:25 -07:00
Tonis Tiigi
2c3cf11fde resolver: update to new registryhosts based config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-03-05 19:56:43 -08:00
Edgar Lee
2edb431a0f Allow previous solve results to be used in new solves
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-01-23 11:02:40 -08:00
Tonis Tiigi
339d4b2fef leaseutil: mark temporary leases with timestamps
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:35:50 -07:00
Tonis Tiigi
6be136db6e cleanup unused code
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:33:01 -07:00
Tonis Tiigi
a9209caa93 clean up temporary leases on startup after crash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:32:57 -07:00
Tonis Tiigi
2d0f9d3cc0 add manifest/config to the lease
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:32:52 -07:00
Tonis Tiigi
688e2c2272 cache: update components to new lease based cache manager
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:32:04 -07:00
Tibor Vass
914d86758c Merge pull request #1024 from tonistiigi/gcr-workaround
pull: add cache for gcr bug
2019-06-17 10:36:49 -07:00
Tonis Tiigi
ab23a1ab1b pull: add cache for gcr bug
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 18:51:55 -07:00
Tonis Tiigi
d72c19071a imageutil: use leases on fetching config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 14:39:05 -07:00
Tonis Tiigi
8e25187cc7 pull: fix content blobs deletion on pull race
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 14:39:05 -07:00
Tonis Tiigi
37567fc826 pull: fix second cachekey for schema1
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-25 17:16:43 -08:00
Akihiro Suda
26010a4864 decouple SessionManager from Worker
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-02-23 22:55:17 +09:00
Tonis Tiigi
ecd4a22546 resolver: add config support for mirrors/plainhttp
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 18:44:45 -07:00
Tonis Tiigi
489246dd28 cache: support for internal/frontend record type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
Tonis Tiigi
9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi
e971b7cddf pull: handle resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:20:55 -07:00
Tonis Tiigi
6be1257f5d export: manifest list support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 16:01:12 -07:00
Akihiro Suda
56e2ea083a Merge pull request #510 from tonistiigi/win-differ-support
diff: implement windows layer support for linux
2018-07-17 11:29:54 +09:00
Tonis Tiigi
bc765861be diff: implement windows layer support for linux
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 16:33:21 -07:00
Tonis Tiigi
b3572c6648 image: fix platform passing on config resolve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 13:51:58 -07:00
Tonis Tiigi
f9383c48a3 source: image source platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Ian Campbell
ef1a79a1c1 Allow caller to specify platform for use with imagemetaresolver.
This needs to be plumbed through to `imageutil.Config` which is also used from
the `containerimage` package. Since that is used in the context of a Worker the
`platforms.Default()` (corresponding to the platform of the worker itself) is
appropriate so apply a default at that level.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-05-24 11:23:35 +01:00
Tonis Tiigi
8084e8cde9 llbsolver: allow empty returns for opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Akihiro Suda
d22e5a1721 split puller from source/containerimage into util/pull
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-05-18 06:14:06 +09:00