641 Commits

Author SHA1 Message Date
Justin Chadwell
d6bdf7bc54 chore: remove unused blobIDs for nydus compression
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-07-27 09:40:29 +01:00
Justin Chadwell
f272167c79 nydus: fix missing imports
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-07-26 16:59:42 +01:00
Akihiro Suda
67f3e79553 Dedupe "containerd.io/uncompressed" constants and literals
Dedupe the several "containerd.io/uncompressed" constants and literals
into `github.com/containerd/containerd/labels.LabelUncompressed`

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-07-25 03:02:09 +09:00
Alex Couture-Beil
caa9579cb0 contenthash: data race
Multiple calls to GetCacheContext may occur from multiple threads;
however they should only occur when dealing with an ImmutableRef.
For this case we still need to perform a lock to prevent data race
warnings; however in reality the linkMap will be unused.

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2023-07-11 15:56:58 -07:00
Tonis Tiigi
d4bcef42bf improve error handling in ReadFile
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-07-06 22:02:11 -07:00
Justin Chadwell
38040ae03e gateway: restore original filename in ReadFile error message
All messages returned by os.Open are guaranteed to return a PathError.
However, as these error messages are printed, they include the temporary
directory for the mounted reference which is not useful to the caller.

On an error, we can restore the filename in the PathError to the
requested filename, as also seen in os.DirFS.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-07-05 11:00:22 +01:00
Tonis Tiigi
8ffc03b8f0 move flightcontrol to use generics
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-06-28 23:44:05 -07:00
Erik Sipsma
273aeab1d1 cache: add a few more fields to ref trace logs.
The trace logs now also include the pointer value of the ref, which
serves as its "ID" amongst all the refs for the underlying record.

They also now include equal{Mutable,Immutable} IDs, which are otherwise
hard to determine with the Usage API.

Finally, LazyStackTrace now has a MarshalText method which returns the
same value as the existing String method. This allows it to work with
custom logrus hooks that rely on marshaling the fields.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2023-05-25 15:38:07 -07:00
Erik Sipsma
dd53dca59c Add trace logs for cache leaks.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2023-05-15 16:03:29 -07:00
CrazyMax
a9e8e39bae Merge pull request #3814 from changweige/update-nydus
nydus: update nydus-snapshotter dependency to v0.8.0
2023-05-15 14:00:52 +02:00
Matt Kang
29fd071f09 feedback changes for moby/buildkit #2251
Signed-off-by: Matt Kang <impulsecss@gmail.com>
2023-05-12 16:05:34 -07:00
Kang, Matthew
797156ac89 added import/export support for OCI compatible image manifest version of cache manifest (opt-in on export, inferred on import) moby/buildkit #2251
Signed-off-by: Kang, Matthew <impulsecss@gmail.com>
2023-05-11 09:45:13 -07:00
Tõnis Tiigi
7a5ca71ce6 Merge pull request #3830 from amurzeau/contrib5
overlay: use function to check for overlay-based mounts
2023-05-10 21:56:28 -07:00
Justin Chadwell
6d4901c10b Merge pull request #3847 from thaJeztah/image_spec_1.1.0-rc3 2023-05-10 10:07:23 +01:00
Erik Sipsma
f1223ede9c cache: always release ref when getting size in usage.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2023-05-08 09:12:20 -07:00
Sebastiaan van Stijn
647a2b5892 vendor: github.com/opencontainers/image-spec v1.1.0-rc3
full diff: https://github.com/opencontainers/image-spec/compare/3a7f492d3f1b...v1.1.0-rc3

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-05-07 20:54:31 +02:00
Changwei Ge
483e87725e nydus: update nydus-snapshotter dependency to v0.8.0
Nydus-snapshotter/converter does not record image's blobs digest on
nydus image anymore since it is easy to overflow annotations' limiitations
of Containerd. Nydus-snapshotter now relies on Containerd to GC.

Signed-off-by: Changwei Ge <gechangwei@bytedance.com>
2023-05-06 10:24:37 +08:00
Alexis Murzeau
cbd1a73339 overlay: use function to check for overlay-based mounts
Factorize code that check for an overlay mount type by using a function
instead.

This will allow supporting other overlay-based mount types more easily
in the future (for example fuse-overlayfs).

Signed-off-by: Alexis Murzeau <amubtdx@gmail.com>
2023-05-02 10:41:37 +00:00
Bertrand Paquet
fb339763bc Revert "Problem: can't use anonymous S3 credentials"
Signed-off-by: Bertrand Paquet <bertrand.paquet@gmail.com>
2023-04-13 21:50:11 +02:00
CrazyMax
86c3b26413 Merge pull request #3692 from yrashk/patch-1
Problem: can't use anonymous S3 credentials
2023-03-31 10:34:49 +02:00
coryb
a8aa7b200c use bklog.G(ctx) instead of logrus directly
Signed-off-by: coryb <cbennett@netflix.com>
2023-03-15 12:07:44 -07:00
Yurii Rashkovskii
b70350975a Problem: can't use anonymous S3 credentials
When trying to use S3 cache with anonymous credentials (for example, for importing publicly available layers), the cache is not used.

Solution: enable anonymous credentials

According to the API documentation:

"If using the `NewFromConfig` constructor you'll need to explicitly set
the `Credentials` member to nil, if the external config resolved a
credential provider."

Signed-off-by: Yurii Rashkovskii <yrashk@gmail.com>
2023-03-13 04:41:27 -07:00
Tõnis Tiigi
eb7080c5c4 Merge pull request #3697 from AkihiroSuda/fix-3098
rootless: support Bottlerocket OS
2023-03-10 18:36:59 -08:00
Akihiro Suda
2fed51aea3 rootless: fix up unprivileged mount opts
Port https://github.com/moby/moby/blob/v23.0.1/daemon/oci_linux.go#L430-L460

> // Get the set of mount flags that are set on the mount that contains the given
> // path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
> // bind-mounting "with options" will not fail with user namespaces, due to
> // kernel restrictions that require user namespace mounts to preserve
> // CL_UNPRIVILEGED locked flags.

Fix issue 3098

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-03-09 21:09:58 +09:00
Akihiro Suda
98deacf59c go.mod: github.com/containerd/containerd v1.7.0-rc.3
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-03-09 15:07:22 +09:00
Yan Song
39bd8c6cf5 fix a possible panic on cache
```
newDesc.Annotations = nil
for _, k := range addAnnotations {
  newDesc.Annotations[k] = desc.Annotations[k]
}
```

The codes may cause buildkitd panic: assignment to entry in nil map

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
2023-02-24 11:39:57 +00:00
Tõnis Tiigi
0ad8d61575 Merge pull request #3109 from ktock/reuseremotelayers
Fix cache cannot reuse lazy layers
2023-02-14 16:51:41 -08:00
Tonis Tiigi
488791a5a1 gha: avoid range requests with too big offset
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-02-10 19:23:39 -08:00
CrazyMax
82949ae6cc Merge pull request #3607 from crazy-max/remotecache
remotecache: small enhancements
2023-02-10 19:15:58 +01:00
Tõnis Tiigi
a196d7b368 Merge pull request #3566 from tonistiigi/fix-blobonly
cache: don’t link blobonly based on chainid
2023-02-09 09:49:36 -08:00
CrazyMax
1db81665fd remotecache: proper exporter naming for gha, s3 and azblob
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-02-09 18:46:15 +01:00
CrazyMax
cf4e1bb7d2 remotecache: explicit names for registry and local
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-02-09 18:46:15 +01:00
CrazyMax
9081ade948 remotecache: mutualize compression parsing attrs
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-02-09 18:46:14 +01:00
Tonis Tiigi
2c0c429530 cache: add fallback for snapshotID
In older BuildKit versions snapshotID was not always set if record
was not created with GetByBlob method. Old code defaulted to cache
record ID in that case but that broke with the metadata interface
refactor.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-02-08 12:54:47 -08:00
Tonis Tiigi
39ff1c04fe inline cache: fix blob indexes by uncompressed digest
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-02-03 18:07:50 -08:00
Tonis Tiigi
99bd0d8046 cache: don’t link blobonly based on chainid
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-02-01 12:10:04 -08:00
Tõnis Tiigi
fd0c25c747 Merge pull request #3493 from ktock/unlazylocal
Make local cache importer non-lazy
2023-01-26 09:10:16 -08:00
Kohei Tokunaga
085bd8af7f Fix cache cannot reuse lazy layers
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2023-01-24 16:23:56 +09:00
Justin Chadwell
d301d3709d cache: add registry.insecure option to registry exporter
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-01-12 11:42:01 +00:00
Kohei Tokunaga
c68af33fb6 Make local cache non-lazy
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2023-01-11 23:25:08 +09:00
CrazyMax
ef0a35260e azblob cache: account_name attribute
By default account name is parsed from request uri host (account url)
but when product style url is disabled, account name is not part of the
host. This new attribute allows to specify the account name in such case.

Also return error if account name is empty.

Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-01-10 13:51:39 +01:00
lomot
f3512a696f fix: updated_at -> updated-at
Signed-off-by: lomot <lomot@qq.com>
2022-12-31 11:24:24 +08:00
Tonis Tiigi
ab0d1a6eea cache: make sure mutable refs are not finalized on exporting cache
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-12-12 22:15:41 -08:00
Tõnis Tiigi
1bc934e23b Merge pull request #3258 from jedevc/sbom-filelist
Supplement generated SBOMs with layer information post-build
2022-11-23 12:17:02 -08:00
Tõnis Tiigi
6f13ac6232 Merge pull request #3305 from jedevc/errors-linting
chore: remove fmt.Errorf across the codebase in preference of errors.Errorf/Wrap
2022-11-23 10:06:27 -08:00
Tonis Tiigi
71a62a9ffe cache: add filelist methods for file to layer mapping
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-11-23 17:16:14 +00:00
Justin Chadwell
b79f6f64bd chore: refactor azblob cache to use errors pkg
Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-11-23 12:29:27 +00:00
Justin Chadwell
feaba880de chore: refactor cache to use errors pkg
Signed-off-by: Justin Chadwell <me@jedevc.com>
2022-11-23 12:29:27 +00:00
Tonis Tiigi
f84058eb1f contenthash: fix uid/gid in contenthash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-11-22 22:25:43 -08:00
Tonis Tiigi
a20e48f36d provenance: add layers support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2022-11-15 19:37:03 -08:00