Avoid keeping the cache manager lock for too long. Note
that prune is already batched based on deletion criteria
and when items become releaseable. This adds extra limit
to make sure it never gets very large.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
We found that when exporting an image using the nydus compression type,
the export fails due to the 20s timeout in `func (s *Solver) recordBuildHistory`
of `solver/llbsolver/solver.go`:
```golang
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
```
This issue occurs from the execution of the `makeProvenance` func, where
the execution of the `NewProvenanceCreator` -> `LoadRemotes` -> `GetRemotes`
process in `solver/llbsolver/solver.go`.
Because nydus compression type can't be mixed with other compression types
by the logic definition in `needsForceCompression` of `cache/compression_nydus.go`,
layers are compressed to the default gzip compression type by the logic in `worker/cacheresult.go`:
```golang
if compressionopt == nil {
comp := compression.New(compression.Default)
compressionopt = &comp
all = false
}
```
Which can lead to the timeout behavior above.
The fixup removes force compression handle for nydus, it need be set by the
export option `--output type=image,compression=nydus,force-compression=true`.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
These happen when a container image is built using only metadata
actions, and hence does not generate any filesystem changes.
Such a degenerate cache is not interesting, and some OCI registries
reject OCI images with no layers.
Specifically, this only affects the `registry` and `local` exporters.
The `inline` exporter already early-outs in this case with the same
warning, and the `gha`, `s3`, and `azblob` exporters are unchanged.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Follow-up to 38040ae03e.
Messages returned by fsutil.Stat are guaranteed to return a PathError
wrapped by a stack trace (since it calls os.Lstat). However, as these
error messages are printed, they include the temporary directory for the
mounted reference which is not useful to the caller.
On an error, we can restore the filename in the PathError to the
requested filename, as also seen in os.DirFS.
Signed-off-by: Justin Chadwell <me@jedevc.com>
The "reference" package was moved to a separate module, which was extracted
from b9b19409cf
Also updating docker/docker, which also switched to this new module;
vendor: github.com/docker/docker 032797ea4bcb (v25.0.0-dev)
full diff: afd4805278...032797ea4b
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I saw a panic while using the stargz snapshotter caused
by the labels being nil.
This adds a check to prevent the panic.
Signed-off-by: Chris Goller <goller@gmail.com>
This adds trace logs to the solver's cache manager to assist debugging
cache misses+hits.
It also replace the LazyStackTrace struct with a function that returns a
stack trace string only if trace level is enabled, an empty string
otherwise. This makes it easier to use with logrus hooks that pass the
fields around to different goroutines without needing to try to "unlazy"
the fields first.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Buildkit code is mostly generic enough to support FreeBSD, however
there are some quirks / infrastructural pieces that need to be
addressed for full support, to name some
- contenthash.NewFromStat attempts to set Devmajor / Devminor for
regular files, assuming that RDev is zero for regular
files. Unlike on Linux, it's not the case for FreeBSD.
- containerdexecutor.Run uses bind mounts for rootfs. Bind mounts
are not supported in FreeBSD and we should use nullfs instead
- There is no CI job to run tests on FreeBSD
- Some dependencies weren't ported
This change ports buildkit to FreeBSD
Signed-off-by: Artem Khramov <akhramov@pm.me>
Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com>
Dedupe the several "containerd.io/uncompressed" constants and literals
into `github.com/containerd/containerd/labels.LabelUncompressed`
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Multiple calls to GetCacheContext may occur from multiple threads;
however they should only occur when dealing with an ImmutableRef.
For this case we still need to perform a lock to prevent data race
warnings; however in reality the linkMap will be unused.
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
All messages returned by os.Open are guaranteed to return a PathError.
However, as these error messages are printed, they include the temporary
directory for the mounted reference which is not useful to the caller.
On an error, we can restore the filename in the PathError to the
requested filename, as also seen in os.DirFS.
Signed-off-by: Justin Chadwell <me@jedevc.com>
The trace logs now also include the pointer value of the ref, which
serves as its "ID" amongst all the refs for the underlying record.
They also now include equal{Mutable,Immutable} IDs, which are otherwise
hard to determine with the Usage API.
Finally, LazyStackTrace now has a MarshalText method which returns the
same value as the existing String method. This allows it to work with
custom logrus hooks that rely on marshaling the fields.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Nydus-snapshotter/converter does not record image's blobs digest on
nydus image anymore since it is easy to overflow annotations' limiitations
of Containerd. Nydus-snapshotter now relies on Containerd to GC.
Signed-off-by: Changwei Ge <gechangwei@bytedance.com>
Factorize code that check for an overlay mount type by using a function
instead.
This will allow supporting other overlay-based mount types more easily
in the future (for example fuse-overlayfs).
Signed-off-by: Alexis Murzeau <amubtdx@gmail.com>
When trying to use S3 cache with anonymous credentials (for example, for importing publicly available layers), the cache is not used.
Solution: enable anonymous credentials
According to the API documentation:
"If using the `NewFromConfig` constructor you'll need to explicitly set
the `Credentials` member to nil, if the external config resolved a
credential provider."
Signed-off-by: Yurii Rashkovskii <yrashk@gmail.com>
Port https://github.com/moby/moby/blob/v23.0.1/daemon/oci_linux.go#L430-L460
> // Get the set of mount flags that are set on the mount that contains the given
> // path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
> // bind-mounting "with options" will not fail with user namespaces, due to
> // kernel restrictions that require user namespace mounts to preserve
> // CL_UNPRIVILEGED locked flags.
Fix issue 3098
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
```
newDesc.Annotations = nil
for _, k := range addAnnotations {
newDesc.Annotations[k] = desc.Annotations[k]
}
```
The codes may cause buildkitd panic: assignment to entry in nil map
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
