When trying to use S3 cache with anonymous credentials (for example, for importing publicly available layers), the cache is not used.
Solution: enable anonymous credentials
According to the API documentation:
"If using the `NewFromConfig` constructor you'll need to explicitly set
the `Credentials` member to nil, if the external config resolved a
credential provider."
Signed-off-by: Yurii Rashkovskii <yrashk@gmail.com>
Port https://github.com/moby/moby/blob/v23.0.1/daemon/oci_linux.go#L430-L460
> // Get the set of mount flags that are set on the mount that contains the given
> // path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
> // bind-mounting "with options" will not fail with user namespaces, due to
> // kernel restrictions that require user namespace mounts to preserve
> // CL_UNPRIVILEGED locked flags.
Fix issue 3098
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
```
newDesc.Annotations = nil
for _, k := range addAnnotations {
newDesc.Annotations[k] = desc.Annotations[k]
}
```
The codes may cause buildkitd panic: assignment to entry in nil map
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
In older BuildKit versions snapshotID was not always set if record
was not created with GetByBlob method. Old code defaulted to cache
record ID in that case but that broke with the metadata interface
refactor.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
By default account name is parsed from request uri host (account url)
but when product style url is disabled, account name is not part of the
host. This new attribute allows to specify the account name in such case.
Also return error if account name is empty.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
This prevents unnecessary calls (and disk writes) to the metadata cache db when there are
no items in the queue.
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
Nydus image is a container accelerated image format provided by the
Dragonfly image-service project, which offers the ability to pull
image data on demand, without waiting for the entire image pull to
complete and then start the container. It has been put in production
usage and shown vast improvements over the old OCI image format in
terms of container launching speed, image space, and network bandwidth
efficiency, as well as data integrity. Nydus image can be flexibly
configured as a FUSE-based user-space filesystem or in-kernel
EROFS (from Linux kernel v5.16) with Nydus daemon in user-space,
integrating with VM-based container runtime like KataContainers
is much easier.
Nydus has provided a conversion tool Nydusify for converting OCIv1
image to Nydus image and integrated into Harbor Acceld as a conversion
driver, which assumes that the OCI image is already available in the
registry, but a better way would be to build the Nydus images directly
from the build system instead of using the conversion tool, which would
increase the speed of the image export, so we experimentally integrated
the Nydus export in Buildkit.
Unlike other compression formats (gzip, estargz, etc.) in OCI image,
nydus is divided into two types of layer, blob, and bootstrap, where
blob serves as the data part of each layer of the image, and bootstrap
serves as the metadata of the whole image, the bootstrap is equivalent
to the view of the whole image filesystem after all layers overlay. For
example, for an OCI image with 3 layers, the corresponding nydus image
is 4 layers (3 layers of blob + 1 layer of bootstrap).
The nydus-snapshotter project provides a package to do the actual layer
compression, this commit imports the package to implement the export of
nydus compression type.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Using filepath.Join() breaks convertPathToKey() on Windows.
Windows should be able to deal with both forward and backslashes.
Using backslashes in paths here, will break the way we generate
cache keys. convertPathToKey() currently replaces '/' with 0.
Changing every code path to accomodate the Windows specific path
separator is a lot more involved than allowing Windows to just
handle the forward slash paths.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
We should be able to detect nolint comments that point to linters that
are disabled (such as with the removed structcheck).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Introduce a new compression.Type interface, which needs
to be implemented for each compression type, by that we can
reduce the number of switch case statements and ensure that
we don't miss the handle of any compression types, and also
make more easily for supporting new compression types.
This is a commit for code improvement, so no logical changes.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
This adds experimental support for Azure Blob Storage based remote cache to buildkit. For usage instructions please refer to the updated Readme. We have tried to keep it similar to the S3 based implementation while using equivalent Azure Blob Storage specific semantics where appropriate.
This also adds end-to-end tests to exercise the Azure Blob Storage based cache using the [Azurite emulator](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite).
Co-authored-by: Amr Mahdi <amrh@microsoft.com>
Co-authored-by: Pranav Pandit <pranavp@microsoft.com>
Signed-off-by: Pranav Pandit <pranavp@microsoft.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
