This prevents unnecessary calls (and disk writes) to the metadata cache db when there are
no items in the queue.
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
Nydus image is a container accelerated image format provided by the
Dragonfly image-service project, which offers the ability to pull
image data on demand, without waiting for the entire image pull to
complete and then start the container. It has been put in production
usage and shown vast improvements over the old OCI image format in
terms of container launching speed, image space, and network bandwidth
efficiency, as well as data integrity. Nydus image can be flexibly
configured as a FUSE-based user-space filesystem or in-kernel
EROFS (from Linux kernel v5.16) with Nydus daemon in user-space,
integrating with VM-based container runtime like KataContainers
is much easier.
Nydus has provided a conversion tool Nydusify for converting OCIv1
image to Nydus image and integrated into Harbor Acceld as a conversion
driver, which assumes that the OCI image is already available in the
registry, but a better way would be to build the Nydus images directly
from the build system instead of using the conversion tool, which would
increase the speed of the image export, so we experimentally integrated
the Nydus export in Buildkit.
Unlike other compression formats (gzip, estargz, etc.) in OCI image,
nydus is divided into two types of layer, blob, and bootstrap, where
blob serves as the data part of each layer of the image, and bootstrap
serves as the metadata of the whole image, the bootstrap is equivalent
to the view of the whole image filesystem after all layers overlay. For
example, for an OCI image with 3 layers, the corresponding nydus image
is 4 layers (3 layers of blob + 1 layer of bootstrap).
The nydus-snapshotter project provides a package to do the actual layer
compression, this commit imports the package to implement the export of
nydus compression type.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Using filepath.Join() breaks convertPathToKey() on Windows.
Windows should be able to deal with both forward and backslashes.
Using backslashes in paths here, will break the way we generate
cache keys. convertPathToKey() currently replaces '/' with 0.
Changing every code path to accomodate the Windows specific path
separator is a lot more involved than allowing Windows to just
handle the forward slash paths.
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
We should be able to detect nolint comments that point to linters that
are disabled (such as with the removed structcheck).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Introduce a new compression.Type interface, which needs
to be implemented for each compression type, by that we can
reduce the number of switch case statements and ensure that
we don't miss the handle of any compression types, and also
make more easily for supporting new compression types.
This is a commit for code improvement, so no logical changes.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
This adds experimental support for Azure Blob Storage based remote cache to buildkit. For usage instructions please refer to the updated Readme. We have tried to keep it similar to the S3 based implementation while using equivalent Azure Blob Storage specific semantics where appropriate.
This also adds end-to-end tests to exercise the Azure Blob Storage based cache using the [Azurite emulator](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite).
Co-authored-by: Amr Mahdi <amrh@microsoft.com>
Co-authored-by: Pranav Pandit <pranavp@microsoft.com>
Signed-off-by: Pranav Pandit <pranavp@microsoft.com>
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Before this, if a ref with multiple parents loaded some parents but then
failed, the already loaded parents would not be released and thus
leaked.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, merge and diff refs were incorrectly not being marked as
committed, which meant that when cache was reloaded they would be marked
as mutable, which in turn lead to refs with them as an ancestor being
removed by the cache manager's init method. Additionally, if the merge/diff
ref was a parent, more inconsistent states could arise.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This centralizes the location where ProgressKey gets set, which works
because it only needs information about the vertex, nothing op-specific.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, if you called DiskUsage when there was a Diff ref that had
a nil upper and that ref had not yet had a blob set, the
layerDigestChain method would get a nil pointer exception.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This just makes sure the logic for the layer conversion is all in one
place and settable by a common option.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Before this change buildkit was changing the media type for
non-distributable layers to normal layers.
It was also clearing out the urls to get those blobs.
Now the layer mediatype and URL's are preserved.
If a layer blob is seen more than once, if it has extra URL's they will
be appended to the stored value.
On export there is now a new exporter option to preserve the
non-distributable data values.
All URL's seen by buildkit will be added to the exported content.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This allows clients to specify that LLB states should be grouped in
progress output under a custom name. Status updates for all vertexes in
the group will show up under a single vertex in the output.
The intended use cases are for Dockerfile COPY's that use MergeOp as a
backend and for grouping some other internal vertexes during frontend
builds.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
