I saw a panic while using the stargz snapshotter caused
by the labels being nil.
This adds a check to prevent the panic.
Signed-off-by: Chris Goller <goller@gmail.com>
This adds trace logs to the solver's cache manager to assist debugging
cache misses+hits.
It also replace the LazyStackTrace struct with a function that returns a
stack trace string only if trace level is enabled, an empty string
otherwise. This makes it easier to use with logrus hooks that pass the
fields around to different goroutines without needing to try to "unlazy"
the fields first.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Dedupe the several "containerd.io/uncompressed" constants and literals
into `github.com/containerd/containerd/labels.LabelUncompressed`
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The trace logs now also include the pointer value of the ref, which
serves as its "ID" amongst all the refs for the underlying record.
They also now include equal{Mutable,Immutable} IDs, which are otherwise
hard to determine with the Usage API.
Finally, LazyStackTrace now has a MarshalText method which returns the
same value as the existing String method. This allows it to work with
custom logrus hooks that rely on marshaling the fields.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Factorize code that check for an overlay mount type by using a function
instead.
This will allow supporting other overlay-based mount types more easily
in the future (for example fuse-overlayfs).
Signed-off-by: Alexis Murzeau <amubtdx@gmail.com>
Port https://github.com/moby/moby/blob/v23.0.1/daemon/oci_linux.go#L430-L460
> // Get the set of mount flags that are set on the mount that contains the given
> // path and are locked by CL_UNPRIVILEGED. This is necessary to ensure that
> // bind-mounting "with options" will not fail with user namespaces, due to
> // kernel restrictions that require user namespace mounts to preserve
> // CL_UNPRIVILEGED locked flags.
Fix issue 3098
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Nydus image is a container accelerated image format provided by the
Dragonfly image-service project, which offers the ability to pull
image data on demand, without waiting for the entire image pull to
complete and then start the container. It has been put in production
usage and shown vast improvements over the old OCI image format in
terms of container launching speed, image space, and network bandwidth
efficiency, as well as data integrity. Nydus image can be flexibly
configured as a FUSE-based user-space filesystem or in-kernel
EROFS (from Linux kernel v5.16) with Nydus daemon in user-space,
integrating with VM-based container runtime like KataContainers
is much easier.
Nydus has provided a conversion tool Nydusify for converting OCIv1
image to Nydus image and integrated into Harbor Acceld as a conversion
driver, which assumes that the OCI image is already available in the
registry, but a better way would be to build the Nydus images directly
from the build system instead of using the conversion tool, which would
increase the speed of the image export, so we experimentally integrated
the Nydus export in Buildkit.
Unlike other compression formats (gzip, estargz, etc.) in OCI image,
nydus is divided into two types of layer, blob, and bootstrap, where
blob serves as the data part of each layer of the image, and bootstrap
serves as the metadata of the whole image, the bootstrap is equivalent
to the view of the whole image filesystem after all layers overlay. For
example, for an OCI image with 3 layers, the corresponding nydus image
is 4 layers (3 layers of blob + 1 layer of bootstrap).
The nydus-snapshotter project provides a package to do the actual layer
compression, this commit imports the package to implement the export of
nydus compression type.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Introduce a new compression.Type interface, which needs
to be implemented for each compression type, by that we can
reduce the number of switch case statements and ensure that
we don't miss the handle of any compression types, and also
make more easily for supporting new compression types.
This is a commit for code improvement, so no logical changes.
Signed-off-by: Yan Song <imeoer@linux.alibaba.com>
Before this, if you called DiskUsage when there was a Diff ref that had
a nil upper and that ref had not yet had a blob set, the
layerDigestChain method would get a nil pointer exception.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This just makes sure the logic for the layer conversion is all in one
place and settable by a common option.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Before this change buildkit was changing the media type for
non-distributable layers to normal layers.
It was also clearing out the urls to get those blobs.
Now the layer mediatype and URL's are preserved.
If a layer blob is seen more than once, if it has extra URL's they will
be appended to the stored value.
On export there is now a new exporter option to preserve the
non-distributable data values.
All URL's seen by buildkit will be added to the exported content.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
compression-level option can be set on export to
define the preferred speed vs compression ratio. The
value is a number dependent on the compression algorithm.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This allows you to create refs that are single layers representing the
diff between any two arbitrary refs. The primary use case for this is
to allows users to extract the changes created by ops like Exec and
rebase them elsewhere through MergeOp. However, there is no restriction
on the inputs to DiffOp and the resulting ref's layer is simply the
layer created by running the differ on the two inputs refs
(specifically, the same differ used during exports).
A Diff ref can be mounted by itself, in which case it is defined as the
result of applying the diff to Scratch. Most use cases though will use
Diff refs as the input to a MergeOp, in which case the diff is just
applied on top of the lower merge inputs, as was the case before.
In cases like Diff(A, A->B->C) (i.e. cases where the diff is between two
refs where the lower is an ancestor of upper), the diff will be defined
as the layers separating the two refs. In other cases, the diff is just
a single layer, not re-used from the inputs, representing the diff
between the two refs (which can be defined as the layer "Diff(A,B)" that
satisfies "Merge(A, Diff(A,B)) == B").
Note that there is technically a meaningful difference between the
"unmerge" behavior of extracting the layers separating diffs and the
"simple diff" of just running the differ on the two refs. Namely, in the
case where there are "intermediate deletes" (i.e. deletes that only
exist in layers between A and B but not between A and B by themselves),
then the simple diff and unmerge can create different results when
plugged into a MergeOp. This is due to the fact that intermediate
deletes will apply to the merge when using the unmerge behavior, but not
when using the simple diff. This is on top of the fact that the simple
diff inherently has a "flattening" behavior where multiple layers are
squashed into a single one.
So, in the case where lower is an ancestor of upper, we choose to follow
the unmerge behavior, but it's possible users may prefer the simple diff
behavior. As of right now, they won't be able to do so, but if needed we
can add the ability to choose which behavior is followed in the future.
This could be done through a flag provided to DiffOp or possibly by
adapting llb.Copy to support this type of behavior with the same
efficiency as DiffOp.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This is mostly just preparation for merge-op. The existing
Extract method is updated to be usable for unlazying any type of refs
rather than just lazy blobs. The way views are created is simplified and
centralized in one location.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
The Parent method will no longer make sense with forthcoming Merge and
Diff support as refs will become capable of having multiple parents. It
was also only ever used externally to get the full chain of refs for
each layer in the ref's chain.
The newly added LayerChain method replaces Parents with a method that
just returns a slice of refs for each layer in the ref's chain. This
will work more seamlessly with Merge and Diff (in which case it returns
the "flattened" ancestors of the ref) in addition to being a bit easier
to use for the exiting cases anyways.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, descriptor handlers were not included with calls to the
exporter, which then sometimes called LoadRef and failed to get a ref
because it was lazy. This change results in the DescHandlers of the
already loaded refs to get plugged into context so they can be re-used
by the exporter if it needs to load the ref again.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
It turns out that while Buildkit code did not need this method to
be public, moby code does still use it, so we have to re-add it
after its removal in #2216 (commit b85ef15).
This commit is not a revert because some of the changes are
still desireable, namely the removal of the "commit" parameter
which didn't serve any purpose.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
won't make sense soon due to other potential changes. For example,
there can now be multiple blobs associated with a ref (for different
compression types), so the fact that you could access the "Blob"
field from the Info method on Ref incorrectly implied there was just
a single blob for the ref. This is on top of the fact that there is
no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
Previously, many parts of the code outside the cache package could
obtain the bolt storage item for any ref and read/write it directly.
This made it hard to understand what fields are used and when. Now,
the Metadata method has been removed from the Ref interface and
replaced with getters+setters for metadata fields we want to expose
outside the package, which makes it much easier to track and
understand. Similar changes have been made to the metadata search
interface.
3. Use a consistent getter+setter interface for metadata, replacing
the mix of interfaces like Metadata(), Size(), Info() and other
inconsistencies.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Previously, the flightcontrol group was being given a key just set to
the ref's ID, which meant that concurrent calls using different values
of compressionType, createIfNeeded and forceCompression would
incorrectly be de-duplicated.
The change here splits up the flightcontrol group into a few separate
calls and ensures that all the correct input variables are put into the
flightcontrol keys.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Finalize was only used outside the cache package in one place, which
called it with the commit arg set to false. The code path followed
when commit==false turned out to essentially be a no-op because
it set "retain cache" to true if it was already set to true.
It was thus safe to remove the only external call to it and remove it
from the interface. This should be helpful for future efforts to
simplify the equal{Mutable,Immutable} fields in cacheRecord, which exist
due to the "lazy commit" feature that Finalize is tied into.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
