Commit Graph

283 Commits

Author SHA1 Message Date
Sebastiaan van Stijn
6ff3e09d20 migrate platforms package to github.com/containerd/platforms
This updates the platforms package to be an alias for the new platforms module.
This helps transitioning consumers to the new module, and makes sure that
containerd v2 and v1 use the same definitions.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-06-13 00:58:28 +02:00
Sebastiaan van Stijn
0af6825b1e migrate logs imports to github.com/containerd/log module
Import the log module directly, instead of using the aliases that are
provided in the log package. This helps find code-paths that still depend
on the old location, and that haven't migrated yet to the new module.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-05-27 14:38:51 +02:00
Sebastiaan van Stijn
308341a446 replace uses of github.com/containerd/containerd/errdefs
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-05-26 13:21:40 +02:00
Danny Canter
bd423bf84d Snapshotters: Export the root path
Some of the snapshotters that allow you to change their root location
were already doing this, this just makes all of them follow the same
pattern.

Signed-off-by: Danny Canter <danny@dcantah.dev>
(cherry picked from commit 32caaee)
Signed-off-by: Kern Walster <walster@amazon.com>
2024-04-18 22:51:58 +00:00
Fu Wei
c558dc5cf2 Merge pull request #9837 from hinshun/backport/8945-and-9028
[release/1.7] Add WithMetaStore to overlay snapshotter and missing unpacker.Wait for image import
2024-02-22 11:16:15 +08:00
Robbie Buxton
31ea2d7d91 Add WithMetaStore to overlay snapshotter to allow bringing your own
Signed-off-by: Robbie Buxton <138501839+rbpdt@users.noreply.github.com>
2024-02-18 02:48:31 -05:00
Sebastiaan van Stijn
2ad2a2e835 switch back from golang.org/x/sys/execabs to os/exec (go1.19)
This is effectively a revert of 2ac9968401, which
switched from os/exec to the golang.org/x/sys/execabs package to mitigate
security issues (mainly on Windows) with lookups resolving to binaries in the
current directory.

from the go1.19 release notes https://go.dev/doc/go1.19#os-exec-path

> ## PATH lookups
>
> Command and LookPath no longer allow results from a PATH search to be found
> relative to the current directory. This removes a common source of security
> problems but may also break existing programs that depend on using, say,
> exec.Command("prog") to run a binary named prog (or, on Windows, prog.exe) in
> the current directory. See the os/exec package documentation for information
> about how best to update such programs.
>
> On Windows, Command and LookPath now respect the NoDefaultCurrentDirectoryInExePath
> environment variable, making it possible to disable the default implicit search
> of “.” in PATH lookups on Windows systems.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2af6db672e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-01-05 10:09:39 +01:00
Amit Barve
ed7c6895bd Don't block snapshot garbage collection on Remove failures
If a snapshot removal fails (during garbage collection), the entire garbage collection operation is
cancelled. This is problematic because once cleanup of any snapshot fails no other snapshots will be cleaned
and the disk usage will just keep increasing.
Solution is to return snapshot removal errors wrapped as "ErrFailedPrecondition" errors. The garbage
collectors continues cleanup if the error is of this type.

Signed-off-by: Amit Barve <ambarve@microsoft.com>
(cherry picked from commit ad96fded4c)
Signed-off-by: Amit Barve <ambarve@microsoft.com>
2023-12-07 09:20:54 -08:00
Danny Canter
ed4bac11e5 Blockfile: Enlighten blockfile copy on Darwin
The Go stdlib does not seem to have an efficient os.File.ReadFrom
routine for other platforms like it does on Linux with
copy_file_range. For Darwin at least we can use clonefile
in its place, otherwise if we have a sparse file we'd have
a fun surprise with the io.Copy approach..

We should see if there's other platforms that we can enhance here.
I've forgotten what's the right route on Windows.

Signed-off-by: Danny Canter <danny@dcantah.dev>
(cherry picked from commit 13ff185ba0)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-09-01 10:32:18 -07:00
Derek McGowan
0f4885c166 Add configurable mount options to overlay
Allows default mount options to be provided through configuration.

Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit d115129d1e)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-08-24 09:03:18 +08:00
Cardy.Tang
8804a27d72 feat: make overlay sync removal configurable
Signed-off-by: Cardy.Tang <zuniorone@gmail.com>
(cherry picked from commit b9f2e48e37)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2023-08-24 09:02:59 +08:00
Wei Fu
30a1630869 snapshots|pkg: umount without DETACH and nosync after umount
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 6dfb16f99a)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 05:23:08 -07:00
James Jenkins
1c2977db01 Modify loopback size
Modify the loopback size in the blockfile snapshotter test setup.
Set the loopback size to 16MB when the page size is greater than 4096.

Signed-off-by: James Jenkins <James.Jenkins@ibm.com>
(cherry picked from commit 5c480d9c88)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:14:32 -07:00
Wei Fu
cf9c78d2d9 snapshots/blockfile: fix lint issue
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit a207b430e3)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:14:24 -07:00
Wei Fu
b823b4f546 snapshots/blockfile: deflaky the testsuite
* Use direct-io mode to reduce IO.

* Add testViewHook helper to recovery the backing file since the ext4
  might need writable permission to handle recovery. If the backing file
  needs recovery and it's for View snapshot, the readonly mount will
  cause error.

* Use 8 MiB as capacity to reduce the IO.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 7de95cbc4c)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:14:15 -07:00
Cardy.Tang
6daeceb116 snapshots/blockfile: use passed in dst in scratchGenerator
Signed-off-by: Cardy.Tang <zuniorone@gmail.com>
(cherry picked from commit 9702d4aeec)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:13:59 -07:00
Wei Fu
99b61aee34 snapshots/blockfile: use loop if options is empty
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit bb82f3683e)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:13:50 -07:00
Derek McGowan
fd39769ac7 Add mount options to blockfile snapshotter
Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit 7beaa5e859)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:13:40 -07:00
Derek McGowan
8d732c6c22 Add sync before unmount on snapshotter layer test
Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit 1be571b14f)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:10:54 -07:00
Derek McGowan
07cc24b32a Add blockfile snapshotter
Signed-off-by: Derek McGowan <derek@mcg.dev>
(cherry picked from commit 0c386e2ebd)
Signed-off-by: Danny Canter <danny@dcantah.dev>
2023-08-18 03:10:27 -07:00
Ben Foster
2eaeb3205a Fix ro mount option being passed
"ro" was not parsed out of the string, so it was passed as part of data
to mount().
This would lead to mount() returning an invalid argument code.
Separate out the "ro" option, much like "userxattr", which will allow
the MS-RDONLY mountflag to get set.

Signed-off-by: Ben Foster <bpfoster@gmail.com>
(cherry picked from commit f3daf32c73)
2023-07-28 15:08:45 -04:00
Gabriel Adrian Samfira
acff3eefa6 Improve error messages and remove check
* Improve error messages
  * remove a check for the existance of unmount target. We probably
    should not mask that the target was missing.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:12:32 +03:00
Gabriel Adrian Samfira
8f37b1c637 Remove "bind" code path from diff
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:12:12 +03:00
Gabriel Adrian Samfira
9139208b31 Properly mount base layers
As opposed to a writable layer derived from a base layer, the volume
path of a base layer, once activated and prepared will not be a WCIFS
volume, but the actual path on disk to the snapshot. We cannot directly
mount this folder, as that would mean a client may gain access and
potentially damage important metadata files that would render the layer
unusabble.

For base layers we need to mount the Files folder which must exist in
any valid base windows-layer.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:12:08 +03:00
Gabriel Adrian Samfira
e61e7b3126 Skip parent layer options on bind mounts
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:11:58 +03:00
Gabriel Adrian Samfira
0277b9b01a Remove escalated privileges
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:11:40 +03:00
Gabriel Adrian Samfira
375172604d Fix go.mod, simplify boolean logic, add logging
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:09:52 +03:00
Gabriel Adrian Samfira
96fbe5bc88 Fix layer comparison and enable read-only checks
fstest.CheckDirectoryEqual checks if any files in the diff matches a
list of known metadataFiles. This only happens if we specify the initial
layer as the first parameter and the mutated layer as the second.

This also enables the read-only view checks, as the bind filter allows
us to mount a layer as ro.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:06:28 +03:00
Gabriel Adrian Samfira
279e0d3c9a Use bind filer for mounts
The bind filter supports bind-like mounts and volume mounts. It also
allows us to have read-only mounts.

Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
2023-05-31 02:06:24 +03:00
Paul "TBBle" Hampson
3a3da693ae Run Windows snapshotter through the test suite
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:06:12 +03:00
Paul "TBBle" Hampson
e7b62322f9 Fix misspelling of 'Native' as 'Naive'
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:06:08 +03:00
Paul "TBBle" Hampson
e1f999a182 Add paired 'mount' log for 'unmount'
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:06:01 +03:00
Paul "TBBle" Hampson
5788d6e520 Don't use all-upper-case filenames in snapshot tests
NTFS, when presented with an all-caps filename, assumes you are just
being loud for no reason, and instead stores an all-lower-case filename.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:05:56 +03:00
Paul "TBBle" Hampson
3cdcb2f108 Skip tests that do not apply to WCOW on Windows
Filesystem permissions and ownership are not modifiable via an image
mount.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:05:51 +03:00
Paul "TBBle" Hampson
b0968b8bb9 Ensure mounts are unmounted before leaving the test
This is necessary on Windows, as it's not possible to delete a snapshot
while it is still mounted, even if the mount-point has been deleted.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:05:44 +03:00
Paul "TBBle" Hampson
1a64ee1835 Implement WCOW parentless active snapshots and view snapshots
The WCOW layer support does not support creating sandboxes with no
parent.  Instead, parentless scratch layers must be layed out as a
directory containing only a directory named 'Files', and all data stored
inside 'Files'. At commit-time, this will be converted in-place into a
read-only layer suitable for use as a parent layer.

The WCOW layer support also does not deal with making read-only layers,
i.e. layers that are prepared to be parent layers, visible in a
read-only manner. A bind-mount or junction point cannot be made
read-only, so a view must instead be a small sandbox layer that we can
mount via WCOW, and discard later, to protect the layer against
accidental or deliberate modification.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2023-05-31 02:05:20 +03:00
Akihiro Suda
9e60300ea0 snapshots/testsuite: Rename: fix fuse-overlayfs incompatibility
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
(cherry picked from commit 5dbae38191)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-05-12 08:54:13 +09:00
Fu Wei
8cb00f45c9 Merge pull request #8143 from mxpv/log
Add Fields type alias to log package
2023-02-21 10:22:23 +08:00
Maksym Pavlenko
06e085c8b5 Add Fields type alias to log package
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2023-02-20 17:29:08 -08:00
Akihiro Suda
d8b68e3ccc Stop using math/rand.Read and rand.Seed (deprecated in Go 1.20)
From golangci-lint:

> SA1019: rand.Read has been deprecated since Go 1.20 because it
>shouldn't be used: For almost all use cases, crypto/rand.Read is more
>appropriate. (staticcheck)

> SA1019: rand.Seed has been deprecated since Go 1.20 and an alternative
>has been available since Go 1.0: Programs that call Seed and then expect
>a specific sequence of results from the global random source (using
>functions such as Int) can be broken when a dependency changes how
>much it consumes from the global random source. To avoid such breakages,
>programs that need a specific result sequence should use
>NewRand(NewSource(seed)) to obtain a random generator that other
>packages cannot access. (staticcheck)

See also:

- https://pkg.go.dev/math/rand@go1.20#Read
- https://pkg.go.dev/math/rand@go1.20#Seed

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-02-16 03:50:23 +09:00
Phil Estes
024a748c09 Merge pull request #7933 from AkihiroSuda/drop-libbtrfs
btrfs: depend on kernel UAPI instead of libbtrfs
2023-02-10 16:13:19 -05:00
Akihiro Suda
52f82acb7b btrfs: depend on kernel UAPI instead of libbtrfs
See containerd/btrfs PR 40 and moby/moby PR 44761. (Thanks to [@]neersighted.)

The containerd/btrfs library now requires headers from kernel 4.12 or newer:
- https://github.com/torvalds/linux/blob/master/include/uapi/linux/btrfs.h
- https://github.com/torvalds/linux/blob/master/include/uapi/linux/btrfs_tree.h

These files are licensed under the GPL-2.0 WITH Linux-syscall-note, so it should be compatible with the Apache License 2.0.
https://spdx.org/licenses/Linux-syscall-note.html

The dependency on the kernel headers only affects users building from source.
Users on older kernels may opt to not compile this library (`BUILDTAGS=no_btfs`),
or to provide headers from a newer kernel.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2023-02-10 10:07:34 +09:00
Swagat Bora
6ae3e5df6a Fix retry logic within devmapper device deactivation
Signed-off-by: Swagat Bora <sbora@amazon.com>
2023-02-09 23:40:18 +00:00
Edgar Lee
34d5878185 Use mount.Target to specify subdirectory of rootfs mount
- Add Target to mount.Mount.
- Add UnmountMounts to unmount a list of mounts in reverse order.
- Add UnmountRecursive to unmount deepest mount first for a given target, using
moby/sys/mountinfo.

Signed-off-by: Edgar Lee <edgarhinshunlee@gmail.com>
2023-01-27 09:51:58 +08:00
iyear
9df5a1714d snapshots: refactor metastore transaction
Signed-off-by: Junyu Liu <ljyngup@gmail.com>
2023-01-05 15:53:23 +08:00
Samuel Karp
b0b28f1d8e Merge pull request #7879 from fuweid/clean-build-tags 2022-12-30 00:22:03 -08:00
Mike Brown
66f186d42d Merge pull request #7679 from kinvolk/rata/userns-stateless-pods
Add support for user namespaces in stateless pods (KEP-127)
2022-12-29 14:08:24 -06:00
Wei Fu
6b7e237fc7 chore: use go fix to cleanup old +build buildtag
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2022-12-29 14:25:14 +08:00
Derek McGowan
cfe7ac9956 Merge pull request #7529 from iyear/refactor-metastore-tx
Refactor metastore transaction
2022-12-28 20:08:38 -08:00
iyear
1d0619bc0c Refactor metastore transaction
Signed-off-by: Junyu Liu <ljyngup@gmail.com>
2022-12-28 18:37:28 +08:00