Building off insights from the great work Cory Snider has been doing,
this replaces a reexec with a much lower overhead implementation which
performs the `Chddir` in a new goroutine that is locked to a specific
thread with CLONE_FS unshared.
The thread is thrown away afterwards and the Chdir does effectively the
same thing as what the reexec was being used for.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
go-winio now defines this function, so we can consume that.
Note that there's a difference between the old implementation and the original
one (added in 1cb9e9b44e). The old implementation
had special handling for win32 error codes, which was removed in the go-winio
implementation in 0966e1ad56
As `go-winio.GetFileSystemType()` calls `filepath.VolumeName(path)` internally,
this patch also removes the `string(home[0])`, which is redundant, and could
potentially panic if an empty string would be passed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 955c1f881a (Docker v17.12.0) replaced
detection of support for multiple lowerdirs (as required by overlay2) to not
depend on the kernel version. The `overlay2.override_kernel_check` was still
used to print a warning that older kernel versions may not have full support.
After this, commit e226aea280 (Docker v20.10.0,
backported to v19.03.7) removed uses of the `overlay2.override_kernel_check`
option altogether, but we were still parsing it.
This patch changes the `parseOptions()` function to not parse the option,
printing a deprecation warning instead. We should change this to be an error,
but the `overlay2.override_kernel_check` option was not deprecated in the
documentation, so keeping it around for one more release.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
On Linux, when (os/exec.Cmd).SysProcAttr.Pdeathsig is set, the signal
will be sent to the process when the OS thread on which cmd.Start() was
executed dies. The runtime terminates an OS thread when a goroutine
exits after being wired to the thread with runtime.LockOSThread(). If
other goroutines are allowed to be scheduled onto a thread which called
cmd.Start(), an unrelated goroutine could cause the thread to be
terminated and prematurely signal the command. See
https://github.com/golang/go/issues/27505 for more information.
Prevent started subprocesses with Pdeathsig from getting signaled
prematurely by wiring the starting goroutine to the OS thread until the
subprocess has exited. No other goroutines can be scheduled onto a
locked thread so it will remain alive until unlocked or the daemon
process exits.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The pkg/fsutils package was forked in containerd, and later moved to
containerd/continuity/fs. As we're moving more bits to containerd, let's also
use the same implementation to reduce code-duplication and to prevent them from
diverging.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
daemon/graphdriver/aufs/aufs.go:239:80: empty-lines: extra empty line at the start of a block (revive)
daemon/graphdriver/graphtest/graphbench_unix.go:249:27: empty-lines: extra empty line at the start of a block (revive)
daemon/graphdriver/graphtest/testutil.go:271:30: empty-lines: extra empty line at the end of a block (revive)
daemon/graphdriver/graphtest/graphbench_unix.go:179:32: empty-block: this block is empty, you can remove it (revive)
daemon/graphdriver/zfs/zfs.go:375:48: empty-lines: extra empty line at the end of a block (revive)
daemon/graphdriver/overlay/overlay.go:248:89: empty-lines: extra empty line at the start of a block (revive)
daemon/graphdriver/devmapper/deviceset.go:636:21: empty-lines: extra empty line at the end of a block (revive)
daemon/graphdriver/devmapper/deviceset.go:1150:70: empty-lines: extra empty line at the start of a block (revive)
daemon/graphdriver/devmapper/deviceset.go:1613:30: empty-lines: extra empty line at the end of a block (revive)
daemon/graphdriver/devmapper/deviceset.go:1645:65: empty-lines: extra empty line at the start of a block (revive)
daemon/graphdriver/btrfs/btrfs.go:53:101: empty-lines: extra empty line at the start of a block (revive)
daemon/graphdriver/devmapper/deviceset.go:1944:89: empty-lines: extra empty line at the start of a block (revive)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Remove the "deadcode", "structcheck", and "varcheck" linters, as they are
deprecated:
WARN [runner] The linter 'deadcode' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [runner] The linter 'structcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [runner] The linter 'varcheck' is deprecated (since v1.49.0) due to: The owner seems to have abandoned the linter. Replaced by unused.
WARN [linters context] structcheck is disabled because of generics. You can track the evolution of the generics support by following the https://github.com/golangci/golangci-lint/issues/2649.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Driver abstraction was needed for Linux Containers on Windows,
support for which has since been removed.
There is no direct equivalent to Lchmod() in the standard library so
continue to use the containerd/continuity version.
Signed-off-by: Cory Snider <csnider@mirantis.com>
doCopyXattrs() never reached due to copyXattrs boolean being false, as
a result file capabilities not being copied.
moved copyXattr() out of doCopyXattrs()
Signed-off-by: Illo Abdulrahim <abdulrahim.illo@nokia.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Older versions of Go don't format comments, so committing this as
a separate commit, so that we can already make these changes before
we upgrade to Go 1.19.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These seemed to prevent cleaning up directories;
On arm64:
=== RUN TestSysctlOverride
testing.go:1090: TempDir RemoveAll cleanup: unlinkat /tmp/TestSysctlOverride2860094781/001/mounts/shm: device or resource busy
--- FAIL: TestSysctlOverride (0.00s)
On Windows:
=== Failed
=== FAIL: github.com/docker/docker/daemon TestLoadOrCreateTrustKeyInvalidKeyFile (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestLoadOrCreateTrustKeyInvalidKeyFile2014634395\001\keyfile4156691647: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/daemon/graphdriver TestIsEmptyDir (0.01s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestIsEmptyDir1962964337\001\dir-with-empty-file\file2523853824: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/pkg/directory TestSizeEmptyFile (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeEmptyFile1562416712\001\file16507846: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/pkg/directory TestSizeNonemptyFile (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeNonemptyFile1240832785\001\file3265662846: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/pkg/directory TestSizeFileAndNestedDirectoryEmpty (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeFileAndNestedDirectoryEmpty2163416550\001\file3715413181: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/pkg/directory TestSizeFileAndNestedDirectoryNonempty (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeFileAndNestedDirectoryNonempty878205470\001\file3280422273: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/volume/service TestSetGetMeta (0.01s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSetGetMeta3332268057\001\db: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/volume/service TestList (0.03s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestList2846947953\001\volumes\metadata.db: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/volume/service TestRestore (0.02s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestRestore3368254142\001\volumes\metadata.db: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/daemon/graphdriver TestIsEmptyDir (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestIsEmptyDir2823795693\001\dir-with-empty-file\file2625561089: The process cannot access the file because it is being used by another process.
=== FAIL: github.com/docker/docker/pkg/directory TestSizeFileAndNestedDirectoryNonempty (0.00s)
testing.go:1090: TempDir RemoveAll cleanup: remove C:\Users\CONTAI~1\AppData\Local\Temp\TestSizeFileAndNestedDirectoryNonempty4246252950\001\nested3442260313\file21164327: The process cannot access the file because it is being used by another process.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Support for overlay on a backing filesystem without d_type was deprecated in
0abb8dec3f (Docker 17.12), with an exception
for existing installations (0a4e793a3d).
That deprecation was nearly 5 years ago, and running without d_type is known to
cause serious issues (so users will likely already have run into other problems).
This patch removes support for running overlay and overlay2 on these filesystems,
returning the error instead of logging it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously, we only printed a warning if a storage driver was deprecated. The
intent was to continue supporting these drivers, to allow users to migrate
to a different storage driver.
This patch changes the behavior; if the user has no storage driver specified
in the daemon configuration (so if we try to detect the previous storage
driver based on what's present in /var/lib/docker), we now produce an error,
informing the user that the storage driver is deprecated (and to be removed),
as well as instructing them to change the daemon configuration to explicitly
select the storage driver (to allow them to migrate).
This should make the deprecation more visible; this will be disruptive, but
it's better to have the failure happening *now* (while the drivers are still
there), than for users to discover the storage driver is no longer there
(which would require them to *downgrade* the daemon in order to migrate
to a different driver).
With this change, `docker info` includes a link in the warnings that:
/ # docker info
Client:
Context: default
Debug Mode: false
Server:
...
Live Restore Enabled: false
WARNING: The overlay storage-driver is deprecated, and will be removed in a future release.
Refer to the documentation for more information: https://docs.docker.com/go/storage-driver/
When starting the daemon without a storage driver configured explicitly, but
previous state was using a deprecated driver, the error is both logged and
printed:
...
ERRO[2022-03-25T14:14:06.032014013Z] [graphdriver] prior storage driver overlay is deprecated and will be removed in a future release; update the the daemon configuration and explicitly choose this storage driver to continue using it; visit https://docs.docker.com/go/storage-driver/ for more information
...
failed to start daemon: error initializing graphdriver: prior storage driver overlay is deprecated and will be removed in a future release; update the the daemon configuration and explicitly choose this storage driver to continue using it; visit https://docs.docker.com/go/storage-driver/ for more information
When starting the daemon and explicitly configuring it with a deprecated storage
driver:
WARN[2022-03-25T14:15:59.042335412Z] [graphdriver] WARNING: the overlay storage-driver is deprecated and will be removed in a future release; visit https://docs.docker.com/go/storage-driver/ for more information
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- use pkg/errors for errors and fix error-capitalisation
- remove one redundant call to logDeprecatedWarning() (we're already skipping
deprecated drivers in that loop).
- rename `list` to `priorityList` for readability.
- remove redundant "skip" for the vfs storage driver, as it's already
excluded by `scanPriorDrivers()`
- change one debug log to an "info", so that the daemon logs contain the driver
that was configured, and include "multiple prior states found" error in the
daemon logs, to assist in debugging failed daemon starts.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Finish the refactor which was partially completed with commit
34536c498d, passing around IdentityMapping structs instead of pairs of
[]IDMap slices.
Existing code which uses []IDMap relies on zero-valued fields to be
valid, empty mappings. So in order to successfully finish the
refactoring without introducing bugs, their replacement therefore also
needs to have a useful zero value which represents an empty mapping.
Change IdentityMapping to be a pass-by-value type so that there are no
nil pointers to worry about.
The functionality provided by the deprecated NewIDMappingsFromMaps
function is required by unit tests to to construct arbitrary
IdentityMapping values. And the daemon will always need to access the
mappings to pass them to the Linux kernel. Accommodate these use cases
by exporting the struct fields instead. BuildKit currently depends on
the UIDs and GIDs methods so we cannot get rid of them yet.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Trying to build Docker images with buildkit using a ZFS-backed storage
was unreliable due to apparent race condition between adding and
removing layers to the storage (see: https://github.com/moby/buildkit/issues/1758).
The issue describes a similar problem with the BTRFS driver that was
resolved by adding additional locking based on the scheme used in the
OverlayFS driver. This commit replicates the scheme to the ZFS driver
which makes the problem as reported in the issue stop happening.
Signed-off-by: Tomasz Mańko <hi@jaen.me>
pkg/system historically has been a bit of a kitchen-sink of things that were
somewhat "system" related, but didn't have a good place for. EnsureRemoveAll()
is one of those utilities. EnsureRemoveAll() is used to both unmount and remove
a path, for which it depends on both github.com/moby/sys/mount, which in turn
depends on github.com/moby/sys/mountinfo.
pkg/system is imported in the CLI, but neither EnsureRemoveAll(), nor any of its
moby/sys dependencies are used on the client side, so let's move this function
somewhere else, to remove those dependencies from the CLI.
I looked for plausible locations that were related; it's used in:
- daemon
- daemon/graphdriver/XXX/
- plugin
I considered moving it into a (e.g.) "utils" package within graphdriver (but not
a huge fan of "utils" packages), and given that it felt (mostly) related to
cleaning up container filesystems, I decided to move it there.
Some things to follow-up on after this:
- Verify if this function is still needed (it feels a bit like a big hammer in
a "YOLO, let's try some things just in case it fails")
- Perhaps it should be integrated in `containerfs.Remove()` (so that it's used
automatically)
- Look if there's other implementations (and if they should be consolidated),
although (e.g.) the one in containerd is a copy of ours:
https://github.com/containerd/containerd/blob/v1.5.9/pkg/cri/server/helpers_linux.go#L200
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
daemon/graphdriver/fuse-overlayfs/fuseoverlayfs.go:101:63: SA9002: file mode '700' evaluates to 01274; did you mean '0700'? (staticcheck)
if err := idtools.MkdirAllAndChown(path.Join(home, linkDir), 700, currentID); err != nil {
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use the syscall method instead of repeating the type conversions for
the syscall.Stat_t Atim/Mtim members. This also allows to drop the
//nolint: unconvert comments.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Do not use 0701 perms.
0701 dir perms allows anyone to traverse the docker dir.
It happens to allow any user to execute, as an example, suid binaries
from image rootfs dirs because it allows traversal AND critically
container users need to be able to do execute things.
0701 on lower directories also happens to allow any user to modify
things in, for instance, the overlay upper dir which neccessarily
has 0755 permissions.
This changes to use 0710 which allows users in the group to traverse.
In userns mode the UID owner is (real) root and the GID is the remapped
root's GID.
This prevents anyone but the remapped root to traverse our directories
(which is required for userns with runc).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit ef7237442147441a7cadcda0600be1186d81ac73)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 93ac040bf0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The io/ioutil package has been deprecated in Go 1.16. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Add hints for "Failed to destroy btrfs snapshot <DIR> for <ID>: operation not permitted" on rootless
Related to issue 41762
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The daemon uses a priority list to automatically select the best-matching storage
driver for the backing filesystem that is used.
Historically, overlay2 was not supported on Btrfs and ZFS, and the daemon would
automatically pick the `btrfs` or `zfs` storage driver if that was the Backing
File System.
Commits 649e4c8889 and e226aea280
improved our detection to check if overlay2 was supported on the backing file-
system, allowing overlay2 to be used on top of Btrfs or ZFS, but did not change
the priority list.
While both Btrfs and ZFS have advantages for certain use-cases, and provide
advanced features that are not available to overlay2, they also are known
to require more "handholding", and are generally considered to be mostly
useful for "advanced" users.
This patch changes the storage-driver priority list, to prefer overlay2 (if
supported by the backing filesystem), and effectively makes btrfs and zfs
opt-in storage drivers.
This change does not affect existing installations; the daemon will detect
the storage driver that was previously in use (based on the presence of
storage directories in `/var/lib/docker`).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Probably needs a similar change as c208f03fbd,
but this code makes my head spin, so for now suppressing, and created a
tracking issue:
daemon/graphdriver/graphtest/graphtest_unix.go:305:12: unsafeptr: possible misuse of reflect.SliceHeader (govet)
header := *(*reflect.SliceHeader)(unsafe.Pointer(&buf))
^
daemon/graphdriver/graphtest/graphtest_unix.go:308:36: unsafeptr: possible misuse of reflect.SliceHeader (govet)
data := *(*[]byte)(unsafe.Pointer(&header))
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
