TestInspectNetwork creates 4 daemons (3 managers + 1 worker) per run
but never called Cleanup, leaving overlay storage, Raft state and logs
on disk. When the stress runner executes the test 10 times in a row
with -test.count 10, the accumulated directories fill the disk:
no space left on device: could not create daemon root
".../bundles/test-integration-flaky/TestInspectNetwork/d8488af0.../root"
Defer Cleanup after Stop for every daemon. Defers run LIFO so Stop
fires before Cleanup for each daemon, which is required because
Cleanup unmounts overlay mounts the running daemon still holds.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
SwarmKit's Raft election timeout is 10 seconds (ElectionTick=10 ×
TickInterval=1 s). RestartNode (Stop+Start in sequence) can complete
in under 10 s on a fast machine, so followers may never detect a
leader gap: the restarted node reconnects before their election timer
fires and resumes leadership unchanged. This is the race that makes
the subtest flaky.
Fix by separating the stop and start steps:
1. Stop the leader and do NOT restart it immediately.
2. Poll a standby manager's API (standbyCli) until a different leader
is elected. Using a non-leader manager avoids depending on the
stopped node's API, which is unavailable during this window.
3. Start the stopped node back up. The new leader has advanced the
Raft term, so the rejoining node is forced to become a follower.
4. Wait for c1 to be responsive again (HasLeader via c1) before
asserting and running the inspect checks.
With the leader fully stopped (not immediately restarted), the
remaining two managers must elect a new leader - they have quorum
and the original node cannot reconnect to reset their election timer.
The election fires within the 10 s timeout, well inside NetworkPoll's
30 s ceiling (50 s on arm). The three-retry loop is removed; the
approach is deterministic.
Add HasLeaderOtherThan to integration/internal/swarm/states.go. Errors
from NodeList are treated as poll.Continue rather than poll.Error so a
brief period of cluster unavailability during the election does not
abort the wait.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add a new type to use for building filter predicates for API requests,
replacing "./api/types/filters".Args in the client. Remove the now
unused api/types/filters package.
Signed-off-by: Cory Snider <csnider@mirantis.com>
As the Engine API requests may be directed at a non-leader Swarm
manager, the information needs to be tunneled through the Swarm API.
Signed-off-by: Cory Snider <csnider@mirantis.com>
This package was originally internal, but was moved out when BuildKit
used it for its integration tests. That's no longer the case, so we
can make it internal again.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These comments were added to enforce using the correct import path for
our packages ("github.com/docker/docker", not "github.com/moby/moby").
However, when working in go module mode (not GOPATH / vendor), they have
no effect, so their impact is limited.
Remove these imports in preparation of migrating our code to become an
actual go module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
integration/network/inspect_test.go:77:3: The copy of the 'for' variable "tc" can be deleted (Go 1.22+) (copyloopvar)
tc := tc
^
integration/network/network_test.go:81:3: The copy of the 'for' variable "ep" can be deleted (Go 1.22+) (copyloopvar)
ep := ep
^
integration/network/network_test.go:143:3: The copy of the 'for' variable "ep" can be deleted (Go 1.22+) (copyloopvar)
ep := ep
^
integration/network/network_test.go:221:3: The copy of the 'for' variable "tc" can be deleted (Go 1.22+) (copyloopvar)
tc := tc
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Fixes#18864, #20648, #33561, #40901.
[This GH comment][1] makes clear network name uniqueness has never been
enforced due to the eventually consistent nature of Classic Swarm
datastores:
> there is no guaranteed way to check for duplicates across a cluster of
> docker hosts.
And this is further confirmed by other comments made by @mrjana in that
same issue, eg. [this one][2]:
> we want to adopt a schema which can pave the way in the future for a
> completely decentralized cluster of docker hosts (if scalability is
> needed).
This decentralized model is what Classic Swarm was trying to be. It's
been superseded since then by Docker Swarm, which has a centralized
control plane.
To circumvent this drawback, the `NetworkCreate` endpoint accepts a
`CheckDuplicate` flag. However it's not perfectly reliable as it won't
catch concurrent requests.
Due to this design decision, API clients like Compose have to implement
workarounds to make sure names are really unique (eg.
docker/compose#9585). And the daemon itself has seen a string of issues
due to that decision, including some that aren't fixed to this day (for
instance moby/moby#40901):
> The problem is, that if you specify a network for a container using
> the ID, it will add that network to the container but it will then
> change it to reference the network by using the name.
To summarize, this "feature" is broken, has no practical use and is a
source of pain for Docker users and API consumers. So let's just remove
it for _all_ API versions.
[1]: https://github.com/moby/moby/issues/18864#issuecomment-167201414
[2]: https://github.com/moby/moby/issues/18864#issuecomment-167202589
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Integration tests will now configure clients to propagate traces as well
as create spans for all tests.
Some extra changes were needed (or desired for trace propagation) in the
test helpers to pass through tracing spans via context.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This field was added in f0e5b3d7d8 to
account for older versions of the engine (Docker EE LTS versions), which
did not yet provide the OSType field in Docker info, and had to be manually
set using the TEST_OSTYPE env-var.
This patch removes the field in favor of the equivalent in DaemonInfo. It's
more verbose, but also less ambiguous what information we're using (i.e.,
the platform the daemon is running on, not the local platform).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix moves multiple places of serviceRunningTasksCount
to one location in integration/internal/swarm, so that
code duplication could be removed.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This introduces `NoTasksForService` and `NoTasks` poller checks, that
can be used to check if no tasks are left in general, or for a specific
service.
Some redundant checks were also removed from some tests.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Clean up and refactor this test;
- make `serviceRunningTasksCount` to use a `desired-state` filter
- use subtests, and inline the `validNetworkVerbose` checks; also use
asserts for the individual checks, so that any failure will log exactly
what failed
- remove helper functions that are no longer needed
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fix adds `t.Name()` to tests in integration/network
so that services created in those tests have unique names.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Since Go 1.7, context is a standard package. Since Go 1.9, everything
that is provided by "x/net/context" is a couple of type aliases to
types in "context".
Many vendored packages still use x/net/context, so vendor entry remains
for now.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
… and do not use the `docker` cli in it. One of the reason of this
move is to not make `integration` package using legacy
`integration-cli` package.
Next move will be to support swarm within this package *and* provide
some helper function using the api (compared to the one using cli in
`integration-cli/daemon` package).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Using the default PollSettings functions to adjust the timeout
value instead of changing the value each time when needed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
Service of inspect network can't be finished within 10s on AArch64 platform,
so we need to adjust the timeout value avoid to paper cover the real issue, plus
to make the integreation test can continue while not terminate with below error:
> === RUN TestInspectNetwork
> --- FAIL: TestInspectNetwork (27.65s)
> daemon.go:285: [de79880f4ed4a] waiting for daemon to start
> daemon.go:317: [de79880f4ed4a] daemon started
> inspect_test.go:57: timeout hit after 10s: waiting for tasks to enter run state
> daemon.go:275: [de79880f4ed4a] exiting daemon
> FAIL
> ---> Making bundle: .integration-daemon-stop (in bundles/test-integration)
> Removing test suite binaries
> Makefile:171: recipe for target 'test-integration' failed
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
