The loopback-test fixes seem to be sufficient to resolve the flakiness
of all the tests aside from TestFlakyNetworkDBIslands.
Signed-off-by: Cory Snider <csnider@mirantis.com>
NetworkDB defaults to binding to the unspecified address for gossip
communications, with no advertise address set. In this configuration,
the memberlist instance listens on all network interfaces and picks one
of the host's public IP addresses as the advertise address.
The NetworkDB unit tests don't override this default, leaving them
vulnerable to flaking out as a result of rogue network traffic
perturbing the test, or the inferred advertise address not being useable
for loopback testing. And macOS prompts for permission to allow the test
executable to listen on public interfaces every time it is rebuilt.
Modify the NetworkDB tests to explicitly bind to, advertise, and join
ports on 127.0.0.1 to make the tests more robust to flakes in CI and
more convenient to run locally.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The NetworkDB unit tests instantiate clusters which communicate over
loopback where every "node" listens on a distinct localhost port. The
tests make use of a NetworkDB configuration knob to set the port. When
the NetworkDB configuration's BindPort field is set to a nonzero value,
its memberlist instance is configured to bind to the specified port
number. However, the advertise port is left at the
memberlist.DefaultLANConfig() default value of 7946. Because of this,
nodes would be unable to contact any of the other nodes in the cluster
learned by gossip as the gossiped addresseses specify the wrong ports!
The flaky tests passed as often as they did thanks to the robustness of
the memberlist module: NetworkDB gossip and and memberlist node
liveness-probe pings to unreachable nodes can all be relayed through
the reachable nodes, the nodes on the bootstrap join list.
Make the NetworkDB unit tests less flaky by setting each node's
advertise port to the bind port.
The daemon is unaffected by this oversight as it unconditionally uses
the default listen port of 7946, which aligns with the advertise port.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Per-network option com.docker.network.bridge.trusted-host-interfaces
accepts a list of interfaces that are allowed to route
directly to a container's published ports in a bridge
network with nat enabled.
This daemon level option disables direct access filtering,
enabling direct access to published ports on container
addresses in all bridge networks, via all host interfaces.
It overlaps with short-term env-var workaround:
DOCKER_INSECURE_NO_IPTABLES_RAW=1
- it does not allow packets sent from outside the host to reach
ports published only to 127.0.0.1
- it will outlive iptables (the workaround was initially intended
for hosts that do not have kernel support for the "raw" iptables
table).
Signed-off-by: Rob Murray <rob.murray@docker.com>
trusted_host_interface have access to published ports on container
addresses - enabling direct routing to the container via those
interfaces.
Signed-off-by: Rob Murray <rob.murray@docker.com>
libnetwork/network.go:637:6: shadow: declaration of "v" shadows declaration at line 634 (govet)
if v, ok := n.generic[netlabel.GenericData]; ok {
^
libnetwork/service_linux.go:125:7: shadow: declaration of "ep" shadows declaration at line 86 (govet)
if ep := sb.getGatewayEndpoint(); ep != nil {
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
libnetwork/portallocator/portallocator_test.go:151:7: shadow: declaration of "i" shadows declaration at line 150 (govet)
for i := 0; i <= p.end-p.begin; i++ {
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
also moved the variables closer to where they're used
libnetwork/networkdb/cluster.go:414:4: shadow: declaration of "nid" shadows declaration at line 397 (govet)
nid := params[0]
^
libnetwork/networkdb/networkdb.go:514:4: shadow: declaration of "nid" shadows declaration at line 506 (govet)
nid := params[0]
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Commit 27adcd5 ("libnet/d/bridge: drop connections to lo mappings, and
direct remote connections") introduced an iptables rule to drop 'direct'
remote connections made to the container's IP address - for each
published port on the container.
The normal filter-FORWARD rules would then drop packets sent directly to
unpublished ports. This rule was only created along with the rest of port
publishing (when a container's endpoint was selected as its gateway). Until
then, all packets addressed directly to the container's ports were dropped
by the filter-FORWARD rules.
But, the rule doesn't need to be per-port. Just drop packets sent
directly to a container's address unless they originate on the host.
That means fewer rules, that can be created along with the endpoint (then
directly-routed get dropped at the same point whether or not the endpoint
is currently the gateway - very slightly earlier than when it's not the
gateway).
Signed-off-by: Rob Murray <rob.murray@docker.com>
Go maintainers started to unconditionally update the minimum go version
for golang.org/x/ dependencies to go1.23, which means that we'll no longer
be able to support any version below that when updating those dependencies;
> all: upgrade go directive to at least 1.23.0 [generated]
>
> By now Go 1.24.0 has been released, and Go 1.22 is no longer supported
> per the Go Release Policy (https://go.dev/doc/devel/release#policy).
>
> For golang/go#69095.
This updates our minimum version to go1.23, as we won't be able to maintain
compatibility with older versions because of the above.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Since commit 925b484 ("No fallback nameservers for internal
resolver"), if the host's resolv.conf has no nameservers and
no servers are supplied via config, the internal resolver will
not use Google's DNS - so the container will not be able to
resolve external DNS requests.
That can happen when container's are "restart-always" and the
docker daemon starts before the host's DNS is configured.
So, to highlight the issue (which may not be an error, but
probably is), include a warning in the container's resolv.conf
file.
Also, log a warning - logs currently say "No non-localhost DNS
nameservers are left in resolv.conf. Using default external
servers". But, that's misleading because it's from an initial
resolv.conf setup, before the internal resolver configured without
those fallbacks - we'll drop the fallbacks completely once the
default bridge has an internal resolver).
Signed-off-by: Rob Murray <rob.murray@docker.com>
Since commit 51d7f95 ("libnet: remove struct endpointCnt") an
endpoint count for networks has not been persisted.
But, on downgrade to a version older than that commit, the
missing field caused daemon startup to fail.
So, create the count in the store - it only needs to exist, it's
no longer maintained as a count of endpoints. On downgrade, the
count is probably zero anyway (the daemon is stopped), but the
older daemon fixes it up on startup if necessary.
Signed-off-by: Rob Murray <rob.murray@docker.com>
It's possible to remove a legacy link from running containers.
When that happens, the Sandbox's Endpoints are removed and
re-added.
Since commit 65120d5 ("Create bridge veth in container netns")
the veth device has been created in the container's netns. When
that happens, a flag is set on the Endpoint to note that it
does not need to be moved into the netns.
But, during the Leave/Join (Sandbox.Refresh) the veth is moved
out of the netns. So, clear the flag during the Leave, to note
that it needs to be moved back in during the Join.
Signed-off-by: Rob Murray <rob.murray@docker.com>
There have been numerous reports of the "has active endpoints" error
over the years. Historically, there were some faulty code paths that
could lead to this error, but we believe they all have been fixed by
now.
However, users are still facing this error from time to time. Either
because they forgot that some containers are still running, or because
we still have bugs lying around.
To help users figure whether this error is legitimate, and what triggers
it, add endpoint names (which are just container names) to the error
message.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Add an OTel span processor copying the 'trigger' baggage member
propagated through contexts to all children spans. It's used to identify
what triggered a trace / span (API call, libnet init, etc...)
All code paths that call libnet's `NewNetwork` set this baggage member
with a unique value.
For instance, this can be used to distinguish bridge's `createNetwork`
spans triggered by daemon / libnet initialization from custom network
creation triggerd by an API call.
Two util functions are added to wrap `baggage.New` and
`baggage.NewMemberRaw` to make it easier to deal with baggage and
members by panicking on error. These should not be used with dynamic
values.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Plumb context from the API down to libnet driver method `CreateNetwork`,
and add an OTel span to the bridge driver's `createNetwork` method.
Include a few attributes describing the network configuration (e.g.
IPv4/IPv6, ICC, internal and MTU).
A new util function, `RecordStatus`, is added to the `otelutil` package
to easily record any error, and update the span status accordingly.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Since commit `eaa84bc`, when a container joins a network, we broadcast
unsolicited ARP/NAs to other containers connected to that network. The
initial advertisement is sent synchronously, and then remaining
advertisements (by default, 2) are sent asynchronously at a regular
interval (by default, 1 second).
An OTel span records the time spent sending subsequent advertisements,
but it's attached to the parent span `libnetwork.osl.AddInterface` so it
affects the overall latency reported by the trace.
The real latency (i.e. excluding the latency of that asynchronous
process), is pretty much what CLI users perceive, so it's an important
measure. It's available on the top-most span in the trace, but limiting
the trace to the real time taken by the Engine to process the HTTP
request seems better.
With this change, the initial advertisement is now tracked by a
dedicated span, and a new independent trace (containing a single span)
is created and linked to the 'initial' span / HTTP trace.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Using iptables.OnReloaded to restore individual per-network rules
on firewalld reload means rules for deleted networks pop back in
to existence (because there was no way to delete the callbacks on
network-delete).
So, on firewalld reload, walk over current networks and ask them
to restore their iptables rules.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Report FirewallBackend in "docker info".
It's currently "iptables" or "iptables+firewalld" on Linux, and
omitted on Windows.
Signed-off-by: Rob Murray <rob.murray@docker.com>
