In Swarm overlay networks, remote nodes program VXLAN FDB and neighbor
entries from the overlay driver's NetworkDB table (overlay_peer_table).
Those entries are installed as static/permanent neighbors and are
removed only when the endpoint's driver table entries are deleted from
NetworkDB and the delete event is gossiped to peers.
Endpoint.sbLeave() updates the local endpoint object in the store via
storeEndpoint(). If that store update fails (notably
datastore.ErrKeyModified from a boltDB CAS conflict during concurrent
endpoint operations), sbLeave() returned early without calling
deleteDriverInfoFromCluster(). This left the endpoint's
overlay_peer_table record behind in NetworkDB, so no DELETE gossip event
was emitted, and remote nodes never removed the corresponding
neighbor/FDB state. This causes stale permanent ARP/FDB entries to
accumulate in overlay network namespaces, eventually blackholing TCP
traffic and producing intermittent 502/504 until the service is
redeployed.
Fix this by removing the early return on storeEndpoint failure.
Instead, accumulate errors and continue the full leave operation.
Assisted-by: GPT-5.2 (AI)
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
During a network disconnect, log rather than returning an error
if it's not possible to set up a new gateway.
This restores the behaviour from before commit 53390f8 ("Put
clearNetworkResources() inline in its only caller"). It's not
ideal, but by the time new gateways are selected the old
endpoint has been disconnected - and nothing puts things back.
Until that's cleaned up, a broken state is inevitable, but
letting endpoint deletion complete means the container can
be restarted or re-connected to the network without a zombie
endpoint causing further issues.
Signed-off-by: Rob Murray <rob.murray@docker.com>
When the endpoint providing a container's default gateway
is removed, there's no need to select a new gateway if the
container is being removed.
Signed-off-by: Rob Murray <rob.murray@docker.com>
If a container is using a docker_gwbridge endpoint as its gateway,
when it's connected to another network that provides a gateway, the
docker_gwbridge endpoint is removed when that endpoint is added (in
a recursive nightmare).
So, the "before" gateway for the container has been removed
before the new gateway is updateExternalConnectivity'd.
Don't pass the old gateway to updateExternalConnectivity in that
case, because the network driver's already forgotten about it.
Signed-off-by: Rob Murray <rob.murray@docker.com>
These drivers did not do anything meaningful in the `Leave` method; they
would check if the network and/or endpoint were missing, in which case
they produced an error, but the network and endpoint (if present) would
not be used, so it was only validation.
Such validation could still be relevant elsewhere, but looking at where
this method is called; the `Driver.Leave()` is called in two places, both
of which don't handle the error, other than logging it as a warning / error;
It's called by `Endpoint.sbJoin()`, as part of the rollback;
d5c838dc5e/daemon/libnetwork/endpoint.go (L539-L545)
And `Endpoint.sbLeave()`, which also discards the error;
d5c838dc5e/daemon/libnetwork/endpoint.go (L772-L776)
Based on he above, this code looks to be redundant, so replacing it with
a stub; returning `nil`.
As replacing the code removed the use of network.getEndpoint, which was effectively
a copy of network.endpoint (which didn't have error handling), I merged the two
methods, and removed custom error-handling elsewhere.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These utilities are very handy to use in integration tests, too. Move
the package so it can be imported by them.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When running:
docker network create --ipv6 b46
docker run --rm -ti \
--network name=b46,driver-opt=com.docker.network.endpoint.sysctls=net.ipv6.conf.IFNAME.disable_ipv6=1 \
busybox
IPv6 is enabled in the container and the network, so an IPv6 address
will be allocated for the endpoint.
But, when the sysctl is applied, the IPv6 address will be removed
from the interface ... so, no unsolicited neighbour advertisement
should be (or can be) sent and, the endpoint should not be treated
as dual-stack when selecting a gateway endpoint and, if it is
selected as the gateway endpoint, setting up an IPv6 route via the
network will fail.
So, if the IPv6 address disappears after sysctls have been applied,
release the address and remove it from the endpoint's config.
Signed-off-by: Rob Murray <rob.murray@docker.com>
And move the Endpoint.populateNetworkResources code into the
all-platforms part of the Sandbox method.
Signed-off-by: Rob Murray <rob.murray@docker.com>
When the SetKey hook is used (by a build container) it's called after
Endpoint.sbJoin, which will have called Sandbox.populateNetworkResources
to set up address, routes, sysctls and so on - but it's not able to do
any config until the osSbox exists. So, Sandbox.populateNetworkResources
is called again by SetKey to finish that config.
But, that means the rest of Endpoint.sbJoin has already happened before
the osSbox existed - it will have configured DNS, /etc/hosts, gateways
and so on before anything was set up for the OS.
So, if the osSbox configuration isn't applied as expected (for example,
a sysctl disables IPv6 on the endpoint), that sbJoin configuration is
incorrect.
To avoid unnecessary config+cleanup in thoses cases - delay the config
currently done by sbJoin until the osSbox exists.
Signed-off-by: Rob Murray <rob.murray@docker.com>
The old deferred error handling cleared ep.sandboxID, but only
in a copy of the Endpoint loaded from the store, not stored or
returned - so the modification was immediately lost.
It also tried to remove the endpoint from the Sandbox's 'endpoints',
but the remove function compared pointers rather than ids, so
nothing was removed. Removing it would have broken rollback anyway.
Signed-off-by: Rob Murray <rob.murray@docker.com>
If an endpoint is still attached to a Sandbox when
Endpoint.Delete is called with force=true, sbLeave is
called. It may change the Sandbox's gateway, which may
conflict with a concurrent Join.
So, acquire the Sandbox's joinLeaveMu to do that, and
clarify the purpose of that mutex in struct Sandbox
comments.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Because it loaded the Endpoint object from store and operated
on that copy rather than its own receiver, sbJoin couldn't
successfully roll back on error if the Endpoint was not included
in the Sandbox's list of endpoints, or its current state had
not been written to store after the error occurred.
So, for example, releaseOSSboxResources() would not be called
to delete interfaces created in the container's netns.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Modernize using maps.Clone, slices.Clone. This method is needed to
satisfy the datastore.KVObject interface, so also assert it does.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The endpointJoinInfo.CopyTo function expected the caller to construct an
EndpointInterface to copy to, but all callsites created an empty struct.
In addition, `CopyTo` would never return an error, so the error return
was redundant.
Replace it with a `Copy()` function, which makes it easier to
consume.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The EndpointInterface.CopyTo function expected the caller to construct an
EndpointInterface to copy to, but all callsites created an empty struct.
In addition, `CopyTo` would never return an error, so the error return
was redundant.
Replace it with a `Copy()` function, which makes it easier to
consume.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The stringid package is used in many places; while it's trivial
to implement a similar utility, let's just provide it as a utility
package in the client, removing the daemon-specific logic.
For integration tests, I opted to use the implementation in the
client, as those should not ideally not make assumptions about
the daemon implementation.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
