When the host has no valid external nameservers (or /etc/resolv.conf is
missing/dangling), TransformForIntNS previously left ExtNameServers empty
and appended 'NO EXTERNAL NAMESERVERS DEFINED'. This caused the embedded
DNS resolver (127.0.0.11) in user-defined networks to have no upstream
nameservers, causing external queries to fail with SERVFAIL.
As already documented in TransformForIntNS:
'If no ExtNameServer addresses are found, use the defaults.'
Fallback to host loopback addresses (127.0.0.1, ::1) with HostLoopback: true,
matching host libc resolver fallback behavior when resolv.conf is missing.
Signed-off-by: arsalanyavari <arya48.yavari79@gmail.com>
The driver creates one network sandbox for each overlay network, and
initSubnetSandbox and setupSubnetSandbox add a subnet's bridge and VXLAN
link to it. Their names, and the "subnet sandbox join failed" error,
read as though each subnet had a sandbox of its own. Rename them to
initSubnet and setupSubnet, rename the subnet's sboxInit flag to
initDone, and report the error as a failure to join the network sandbox
for the subnet.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The overlay driver kept the network namespaces of its network sandboxes
in the netns directory it was given, which the daemon also keeps the
namespaces of the controller's sandboxes in, and told them apart by the
"-" in their names. It cleaned up the network sandboxes an earlier
daemon left behind lazily. On the first join of any network's sandbox it
recorded the VNIs of their VXLAN links, so that it could destroy a stale
sandbox holding a VNI another network needed, and it destroyed a
network's stale sandboxes when the network was next initialized.
None of those sandboxes can be in use. A node cannot be in a Swarm with
live-restore enabled, so no container attached to an overlay network
outlives the daemon. Keep the network sandboxes in an "overlay"
subdirectory of the netns directory, and destroy everything in it when
the driver is first configured, along with the sandboxes earlier daemons
kept in the netns directory itself. Delete each stale sandbox's VXLAN
links before unmounting it: the kernel deletes them when it tears the
namespace down, but it does so asynchronously, and a new VXLAN link
cannot use their VNIs until it has.
A network's sandbox is destroyed before the network can be initialized
again, so the sandbox's key does not need the network's initialization
epoch to be unique. Name it "0-<network ID>". Earlier versions of the
driver walk the whole netns directory, the "overlay" subdirectory
included, and take a namespace with a "-" in its name to be a network
sandbox. They destroy it when a network needs one of its VNIs, or when
they initialize the network whose ID follows the "-", so they can still
clean up the sandboxes a newer daemon leaves behind. Their epochs start
at 1, so "0-" cannot clash with their names.
Signed-off-by: Cory Snider <csnider@mirantis.com>
A task on the ingress network needs rules in its own network namespace
that redirect each published port to its target port and filter all
other traffic addressed to its ingress endpoint. Adding them can fail,
for example when the kernel lacks a feature the rules use.
populateLoadBalancers logged that failure and let the join succeed. The
task then ran, and was reported as running, without serving its
published ports and with every port it listens on reachable from the
rest of the ingress network. The only trace was the daemon log.
Return the error instead, so the join fails and the task fails with the
reason. This applies to both firewall backends.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Regenerate the protobuf bindings using Go 1.27.
The descriptor contents are unchanged. The generated diffs are limited to
the gzip-compressed raw descriptors embedded in the generated Go files.
Go 1.27 changed the `compress/flate` encoder to improve compression
performance. In this case, the new encoder marks the preceding DEFLATE block
as final instead of appending an empty final block, making each compressed
descriptor three or four bytes smaller.
Go 1.27 explicitly allows the exact encoded output to differ as a result of
this encoder change:
https://go.dev/doc/go1.27#compressflatepkgcompressflate
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The getController method only returns the network's controller pointer,
without performing any additional operations.
Replace its callers with direct access to n.ctrlr and remove the
unnecessary accessor. Rename the ctrlr field to "controller" for
clarity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The controller pointer is initialized when a network is created and
is not modified afterward. Reading it therefore does not require
holding the network mutex.
Remove the unnecessary locking from getController.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a test-surface specification for the overlay network driver and the
Swarm service mesh: what each part is expected to do, as observed from
outside the daemon. Each expectation carries a tier for what a failing
test means, the vantage point it is observed from, and what its verdict
depends on: the operating system, a package's upgrade path, the docker
CLI, or none of them. The spec is a model of the daemon's behaviour.
Which expectations a test suite checks, and where, is the suite's
choice.
An expectation holds on a node in its baseline state unless it says
otherwise. For a daemon installed from a package, the baseline is a
clean install of a supported OS, a fresh install of the package, and
the setup that the product's installation documentation prescribes.
For any other daemon, it is a host that meets the host requirements
that each spec lists: the kernel facilities, programs and host
configuration that its expectations rely on.
It describes behaviour only, so it should change when the daemon's
behaviour does, or when a regression shows that an expectation was only
assumed.
Expectations that came up along the way but are not Swarm's go in stubs
beside it, so the Swarm spec can refer to them without owning them.
bridge-networking.md covers the bridge driver and the parts of the
daemon's firewall integration that are not specific to Swarm,
DOCKER-USER among them. ipvlan-networking.md covers the ipvlan driver.
container-dns.md covers name resolution that does not depend on the
network driver. network-common.md covers what every network driver must
do, and the Engine API requests for networks and endpoints. The
conventions all five follow are in a README.md of their own, so that a
new spec can use them without editing an existing one. The specs live
in daemon/libnetwork/docs/spec.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When the daemon started with the nftables backend after a release older
than 28.0.0 had run with iptables, NewCleaner deleted the jumps from
filter-FORWARD to DOCKER-FORWARD and DOCKER-ISOLATION-STAGE-1, then
removed Docker's chains. Those releases also jumped from FORWARD to the
filter table's DOCKER chain for each bridge ("-o docker0 -j DOCKER").
iptables refuses to delete a chain that a rule still jumps to, and
removeIPChains ignores the error. The cleaner's DelNetwork deleted the
per-bridge jump later, when the bridge driver restored the network, but
nothing tried to remove the chain again. The empty DOCKER chain
remained.
When the bridge driver has restored its networks and endpoints, call
the cleaner's new Done method, which removes the chains again.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When the daemon started with the nftables backend after a release older
than 28.0.1 had run with iptables, the iptables cleaner's DelNetwork
left the per-bridge rules those releases put in filter-FORWARD:
"-i docker0 ! -o docker0 -j ACCEPT" and the ICC rules. The iptables
backend deleted them in setupNonInternalNetworkRules and setIcc, but
only when adding a network's rules, and DelNetwork returned early for an
internal network.
Delete all the legacy per-bridge FORWARD rules in
deleteLegacyFilterRules, which the iptables backend and the cleaner call
for every network.
With the iptables backend, setupIPTables deletes the legacy rules after
calling setupNonInternalNetworkRules, which used to delete
"-i docker0 ! -o docker0 -j ACCEPT" before adding the DOCKER-FORWARD
rule that replaces it. Now, for a moment, both rules are present. Both
accept outgoing packets from the bridge.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Add a native nftables implementation of service load balancing which
achieves feature parity with the iptables firewall mode. This is
accomplished using a static nftables ruleset with dynamic data. NAT and
DSR modes are both supported. The backend server for each flow is chosen
randomly rather than round-robin like the iptables+IPVS implementation.
As the load balancer endpoint only sees one side of DSR flows we cannot
use netfilter's conntrack subsystem. And IPVS does not play well with
nftables -- nftables is missing an analogue for the `ipvs` xtables
matcher. DSR flows are made sticky by implementing one-sided connection
tracking directly in the nftables ruleset using maps that are updated in
the packet path.
A network's load-balancer tables have exactly one writer: each
loadBalancer records the data plane it wants, and the complete element set
is composed from every recorded contribution. nat-publish-port is an
interval map keyed on `l4proto . dport . numgen`, so the bucket space for
a published port has to be partitioned by a single owner -- and a service
whose published-port set changes has a separate binding, hence a separate
loadBalancer, per generation until the rolling update drains. Two of them
partitioning a shared port independently would emit overlapping intervals,
which nftables rejects. Composing them instead partitions the port once
over every backend that serves it, so a shared published port is
load-balanced across both generations. IPVS cannot do that: one fwMark
cannot select a different backend set per port.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The ingress ports need to be published through the bridge driver
when the virtual service is initialized, irrespective of the firewall
backend in use. Change the firewall-specific implementation to return a
flag signaling that the service was initialized and arrange to have the
caller publish the ports.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Factor the iptables-specific service mesh configuration out of the
common Linux service mesh implementation in preparation for adding an
nftables implementation.
Signed-off-by: Cory Snider <csnider@mirantis.com>
With the iptables firewall backend, the overlay driver programs iptables
rules which mark VXLAN datagrams of encrypted networks for encryption,
drop cleartext VXLAN datagrams received for those networks, and strip
the encryption mark from datagrams marked by local processes. A
firewalld reload can flush these rules. The node then sends the
network's VXLAN datagrams in cleartext and accepts forged cleartext
datagrams for it until the last local endpoint of the network leaves and
another joins, or the daemon restarts.
On firewalld reload, restore all rules which were previously programmed
by the overlay driver.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Swarm ingress ports are published as ordinary port mappings on the
ingress load balancer's docker_gwbridge endpoint, so they follow the
userland-proxy setting like any other port mapping. With the proxy
enabled, the DNAT rule for a port mapping is not applied to packets
arriving on the bridge the mapping targets; those packets pass through
the INPUT chain to docker-proxy instead. Packets from a task to a
published port at its own node's address arrive on docker_gwbridge and
take that path, so a host firewall that drops unsolicited input, such as
ufw with its default "deny incoming" policy, drops them, even though the
same port remains reachable from other nodes and from the host itself.
Before ingress ports were published as port mappings, their DNAT rule in
the DOCKER-INGRESS chain applied to packets from docker_gwbridge too,
and no docker-proxy was started for them. A task could reach a published
port on its own node whatever the userland-proxy setting, without ufw's
policy getting in the way.
A connection from a container to a published port on one of the host's
addresses, where the port maps to a container on the same bridge, is a
hairpin connection: its packets enter and leave the host through that
bridge. docker-proxy makes hairpin connections work by accepting the
client's connection in the host and opening a new one to the container.
The kernel can handle them without the proxy by DNATing the packets back
across the bridge, but that needs br_netfilter: the server's replies are
bridged straight back to the client, and conntrack sees them and
restores their source address only if br_netfilter passes bridged
packets through netfilter. A container that connects to its own
published port also needs hairpin mode on its bridge port and a
MASQUERADE rule.
The bridge driver also uses br_netfilter to block inter-container
communication (ICC). Loading br_netfilter has a cost: it passes traffic
on every bridge on the host through the host's firewall rules, including
bridges that Docker does not manage. And some hosts cannot load
br_netfilter at all. The bridge driver therefore loads br_netfilter only
when necessary.
Add a NoSelfConnect hint to PortBinding and PublishedPort. It tells the
bridge driver that nothing behind the endpoint connects to the published
port. Packets from the endpoint's own bridge can then be DNATed to the
port without hairpin mode or the MASQUERADE rule, needing only
br_netfilter. The bridge driver uses the hint to bypass docker-proxy
where it can: for a binding with NoSelfConnect set, on a network with
ICC disabled, for which the bridge driver has already loaded
br_netfilter, the iptables and nftables firewallers DNAT packets from
the network's own bridge as well. The existing filter rule that accepts
DNATed packets arriving on an ICC-disabled bridge already lets them
through.
libnetwork sets the hint on ingress ports. They are published on the
load balancer's endpoint, whose sandbox libnetwork creates without a
container, so nothing behind the endpoint can connect to them.
docker_gwbridge also has ICC disabled. Other endpoints on
docker_gwbridge can have a container behind them, and only an endpoint's
owner knows whether one does, so the hint comes from the owner and is
carried on each binding.
For ingress ports, the result is a hybrid of the two userland-proxy
modes. docker-proxy still listens on the ports, but netfilter now DNATs
IPv4 packets arriving on docker_gwbridge before the proxy sees them, as
it already did for packets from other nodes. The proxy is left handling
IPv6 clients, which it forwards to the load balancer over IPv4, and
connections from the host to its loopback addresses.
Hairpin connections to ordinary containers' published ports still go
through docker-proxy. With the userland proxy disabled, the bridge
driver handles them in the kernel: it loads br_netfilter for every
network, and since it cannot know whether a container connects to its
own published port, it enables hairpin mode on every container's bridge
port and adds the MASQUERADE rule. Hairpin mode also sends broadcast and
multicast frames back to the container that sent them, and nothing
filters them out. Doing the same with the proxy enabled would bring that
behaviour, and the cost of br_netfilter, to every container. Closing
that gap is part of making operation without the userland proxy match
operation with it, apart from servers seeing clients' real addresses,
and is a larger change than restoring the ingress ports' earlier
behaviour.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-By: Claude Opus 5.5
The overlay driver subtracted a 50-byte VXLAN allowance from the
network's MTU, which assumes an IPv4 outer header. With an IPv6 data
path the outer header is 40 bytes, so the overhead is 70 bytes and the
overlay MTU was 20 bytes larger than the path can carry. Choose the
allowance from the address family of the data-path address.
If the family is not known, subtract the IPv6 allowance, as the
resulting MTU fits either family.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Provide a temporary origin resolv.conf for both sandboxes so host options such as edns0 and trust-ad cannot affect the DNS option assertions
Signed-off-by: akasakariko <148060537+akasakariko@users.noreply.github.com>
Allocate all requested port bindings before applying their firewall rules, so nftables rewrites each affected chain once per address family rather than once per port group
Keep grouped port allocation, start proxies after the firewall update, and roll back resources and installed rules if either stage fails
Extend the port-mapping tests to check a single firewall update and verify cleanup and rebinding after setup failures
Signed-off-by: akasakariko <148060537+akasakariko@users.noreply.github.com>
With the userland proxy disabled, the per-port "MASQ TO OWN PORT" rule
is keyed on the container's address and port, not the host port. So,
when more than one host port is mapped to the same container port, the
same rule is generated for each binding.
Commit 785ae9a0f9 dropped the code that ignored "rule exists" and "rule
does not exist" errors for this rule, without setting IgnoreExist. So,
mapping a second host port to the same container port failed with
"rule exists".
Set IgnoreExist on the rule, as for the per-port forwarding rule.
Signed-off-by: Rob Murray <rob.murray@docker.com>
The daemon uses firewalld only if firewalld is running when the daemon
starts. If firewalld is started later, the daemon does not react to it
starting or reloading. On firewalld configurations which delete other
applications' iptables rules when firewalld starts, the daemon's rules
stay deleted. The daemon's bridges are not added to the docker zone.
Keep the D-Bus connection when firewalld is not running at startup.
When firewalld acquires its D-Bus name, set up the docker zone and
forwarding policy before running the reload callbacks.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The daemon checks whether firewalld is running by calling
getDefaultZone at startup, and again whenever firewalld's D-Bus name
changes owner. The check infers firewalld's state from whether a
method call succeeds.
firewalld does not handle method calls until it has finished starting.
It is ready as soon as it owns its name. Ask the bus whether the name
has an owner at startup, after subscribing to NameOwnerChanged so that
no change of owner is missed, and take the state from each
NameOwnerChanged signal after that.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When firewalld is stopped with CleanupOnExit=yes (the default), it
deletes the iptables rules, including the daemon's rules for
masquerading, published ports and network isolation. The daemon does
not restore them until firewalld is started again or the daemon is
restarted.
firewalld deletes the rules before it releases its D-Bus name. Run the
reload callbacks when firewalld releases its name, as well as when it
acquires it. By then, UsingFirewalld() reports false and the callbacks
restore the rules using iptables directly.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Swarm ingress ports are published on the ingress sandbox's
docker_gwbridge endpoint. If docker_gwbridge has IPv6 enabled, that
endpoint got an IPv6 address, and host IPv6 traffic was DNATed to it.
The sandbox only forwards IPv4 (its fwmark/IPVS rules are IPv4-only),
so the connection was refused.
Create the ingress sandbox's docker_gwbridge endpoint without IPv6.
With no IPv6 gateway, the IPv4 gateway endpoint also serves IPv6, and
the userland proxy forwards host IPv6 traffic to its IPv4 address, as
it does when docker_gwbridge has no IPv6.
Signed-off-by: Felix Schnellert <felix.schnellert@skriptfabrik.com>
AddInterfaceFirewalld and DelInterfaceFirewalld asked firewalld for the
interfaces in the "docker" zone before adding or removing one. firewalld
already reports what those checks looked for: addInterface fails with
ZONE_ALREADY_SET for an interface already in the zone, and
removeInterface fails with UNKNOWN_INTERFACE or ZONE_CONFLICT for an
interface that isn't in it. AddInterfaceFirewalld already handled
ZONE_ALREADY_SET, so its check only added a D-Bus call.
Drop the checks, and return the same not-found error from
DelInterfaceFirewalld when removeInterface reports that the interface
isn't in the zone.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-by: Claude Opus 5.5
With iptables and ip6tables disabled, the daemon doesn't manage firewall
rules for its bridge networks, but it still added each bridge to
firewalld's "docker" zone, whose target is ACCEPT. That replaced the
zone firewalld would otherwise apply to the bridge, so a user filtering
container traffic with firewalld had to bind each bridge to their own
zone. The firewalld reload handler already skips bridge networks in this
configuration.
Add the bridge to the "docker" zone only when iptables or ip6tables is
enabled. A zone applies to the whole interface, so with either one
enabled the bridge is in the zone for both address families.
Still remove the bridge from the zone when deleting a network, in case
it was added while iptables was enabled, but don't warn when it isn't
there.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-by: Claude Opus 5.5
When a user binds a bridge network's interface to a firewalld zone
other than "docker", firewalld rejects the daemon's attempt to add it
to the "docker" zone with ZONE_CONFLICT. AddInterfaceFirewalld returned
that error, failing the network's setup. On daemon startup the bridge
driver logs the failure and leaves the network out of its state, so
every attempt to connect a container to it fails with "network ... does
not exist" even though "docker network inspect" still shows it.
With iptables enabled, the firewalld integration is initialized before
networks are restored, so such a network already failed to restore.
With iptables disabled, nothing initialized it that early, and
AddInterfaceFirewalld was a no-op until it was initialized. Commit
4888f3a41f ("libnetwork/iptables: Use UsingFirewalld in remaining
firewalld paths") made AddInterfaceFirewalld initialize it on first use,
so the network now fails to restore in that configuration too.
Treat ZONE_CONFLICT as an intentional choice by the user: leave the
interface in its zone and set up the network anyway.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-by: Claude Opus 5.5
firewalld reports an error over D-Bus as a FirewallD1.Exception whose
message is the error's code, optionally followed by ": " and a detail
message. setupDockerForwardingPolicy and AddInterfaceFirewalld each
recognised a code by checking the D-Bus error name and then matching a
prefix of the message, with or without the colon.
Add firewalldErrorCode, which extracts the whole code from a firewalld
error, and use it for both checks.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-by: Claude Opus 5.5
OptionExecRoot did not configure the controller it was passed to: it set
the directory for all network namespaces in package osl, which every
controller in the process shared. The overlay driver found that
directory by generating a sandbox key for a made-up ID and taking the
key's parent directory.
Replace it with OptionNetnsDir, available on Linux and FreeBSD, which
sets the directory in the controller's config. OptionExecRoot was a
no-op on Windows. The controller generates its sandboxes' keys from the
directory, and the overlay driver is given the directory when it is
registered and builds its own keys. Each overlay driver instance scans
its own directory for the VNIs of the sandboxes a previous run of the
daemon left behind. The controller creates the directory when it is
created, instead of osl creating the package-wide directory when it
first creates a namespace.
The daemon still keeps the namespaces in <exec-root>/netns. The
directory has no default: on Linux and FreeBSD, the controller refuses
to create a sandbox when none is configured, as does the overlay driver,
so a test which leaves out the option fails instead of leaving network
namespaces behind in a directory shared with the host. Tests which
create sandboxes keep them in a temporary directory, and unmount the
namespaces left in it when they end.
The key formats do not change on Linux. On FreeBSD, sandbox keys become
paths in the directory, as on Linux. Drop GenerateKey's lookup of an
existing key for an ID starting with "-": no caller passes such an ID.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-By: Claude Opus 5.5
Check the value length before calculating the buffer capacity to
prevent integer overflow when adding the metadata length.
Fixes potential allocation failures reported by CodeQL.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Nothing runs the "libnetwork-setkey" hook any more. Remove the listener
and the hook's reexec entrypoint, along with the controller ID, which
only named the listener's socket, and Config.ExecRoot, which only
located it. OptionExecRoot still sets the osl base path on Linux.
Stopping the listener was all Controller.Stop had left to do, so remove
it too.
Remove the reexec variant of the external key test too. It could not
run, as the package's tests never call reexec.Init.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-By: Claude Opus 5.5
This test was added in 38eca8b1e3, which
was merged while 7f065ad76f removed the
config.OptionDataDir option, causing a compile failure;
```
=== Failed
=== FAIL: daemon/libnetwork (0.00s)
FAIL github.com/moby/moby/v2/daemon/libnetwork [build failed]
=== Errors
daemon/libnetwork/service_linux_test.go:178:38: undefined: config.OptionDataDir
```
updates 38eca8b1e3
updates 7f065ad76f
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Nothing calls Type() on a driverapi.Driver. Network.Type() and
Network.Driver() report the name the driver was registered under,
not the driver's Type(). The only caller of a driver's Type() method
was the Windows overlay driver calling its own, which returns the
package's NetworkType constant.
Remove the method from the interface and its implementations, and
the tests that only asserted its return value. The remote driver no
longer needs to remember its plugin name.
Assisted-By: Claude Opus 5.5
Signed-off-by: Cory Snider <csnider@mirantis.com>
pkg/plugins caches every plugin it loads, keyed by name, for the
life of the process. TestValidRemoteDriver and TestInvalidRemoteDriver
used fixed plugin names, so a repeat run (go test -count=2) got the
plugin cached by the first run, pointing at that run's closed
httptest server. TestValidRemoteDriver then stalled while the plugin
client retried the dead address; TestInvalidRemoteDriver passed
without ever talking to its own server.
Add a random suffix to the plugin name in each run.
Assisted-By: Claude Opus 5.5
Signed-off-by: Cory Snider <csnider@mirantis.com>
TestIpamReleaseOnNetDriverFailures registered a package-level
badDriver and cleared its failNetworkCreation flag partway through.
Nothing set the flag again, so on a repeat run (go test -count=2)
creating the first network on the bad driver succeeded, and creating
the bridge network with the same pool then failed with an overlap.
Give each run its own badDriver.
Assisted-By: Claude Opus 5.5
Signed-off-by: Cory Snider <csnider@mirantis.com>
Paweł Gronowski