Swarm ingress ports are published as ordinary port mappings on the
ingress load balancer's docker_gwbridge endpoint, so they follow the
userland-proxy setting like any other port mapping. With the proxy
enabled, the DNAT rule for a port mapping is not applied to packets
arriving on the bridge the mapping targets; those packets pass through
the INPUT chain to docker-proxy instead. Packets from a task to a
published port at its own node's address arrive on docker_gwbridge and
take that path, so a host firewall that drops unsolicited input, such as
ufw with its default "deny incoming" policy, drops them, even though the
same port remains reachable from other nodes and from the host itself.
Before ingress ports were published as port mappings, their DNAT rule in
the DOCKER-INGRESS chain applied to packets from docker_gwbridge too,
and no docker-proxy was started for them. A task could reach a published
port on its own node whatever the userland-proxy setting, without ufw's
policy getting in the way.
A connection from a container to a published port on one of the host's
addresses, where the port maps to a container on the same bridge, is a
hairpin connection: its packets enter and leave the host through that
bridge. docker-proxy makes hairpin connections work by accepting the
client's connection in the host and opening a new one to the container.
The kernel can handle them without the proxy by DNATing the packets back
across the bridge, but that needs br_netfilter: the server's replies are
bridged straight back to the client, and conntrack sees them and
restores their source address only if br_netfilter passes bridged
packets through netfilter. A container that connects to its own
published port also needs hairpin mode on its bridge port and a
MASQUERADE rule.
The bridge driver also uses br_netfilter to block inter-container
communication (ICC). Loading br_netfilter has a cost: it passes traffic
on every bridge on the host through the host's firewall rules, including
bridges that Docker does not manage. And some hosts cannot load
br_netfilter at all. The bridge driver therefore loads br_netfilter only
when necessary.
Add a NoSelfConnect hint to PortBinding and PublishedPort. It tells the
bridge driver that nothing behind the endpoint connects to the published
port. Packets from the endpoint's own bridge can then be DNATed to the
port without hairpin mode or the MASQUERADE rule, needing only
br_netfilter. The bridge driver uses the hint to bypass docker-proxy
where it can: for a binding with NoSelfConnect set, on a network with
ICC disabled, for which the bridge driver has already loaded
br_netfilter, the iptables and nftables firewallers DNAT packets from
the network's own bridge as well. The existing filter rule that accepts
DNATed packets arriving on an ICC-disabled bridge already lets them
through.
libnetwork sets the hint on ingress ports. They are published on the
load balancer's endpoint, whose sandbox libnetwork creates without a
container, so nothing behind the endpoint can connect to them.
docker_gwbridge also has ICC disabled. Other endpoints on
docker_gwbridge can have a container behind them, and only an endpoint's
owner knows whether one does, so the hint comes from the owner and is
carried on each binding.
For ingress ports, the result is a hybrid of the two userland-proxy
modes. docker-proxy still listens on the ports, but netfilter now DNATs
IPv4 packets arriving on docker_gwbridge before the proxy sees them, as
it already did for packets from other nodes. The proxy is left handling
IPv6 clients, which it forwards to the load balancer over IPv4, and
connections from the host to its loopback addresses.
Hairpin connections to ordinary containers' published ports still go
through docker-proxy. With the userland proxy disabled, the bridge
driver handles them in the kernel: it loads br_netfilter for every
network, and since it cannot know whether a container connects to its
own published port, it enables hairpin mode on every container's bridge
port and adds the MASQUERADE rule. Hairpin mode also sends broadcast and
multicast frames back to the container that sent them, and nothing
filters them out. Doing the same with the proxy enabled would bring that
behaviour, and the cost of br_netfilter, to every container. Closing
that gap is part of making operation without the userland proxy match
operation with it, apart from servers seeing clients' real addresses,
and is a larger change than restoring the ingress ports' earlier
behaviour.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-By: Claude Opus 5.5
Cory Snider
d95c4fbe4e