libnetwork: DNAT gwbridge packets to ingress ports

Swarm ingress ports are published as ordinary port mappings on the
ingress load balancer's docker_gwbridge endpoint, so they follow the
userland-proxy setting like any other port mapping. With the proxy
enabled, the DNAT rule for a port mapping is not applied to packets
arriving on the bridge the mapping targets; those packets pass through
the INPUT chain to docker-proxy instead. Packets from a task to a
published port at its own node's address arrive on docker_gwbridge and
take that path, so a host firewall that drops unsolicited input, such as
ufw with its default "deny incoming" policy, drops them, even though the
same port remains reachable from other nodes and from the host itself.

Before ingress ports were published as port mappings, their DNAT rule in
the DOCKER-INGRESS chain applied to packets from docker_gwbridge too,
and no docker-proxy was started for them. A task could reach a published
port on its own node whatever the userland-proxy setting, without ufw's
policy getting in the way.

A connection from a container to a published port on one of the host's
addresses, where the port maps to a container on the same bridge, is a
hairpin connection: its packets enter and leave the host through that
bridge. docker-proxy makes hairpin connections work by accepting the
client's connection in the host and opening a new one to the container.
The kernel can handle them without the proxy by DNATing the packets back
across the bridge, but that needs br_netfilter: the server's replies are
bridged straight back to the client, and conntrack sees them and
restores their source address only if br_netfilter passes bridged
packets through netfilter. A container that connects to its own
published port also needs hairpin mode on its bridge port and a
MASQUERADE rule.

The bridge driver also uses br_netfilter to block inter-container
communication (ICC). Loading br_netfilter has a cost: it passes traffic
on every bridge on the host through the host's firewall rules, including
bridges that Docker does not manage. And some hosts cannot load
br_netfilter at all. The bridge driver therefore loads br_netfilter only
when necessary.

Add a NoSelfConnect hint to PortBinding and PublishedPort. It tells the
bridge driver that nothing behind the endpoint connects to the published
port. Packets from the endpoint's own bridge can then be DNATed to the
port without hairpin mode or the MASQUERADE rule, needing only
br_netfilter. The bridge driver uses the hint to bypass docker-proxy
where it can: for a binding with NoSelfConnect set, on a network with
ICC disabled, for which the bridge driver has already loaded
br_netfilter, the iptables and nftables firewallers DNAT packets from
the network's own bridge as well. The existing filter rule that accepts
DNATed packets arriving on an ICC-disabled bridge already lets them
through.

libnetwork sets the hint on ingress ports. They are published on the
load balancer's endpoint, whose sandbox libnetwork creates without a
container, so nothing behind the endpoint can connect to them.
docker_gwbridge also has ICC disabled. Other endpoints on
docker_gwbridge can have a container behind them, and only an endpoint's
owner knows whether one does, so the hint comes from the owner and is
carried on each binding.

For ingress ports, the result is a hybrid of the two userland-proxy
modes. docker-proxy still listens on the ports, but netfilter now DNATs
IPv4 packets arriving on docker_gwbridge before the proxy sees them, as
it already did for packets from other nodes. The proxy is left handling
IPv6 clients, which it forwards to the load balancer over IPv4, and
connections from the host to its loopback addresses.

Hairpin connections to ordinary containers' published ports still go
through docker-proxy. With the userland proxy disabled, the bridge
driver handles them in the kernel: it loads br_netfilter for every
network, and since it cannot know whether a container connects to its
own published port, it enables hairpin mode on every container's bridge
port and adds the MASQUERADE rule. Hairpin mode also sends broadcast and
multicast frames back to the container that sent them, and nothing
filters them out. Doing the same with the proxy enabled would bring that
behaviour, and the cost of br_netfilter, to every container. Closing
that gap is part of making operation without the userland proxy match
operation with it, apart from servers seeing clients' real addresses,
and is a larger change than restoring the ingress ports' earlier
behaviour.

Signed-off-by: Cory Snider <csnider@mirantis.com>
Assisted-By: Claude Opus 5.5
This commit is contained in:
Cory Snider committed 2026-10-29 14:52:48 -04:00
1 parent ed01f3a432
commit d95c4fbe4e
13 files changed
+230 -35

No files matched your search

@@ -71,6 +71,22 @@ type NetworkConfigFam struct {
Unprotected bool
}
// DNATFromOwnBridge returns true if packets that arrive from the network's own
// bridge should be DNATed to pb's container address. If it returns false,
// they're left for the userland proxy.
//
// Without the proxy (config.Hairpin), they're always DNATed. With the proxy,
// DNATing them needs two things the proxy avoids. First, br_netfilter: the
// container's replies are bridged straight back to the client, so conntrack
// only sees them, and restores their source address, if bridged packets go
// through netfilter. The network has that when ICC is disabled. Second, for a
// container that connects to its own published port, hairpin mode on its
// bridge port and a MASQUERADE rule. A binding marked NoSelfConnect doesn't
// need those.
func (nc NetworkConfig) DNATFromOwnBridge(config Config, pb types.PortBinding) bool {
return config.Hairpin || (!nc.ICC && pb.NoSelfConnect)
}
// Firewaller implements firewall rules for bridge networks.
type Firewaller interface {
// NewNetwork returns an object that can be used to add published ports and legacy
@@ -15,6 +15,7 @@ import (
"github.com/moby/moby/v2/daemon/libnetwork/types"
"github.com/moby/moby/v2/internal/testutil/netnsutils"
"gotest.tools/v3/assert"
is "gotest.tools/v3/assert/cmp"
"gotest.tools/v3/golden"
"gotest.tools/v3/icmd"
"gotest.tools/v3/skip"
@@ -265,3 +266,63 @@ func testIptabler(t *testing.T, tn string, config firewaller.Config, netConfig f
checkResults("iptables", rnWSL2Mirrored(fmt.Sprintf("%s/cleaned,hairpin=%v", tn, config.Hairpin)), config.IPv4)
checkResults("ip6tables", fmt.Sprintf("%s/cleaned,hairpin=%v", tn, config.Hairpin), config.IPv6)
}
// TestDNATFromOwnBridge checks when a port binding's DNAT rule leaves packets
// arriving from the network's own bridge to the userland proxy, and that the
// rule is deleted with the binding.
func TestDNATFromOwnBridge(t *testing.T) {
skip.If(t, iptables.UsingFirewalld(), "firewalld is running in the host netns, it can't modify rules in the test's netns")
testcases := []struct {
hairpin, icc, noSelfConnect bool
expProxySkip bool
}{
{hairpin: true, icc: true},
{hairpin: true, icc: false, noSelfConnect: true},
{icc: true, expProxySkip: true},
{icc: true, noSelfConnect: true, expProxySkip: true},
{icc: false, expProxySkip: true},
{icc: false, noSelfConnect: true},
}
for _, tc := range testcases {
t.Run(fmt.Sprintf("hairpin=%v/icc=%v/noSelfConnect=%v", tc.hairpin, tc.icc, tc.noSelfConnect), func(t *testing.T) {
defer netnsutils.SetupTestOSContext(t, netnsutils.WithSetNsHandles(false))()
fw, err := NewIptabler(t.Context(), firewaller.Config{IPv4: true, Hairpin: tc.hairpin})
assert.NilError(t, err)
nw, err := fw.NewNetwork(t.Context(), firewaller.NetworkConfig{
IfName: "br-dummy",
ICC: tc.icc,
Masquerade: true,
Config4: firewaller.NetworkConfigFam{Prefix: netip.MustParsePrefix("192.168.0.0/24")},
})
assert.NilError(t, err)
pb := types.PortBinding{
Proto: types.TCP,
IP: net.ParseIP("192.168.0.2"),
Port: 80,
HostIP: net.IPv4zero,
HostPort: 8080,
HostPortEnd: 8080,
NoSelfConnect: tc.noSelfConnect,
}
dnatRules := func() []string {
t.Helper()
out, err := iptables.GetIptable(iptables.IPv4).Raw("-t", "nat", "-S", dockerChain)
assert.NilError(t, err)
return slices.DeleteFunc(strings.Split(string(out), "\n"), func(r string) bool {
return !strings.Contains(r, "-j DNAT")
})
}
assert.NilError(t, nw.AddPorts(t.Context(), []types.PortBinding{pb}))
rules := dnatRules()
assert.Assert(t, is.Len(rules, 1))
assert.Check(t, strings.Contains(rules[0], "! -i br-dummy") == tc.expProxySkip, "DNAT rule: %s", rules[0])
assert.NilError(t, nw.DelPorts(t.Context(), []types.PortBinding{pb}))
assert.Check(t, is.Len(dnatRules(), 0))
})
}
}
@@ -102,7 +102,7 @@ func (n *network) setPerPortNAT(ipv iptables.IPVersion, b types.PortBinding, ena
"-j", "DNAT",
"--to-destination", net.JoinHostPort(b.IP.String(), strconv.Itoa(int(b.Port))),
}
if !n.ipt.config.Hairpin {
if !n.config.DNATFromOwnBridge(n.ipt.config, b) {
args = append(args, "!", "-i", n.config.IfName)
}
if ipv == iptables.IPv6 {
@@ -242,3 +242,69 @@ func TestHairpinMultipleHostPorts(t *testing.T) {
assert.Check(t, is.Equal(countInChain(postRtChain, masqComment), 0))
assert.Check(t, is.Equal(countInChain(natChain, dnatComment), 0))
}
// TestDNATFromOwnBridge checks when a port binding's DNAT rule leaves packets
// arriving from the network's own bridge to the userland proxy, and that the
// rule is deleted with the binding.
func TestDNATFromOwnBridge(t *testing.T) {
nftables.Enable()
defer nftables.Disable()
testcases := []struct {
hairpin, icc, noSelfConnect bool
expProxySkip bool
}{
{hairpin: true, icc: true},
{hairpin: true, icc: false, noSelfConnect: true},
{icc: true, expProxySkip: true},
{icc: true, noSelfConnect: true, expProxySkip: true},
{icc: false, expProxySkip: true},
{icc: false, noSelfConnect: true},
}
for _, tc := range testcases {
t.Run(fmt.Sprintf("hairpin=%v/icc=%v/noSelfConnect=%v", tc.hairpin, tc.icc, tc.noSelfConnect), func(t *testing.T) {
defer netnsutils.SetupTestOSContext(t, netnsutils.WithSetNsHandles(false))()
fw, err := NewNftabler(t.Context(), firewaller.Config{IPv4: true, Hairpin: tc.hairpin})
assert.NilError(t, err)
defer fw.Close()
nw, err := fw.NewNetwork(t.Context(), firewaller.NetworkConfig{
IfName: "br-dummy",
ICC: tc.icc,
Masquerade: true,
Config4: firewaller.NetworkConfigFam{Prefix: netip.MustParsePrefix("192.168.0.0/24")},
})
assert.NilError(t, err)
pb := types.PortBinding{
Proto: types.TCP,
IP: net.ParseIP("192.168.0.2"),
Port: 80,
HostIP: net.IPv4zero,
HostPort: 8080,
HostPortEnd: 8080,
NoSelfConnect: tc.noSelfConnect,
}
dnatRules := func() []string {
t.Helper()
res := icmd.RunCommand("nft", "list", "table", "ip", dockerTable)
assert.Assert(t, res.Error, res.Combined())
var rules []string
for _, r := range strings.Split(res.Stdout(), "\n") {
if strings.Contains(r, "dnat to") {
rules = append(rules, r)
}
}
return rules
}
assert.NilError(t, nw.AddPorts(t.Context(), []types.PortBinding{pb}))
rules := dnatRules()
assert.Assert(t, is.Len(rules, 1))
assert.Check(t, strings.Contains(rules[0], `iifname != "br-dummy"`) == tc.expProxySkip, "DNAT rule: %s", rules[0])
assert.NilError(t, nw.DelPorts(t.Context(), []types.PortBinding{pb}))
assert.Check(t, is.Len(dnatRules(), 0))
})
}
}
@@ -144,10 +144,6 @@ func (n *network) setPerPortHairpinForwarding(pbs []types.PortBinding, updater f
}
func (n *network) setPerPortDNAT(pbs []types.PortBinding, updater func(nftables.Obj), ipv nftables.Family) {
var proxySkip string
if !n.fw.config.Hairpin {
proxySkip = fmt.Sprintf("iifname != %s ", n.config.IfName)
}
var v6LLSkip string
if ipv == nftables.IPv6 {
v6LLSkip = "ip6 saddr != fe80::/10 "
@@ -162,6 +158,10 @@ func (n *network) setPerPortDNAT(pbs []types.PortBinding, updater func(nftables.
if (pb.IP.To4() != nil) != (pb.HostIP.To4() != nil) {
continue
}
var proxySkip string
if !n.config.DNATFromOwnBridge(n.fw.config, pb) {
proxySkip = fmt.Sprintf("iifname != %s ", n.config.IfName)
}
var daddrMatch string
if !pb.HostIP.IsUnspecified() {
daddrMatch = fmt.Sprintf("%s daddr %s ", ipv, pb.HostIP)
@@ -7,6 +7,7 @@ import (
"os"
"testing"
"github.com/moby/moby/v2/daemon/libnetwork/drivers/bridge/internal/firewaller"
"github.com/moby/moby/v2/daemon/libnetwork/drvregistry"
"github.com/moby/moby/v2/daemon/libnetwork/portallocator"
"github.com/moby/moby/v2/daemon/libnetwork/portmappers/nat"
@@ -112,6 +113,23 @@ func TestAddEphemeralPortsTracksSeparately(t *testing.T) {
assert.Check(t, is.Equal(env.ep.ephemeralPortMapping[0].Proto, types.TCP))
}
// TestAddEphemeralPortsNoSelfConnect checks that a published port's
// NoSelfConnect setting reaches the firewaller, which needs it to decide
// whether packets from the bridge are DNATed or left for the userland proxy.
func TestAddEphemeralPortsNoSelfConnect(t *testing.T) {
defer netnsutils.SetupTestOSContext(t)()
env := newEphTestEnv(t, portBindingMode{ipv4: true})
pp := pubPort(types.TCP, 8080)
pp.NoSelfConnect = true
err := env.d.AddEphemeralPorts(t.Context(), ephTestNID, ephTestEID, []types.PublishedPort{pp})
assert.NilError(t, err)
fwPorts := env.n.firewallerNetwork.(*firewaller.StubFirewallerNetwork).Ports
assert.Assert(t, is.Len(fwPorts, 1))
assert.Check(t, fwPorts[0].NoSelfConnect)
}
// TestAddEphemeralPortsNoActiveMode documents a sharp edge rather than asserting
// it is desirable: with no NAT family active on the endpoint, addPortMappings
// generates no bindings and returns no error, so AddEphemeralPorts reports success
@@ -471,12 +471,13 @@ func collectFirewallPorts(pbs []portmapperapi.PortBinding) []types.PortBinding {
// that can be passed to firewaller.Network for setting up a NAT rule.
func toNATBinding(pb portmapperapi.PortBinding) types.PortBinding {
return types.PortBinding{
IP: pb.IP,
Port: pb.Port,
Proto: pb.Proto,
HostIP: pb.NAT.Addr().AsSlice(),
HostPort: pb.NAT.Port(),
HostPortEnd: pb.NAT.Port(),
IP: pb.IP,
Port: pb.Port,
Proto: pb.Proto,
HostIP: pb.NAT.Addr().AsSlice(),
HostPort: pb.NAT.Port(),
HostPortEnd: pb.NAT.Port(),
NoSelfConnect: pb.NoSelfConnect,
}
}
+4
View File
@@ -284,6 +284,10 @@ func publishedPort(pc *PortConfig) (types.PublishedPort, error) {
Proto: types.ParseProtocol(strings.ToLower(pc.Protocol.String())),
Port: uint16(pc.PublishedPort),
HostPort: uint16(pc.PublishedPort),
// The ports are published on the load-balancer sandbox's gateway
// endpoint. The sandbox has no container, so nothing behind the
// endpoint connects to them.
NoSelfConnect: true,
}, nil
}
+2 -2
View File
@@ -49,7 +49,7 @@ func TestPublishedPortIgnoresTargetPort(t *testing.T) {
assert.NilError(t, err)
assert.Check(t, is.Equal(a, b))
assert.Check(t, is.Equal(a, types.PublishedPort{Proto: types.TCP, Port: 8080, HostPort: 8080}))
assert.Check(t, is.Equal(a, types.PublishedPort{Proto: types.TCP, Port: 8080, HostPort: 8080, NoSelfConnect: true}))
// A differing protocol is a different reservation.
udp, err := publishedPort(&PortConfig{Protocol: ProtocolUDP, PublishedPort: 8080, TargetPort: 80})
@@ -84,7 +84,7 @@ func TestIngressPortRefsAcrossGenerations(t *testing.T) {
ppsB, err := publishedPorts([]*PortConfig{{Protocol: ProtocolTCP, PublishedPort: 8080, TargetPort: 8080}})
assert.NilError(t, err)
want := types.PublishedPort{Proto: types.TCP, Port: 8080, HostPort: 8080}
want := types.PublishedPort{Proto: types.TCP, Port: 8080, HostPort: 8080, NoSelfConnect: true}
// The first generation publishes the reservation.
assert.Check(t, is.DeepEqual(refIngressPorts(ppsA), []types.PublishedPort{want}))
+22 -11
View File
@@ -56,6 +56,11 @@ type PortBinding struct {
HostIP net.IP
HostPort uint16
HostPortEnd uint16
// NoSelfConnect is a hint that nothing behind the endpoint connects to
// its own published port through the host. Packets from other endpoints
// on the same network can then be DNATed to the port without hairpin
// mode or masquerading, instead of being left for the userland proxy.
NoSelfConnect bool `json:",omitempty"`
}
// PublishedPort is a request to publish a single container port on the host.
@@ -67,16 +72,20 @@ type PublishedPort struct {
Proto Protocol
Port uint16 // container port
HostPort uint16 // single host port - there is no range
// NoSelfConnect is a hint that nothing behind the endpoint connects to
// its own published port through the host. See [PortBinding.NoSelfConnect].
NoSelfConnect bool
}
// PortBinding returns the single-port [PortBinding] equivalent of p, with
// HostPortEnd pinned to HostPort so it can never denote a range.
func (p PublishedPort) PortBinding() PortBinding {
return PortBinding{
Proto: p.Proto,
Port: p.Port,
HostPort: p.HostPort,
HostPortEnd: p.HostPort,
Proto: p.Proto,
Port: p.Port,
HostPort: p.HostPort,
HostPortEnd: p.HostPort,
NoSelfConnect: p.NoSelfConnect,
}
}
@@ -97,12 +106,13 @@ func (p PortBinding) HostAddr() (net.Addr, error) {
// Copy returns a deep copy of the PortBinding.
func (p PortBinding) Copy() PortBinding {
return PortBinding{
Proto: p.Proto,
IP: slices.Clone(p.IP),
Port: p.Port,
HostIP: slices.Clone(p.HostIP),
HostPort: p.HostPort,
HostPortEnd: p.HostPortEnd,
Proto: p.Proto,
IP: slices.Clone(p.IP),
Port: p.Port,
HostIP: slices.Clone(p.HostIP),
HostPort: p.HostPort,
HostPortEnd: p.HostPortEnd,
NoSelfConnect: p.NoSelfConnect,
}
}
@@ -113,7 +123,8 @@ func (p PortBinding) Equal(o PortBinding) bool {
p.Port == o.Port &&
p.HostIP.Equal(o.HostIP) &&
p.HostPort == o.HostPort &&
p.HostPortEnd == o.HostPortEnd
p.HostPortEnd == o.HostPortEnd &&
p.NoSelfConnect == o.NoSelfConnect
}
// String returns the PortBinding structure in the form "HostIP:HostPort:IP:Port/Proto",
@@ -91,7 +91,11 @@ Note that:
- The published port is set up as an ordinary port mapping on the ingress
load-balancer sandbox's `docker_gwbridge` gateway endpoint (`172.18.0.2`),
using the same rules as any other published container port:
- a DNAT rule in the nat `DOCKER` chain, and
- a DNAT rule in the nat `DOCKER` chain. Unlike a container's port mapping
when the userland proxy is running, it doesn't skip packets arriving on
`docker_gwbridge`. The load-balancer never connects to its own published
port, so a task that reaches the port via its node's address is DNATed
like any other client, and doesn't go through docker-proxy, and
- ACCEPT rules in the filter `DOCKER` chain. There are two, because
`docker_gwbridge` has inter-container communication disabled: one for
packets from outside the network, and one for packets from the network
@@ -118,7 +122,7 @@ And the corresponding nat table:
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 DNAT tcp -- !docker_gwbridge any anywhere anywhere tcp dpt:http-alt to:172.18.0.2:8080
1 0 0 DNAT tcp -- any any anywhere anywhere tcp dpt:http-alt to:172.18.0.2:8080
<details>
@@ -133,7 +137,7 @@ And the corresponding nat table:
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER ! -i docker_gwbridge -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.18.0.2:8080
-A DOCKER -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.18.0.2:8080
</details>
@@ -22,7 +22,11 @@ Note that:
- The published port is set up as an ordinary port mapping on the ingress
load-balancer sandbox's `docker_gwbridge` gateway endpoint (`172.18.0.2`),
using the same rules as any other published container port:
- a DNAT rule in the nat `DOCKER` chain, and
- a DNAT rule in the nat `DOCKER` chain. Unlike a container's port mapping
when the userland proxy is running, it doesn't skip packets arriving on
`docker_gwbridge`. The load-balancer never connects to its own published
port, so a task that reaches the port via its node's address is DNATed
like any other client, and doesn't go through docker-proxy, and
- ACCEPT rules in the filter `DOCKER` chain. There are two, because
`docker_gwbridge` has inter-container communication disabled: one for
packets from outside the network, and one for packets from the network
+17 -7
View File
@@ -995,17 +995,21 @@ func probeUDPPort(t *testing.T, host networking.Host, hostAddr, port string) err
// TestIngressPortFromTaskOnSameNode checks that a container on the node can
// reach a service's published port via one of the node's own addresses, with
// and without the userland proxy.
// and without the userland proxy, while the node drops unsolicited input.
//
// The ingress load balancer's published ports are mapped onto its
// docker_gwbridge endpoint, and docker_gwbridge is created with inter-container
// communication disabled. With docker-proxy, the connection is served from the
// host's netns and never crosses that bridge. Without it, the packets are
// DNATed across the bridge, so they have to get past the rules that implement
// icc=false. Reaching a published port shouldn't depend on the userland proxy,
// so both modes are checked and expected to behave the same.
// communication disabled. The container's packets are DNATed across that
// bridge, so they have to get past the rules that implement icc=false. With
// the userland proxy enabled, packets from a container's own bridge are
// normally left for docker-proxy instead, which receives them through the
// host's INPUT chain, where a host firewall such as ufw drops them. Ingress
// ports are DNATed anyway, so that reaching a published port doesn't depend
// on the userland proxy or the host's input policy. Both modes are checked
// and expected to behave the same.
//
// Regression test for https://github.com/moby/moby/issues/53713
// Regression test for https://github.com/moby/moby/issues/53713 and
// https://github.com/moby/moby/issues/53962
func TestIngressPortFromTaskOnSameNode(t *testing.T) {
skip.If(t, testEnv.IsRemoteDaemon)
skip.If(t, testEnv.IsRootless, "rootless mode doesn't support Swarm-mode")
@@ -1031,6 +1035,12 @@ func TestIngressPortFromTaskOnSameNode(t *testing.T) {
defer l3.Destroy(t)
l3.AddHost(t, l3SegHost, "ns-"+l3SegHost, "eth0", netip.MustParsePrefix(hostAddr+"/24"))
// Drop unsolicited input, like ufw's default "deny incoming" policy.
host := l3.Hosts[l3SegHost]
host.MustRun(t, "iptables", "-A", "INPUT", "-i", "lo", "-j", "ACCEPT")
host.MustRun(t, "iptables", "-A", "INPUT", "-m", "conntrack", "--ctstate", "RELATED,ESTABLISHED", "-j", "ACCEPT")
host.MustRun(t, "iptables", "-P", "INPUT", "DROP")
l3.Hosts[l3SegHost].Do(t, func() {
// Not swarm.NewSwarm, because it can't pass extra daemon flags.
d := daemon.New(t, daemon.WithSwarmIptables(true))