Because we set SO_REUSEADDR on sockets for host ports, if there
are port mappings for INADDR_ANY (the default) as well as for
specific host ports - bind() cannot be used to detect clashes.
That means, for example, on daemon startup, if the port allocator
returns the first port in its ephemeral range for a specific host
adddress, and the next port mapping is for 0.0.0.0 - the same port
is returned and both bind() calls succeed. Then, the container
fails to start later when listen() spots the problem and it's too
late to find another port.
So, bind and listen to each set of ports as they're allocated
instead of just binding.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Go maintainers started to unconditionally update the minimum go version
for golang.org/x/ dependencies to go1.23, which means that we'll no longer
be able to support any version below that when updating those dependencies;
> all: upgrade go directive to at least 1.23.0 [generated]
>
> By now Go 1.24.0 has been released, and Go 1.22 is no longer supported
> per the Go Release Policy (https://go.dev/doc/devel/release#policy).
>
> For golang/go#69095.
This updates our minimum version to go1.23, as we won't be able to maintain
compatibility with older versions because of the above.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
For kernels that don't have CONFIG_IP_NF_RAW, if the env
var DOCKER_INSECURE_NO_IPTABLES_RAW is set to "1", don't
try to create raw rules.
This means direct routing to published ports is possible
from other hosts on the local network, even if the port
is published to a loopback address.
Signed-off-by: Rob Murray <rob.murray@docker.com>
On i386, Linux doesn't provide direct socket syscall but instead
multiplexes them through the socketcall syscall (see `man 2 socketcall`).
This commit fixes compilation for i386 by wrapping the offending syscall
in a new function that uses the socketcall syscall on i386, and
the `setsockopt` syscall on other archs.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
New iptables rules were added by:
- commit 27adcd5 (libnet/d/bridge: port mappings: drop direct-access
when gw_mode=nat)
- commit d216084 (libnet/d/bridge: drop remote connections to port
mapped on lo)
But they're added even when iptables=false or ip6tables=false in daemon
config.
So, group those rules with the other per-port rules, behind a config
check.
Signed-off-by: Rob Murray <rob.murray@docker.com>
In the bridge network driver, filterPortMappedOnLoopback and
filterDirectAccess each set up some per-port iptables rules - but
they're not called by setPerPortIptables.
So, make sure they're called when firewalld is reloaded.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Traditionally when Linux receives remote packets with daddr set to a
loopback address, it reject them as 'martians'. However, when a NAT rule
is applied through iptables this doesn't happen. Our current DNAT rule
used to map host ports to containers is applied unconditionally, even
for such 'martian' packets.
This means a neighbor host (ie. a host connected to the same L2
segment) can send packets to a port mapped on a loopback address. The
purpose of publishing on a loopback address is to make ports
inaccessible to remote hosts -- lack of proper filtering defeats that.
This commit adds an iptables rule to the raw-PREROUTING chain to drop
packets with a loopback dest address and coming from any interface other
than lo.
To accomodate WSL2 mirrored mode, another rule is inserted beforehand to
specifically accept packets coming from the loopback0 interface.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
When a NAT-based port mapping is created, the daemon adds a DNAT rule in
nat-DOCKER to replace the dest addr with the container IP. However, the
daemon never sets up rules to filter packets destined directly to the
container port. This allows a rogue neighbor (ie. a host that shares a
L2 segment with the host) to send packets directly to the container on
its container-side exposed port.
For instance, if container port 5000 is mapped to host port 6000, a
neighbor could send packets directly to the container on its port 5000.
Since nat-DOCKER mangles the dest addr, and the nat table forbids DROP
rules, this change adds a new rule in the raw-PREROUTING chain to filter
ingress connections targeting the container's IP address.
This filtering is only done when gw_mode=nat. For the unprotected
variant, no filtering is done.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
When a NAT-based port mapping is created with a HostIP specified, we
insert a DNAT rule in nat-DOCKER to replace the dest addr with the
container IP. Then, in filter chains, we allow access to the container
port for any packet not coming from the container's network itself (if
hairpinning is disabled), nor from another host bridge.
However we don't set any rule that prevents a rogue neighbor that shares
a L2 segment with the host, but not the one where the port binding is
expected to be published, from sending packets destined to that HostIP.
For instance, if a port binding is created with HostIP == '127.0.0.1',
this port should not be accessible from anything but the lo interface.
That's currently not the case and this provides a false sense of
security.
Since nat-DOCKER mangles the dest addr, and the nat table rejects DROP
rules, this change adds rules into raw-PREROUTING to filter ingress
packets destined to mapped ports based on the input interface, the dest
addr and the dest port.
Interfaces are dynamically resolved when packets hit the host, thanks
to iptables' addrtype extension. This extension does a fib lookup of the
dest addr and checks that it's associated with the interface reached.
Also, when a proxy-based port mapping is created, as is the case when an
IPv6 HostIP is specified but the container is only IPv4-capable, we
don't set any sort of filtering. So the same issue might happen. The
reason is a bit different - in that case, that's just how the kernel
works. But, in order to stay consistent with NAT-based mappings, these
rules are also applied.
The env var `DOCKER_DISABLE_INPUT_IFACE_FILTERING` can be set to any
true-ish value to globally disable this behavior.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Same as "nat" mode, there's masquerading and port mapping from the
host - but no port/protocol filtering for direct access to the
container's address from remote hosts.
This is the old default behaviour for IPv4 when the filter-FORWARD
chain's default policy was "ACCEPT" (the daemon would only set it
to "DROP" when it set sysctl "ip_forward" itself, but it didn't set
up DROP rules for unpublished ports).
Now, port filtering doesn't depend on the filter-FORWARD policy. So,
this mode is added as a way to restore the old/surprising/insecure
behaviour for anyone who's depending on it. Networks will need to
be re-created with this new gateway mode.
Signed-off-by: Rob Murray <rob.murray@docker.com>
commit a0807e7cfe configured golangci-lint
to use go1.23 semantics, which alowed linters like `copyloopvar` to lint
using thee correct semantics.
go1.22 now creates a copy of variables when assigned in a loop; make sure we
don't have files that may downgrade semantics to go1.21 in case that also means
disabling that feature; https://go.dev/ref/spec#Go_1.22
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When a container sends a packet to one of its own published ports on the
host, it's normally picked up by the userland proxy and sent back.
When the userland proxy is disabled, a masquerade rule is needed in
order for responses to the container to have the host's source address.
The masquerade rule matches the container's address as source and dest,
and the published port as the dest. It's only used for the no-proxy
case.
So, when the userland proxy is enabled, don't create the masquerade
rule.
Signed-off-by: Rob Murray <rob.murray@docker.com>
When the host has IPv6 support but the container doesn't, the
default bind address is 0.0.0.0, and a port mapping does not
include a host address - the default behaviour (of the bridge
driver) is to run a docker-proxy to map from any IPv6 host
address to the IPv4 address of a container's gateway endpoint.
Driver option com.docker.network.driver.private.no_proxy_6to4
can now be used to disable that behaviour when configuring
a Sandbox's external connectivity.
Signed-off-by: Rob Murray <rob.murray@docker.com>
When a UDP server is running on a multihomed server, as is the case with
pretty much _all_ Docker hosts (eg. eth0 + docker0), the kernel has to
choose which source address is used when replying to a UDP client. But
that process is based on heuristics and is fallible.
If the address picked doesn't match the original destination address
used by the client, it'll drop the datagram and return an ICMP Port
Unreachable.
To prevent that, we need to:
- `setsockopt(IP_PKTINFO)` on proxy's sockets.
- Extract the original destination address from an ancillary message
every time a new 'UDP connection' is 'established' (ie. every time we
insert a new entry into the UDP conntrack table).
- And finally, pass a control message containing the desired source
address to the kernel, every time we send a response back to the
client.
Also, update the inline comment on read errors in `(*UDPProxy).Run()`.
This comment was misleadingly referencing ECONNREFUSED - Linux's UDP
implementation never returns this error (see [1]). Instead, state why
`net.ErrClosed` is perfectly fine and doesn't need to be logged
(although, docker-proxy currently logs to nowhere).
[1]: https://github.com/search?q=repo%3Atorvalds%2Flinux+ECONNREFUSED+path%3A%2F%5Enet%5C%2F%28ipv4%7Cipv6%29%5C%2F%28udp%7Ctcp%29%2F&type=code
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Since commit b3fabede, the Engine creates the listening sockets used by
docker-proxy by making raw syscalls (ie. socket, setsockopt, bind).
Before that commit, those sockets were created by docker-proxy through
Go's `net.ListenX` functions.
Unlike `net.ListenX` functions, the raw syscall code doesn't set the
`SO_REUSEADDR` option. This option is typically used by TCP servers to
make sure that they can be restarted even if there are client sockets
referencing the server port as their sport (eg. in TIME_WAIT state, or
any other state).
Citing UNIX Network Programming, Section 7.5 (p210):
> By default, when the listening server is restarted by calling socket,
> bind, and listen, the call to bind fails because the listening server
> is trying to bind a port that is part of an existing connection [...]
> _All_ TCP servers should specify this socket option to allow the
> server to be restarted in this situation.
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Link-local connections were DNATed like other non-loopback connections,
but the kernel would drop them even before their reach the container.
This commit changes the DNAT rule inserted in ip6tables to exclude
link-local addresses. Instead, these connections will be proxied by
docker-proxy, at least if --userland-proxy=true.
If dockerd is started with the userland-proxy disabled, link-local
port-bindings won't be supported (ie. silently discarded).
Signed-off-by: Albin Kerouanton <albinker@gmail.com>
Previously, if a port mapping specified a host IP or port that
could not be used because the endpoint's network was in routed
mode (so, there's no host binding), it'd be treated as an error.
However:
- the selected gateway endpoint may change over time, as networks
are connected and disconnected - so the binding may make sense
for some other endpoint.
- the validation was complicated, duplicated logic in order to
fail early, and wasn't complete.
So, just log when fields are ignored, at the point where they're
ignored.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Before commit 4f09af6, when allocating host ports for a new
port mapping, iptables rules were set up then docker-proxy was
started. If the host port was already in-use, docker-proxy exited
with an error, and the iptables rules were removed. That could
potentially interfere with a non-docker service that was already
using the host port for something unrelated.
Commit 4f09af6 swapped that problem for a different one... in
order to check that a port was available before creating iptables
rules, it attempted to start docker-proxy first. If it failed, it
could then try a different host port, without interfering with
any other service. The problem with that is docker-proxy would
start listening before the iptables rules were in place, so it
could accept connections then become unusable because new NAT
rules diverted packets directly to the container. This would leave
the client with a broken connection, causing at-least a delay
while it figured that out and reconnected.
This change creates and binds the socket in the daemon, before
creating iptables rules. If the bind fails, it may try a different
port. When or if the bind succeeds, iptables rules are created,
then the daemon calls listen on the socket. If docker-proxy is
needed, the socket is handed over to it at that point.
In rootless mode, the ports have to be bound to an address in the
rootless network namespace (where dockerd is running). DNAT rules
now use the same address.
If docker-proxy is not needed ("--userland-proxy=false"), the daemon
still listens on TCP sockets as the old dummyProxy would have done.
This makes the socket show up in "netstat" output.
The dummyProxy is no longer needed on Linux. Its job was to bind the
host ports if docker-proxy was disabled, but that's now already
handled by binding the sockets early.
This change doesn't affect SCTP, because it's not currently possible
for docker-proxy to convert the file descriptor into an SCTPListener.
So, docker-proxy is still started early, and the window for lost
connections remains.
If the user has an old docker-proxy in their path and it's given a
listener docker with '-use-listen-fd', it'll fail because of the
unknown option. In this case, the daemon's error message suggests
checking $PATH.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Before this change, when running rootless, instead of running
docker-proxy the daemon would run rootlesskit-docker-proxy.
The job of rootlesskit-docker-proxy was to tell RootlessKit
about mapped host ports before starting docker-proxy, and then
to remove the mapping when it was stopped.
So, rootlesskit-docker-proxy would need to be kept in-step
with changes to docker-proxy (particuarly the upcoming change
to bind TCP/UDP ports in the daemon and pass them to the proxy,
but also possible-future changes like running proxy per-container
rather than per-port-mapping).
This change runs the docker-proxy in rootless mode, instead of
rootlesskit-docker-proxy, and the daemon itself tells RootlessKit
about changes in host port mappings.
Signed-off-by: Rob Murray <rob.murray@docker.com>
For SCTP port mappings, an iptables mangle rule was create to fix
the checksum. The code comment says it was done to fix a problem
caused by the veth driver reporting NETIF_F_SCTP_CRC since kernel
4.9, when used with NICs that do not support NETIF_F_SCTP_CRC.
But the checksum rule fills in the checksum in the IP header,
not an SCTP checksum, so it doesn't seem related. And, quoting
from the github issue, "since kernel v4.19, the xt_CHECKSUM.c
explicitly states it should only be used for UDP and only in the
OUTPUT chain - while docker is using it for SCTP in the PREROUTING
chain".
Furthermore, the rule is reported to be causing errors for SCTP
between containers.
Because we don't entirely understand why the rule exists, it
is now only added if the daemon's env has
DOCKER_IPTABLES_SCTP_CHECKSUM=1. If no problems are reported, that
escape-hatch and the code to add the rule will be removed in a
future release.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Regression introduced in 01eecb6.
A port mapping from a specific IPv6 host address can be used
by a container on an IPv4-only network, docker-proxy makes the
connection.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Following changes to the port mapping code, the DNAT iptables rule was
inserted into the nat table rather than appended.
This meant DNAT was applied before the rule that should have skipped
it when a packet was from a bridge network.
So, packets sent from a container on one network to a mapped port on
the host's address were DNAT'd before docker-proxy could pick them up,
then they were dropped by a rule intended to isolate the networks.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Looks like some packages fail in go module mode, because they require
recent Go versions:
GO111MODULE=on go test -v
# github.com/docker/docker/libnetwork/ipamutils
../../libnetwork/ipamutils/utils.go:46:9: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipamutils/utils.go:51:9: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
# github.com/docker/docker/libnetwork/portallocator
../../libnetwork/portallocator/portallocator.go:179:7: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
# github.com/docker/docker/libnetwork/netutils
../../libnetwork/netutils/utils_linux.go:66:14: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/netutils/utils_linux.go:75:2: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
# github.com/docker/docker/api/server/router/grpc
../../api/server/router/grpc/grpc.go:56:48: predeclared any requires go1.18 or later (-lang was set to go1.16; check go.mod)
# github.com/docker/docker/container
../../container/view.go:335:47: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
# github.com/docker/docker/libnetwork/ipams/defaultipam
../../libnetwork/ipams/defaultipam/address_space.go:33:2: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipams/defaultipam/address_space.go:53:2: clear requires go1.21 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipams/defaultipam/address_space.go:124:10: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipams/defaultipam/address_space.go:125:21: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipams/defaultipam/address_space.go:146:22: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipams/defaultipam/address_space.go:310:14: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/ipams/defaultipam/address_space.go:311:22: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
# github.com/docker/docker/libnetwork/drivers/bridge
../../libnetwork/drivers/bridge/port_mapping_linux.go:76:15: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
../../libnetwork/drivers/bridge/port_mapping_linux.go:201:2: implicit function instantiation requires go1.18 or later (-lang was set to go1.16; check go.mod)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When bridge driver opt com.docker.network.bridge.gatway_mode_ipv[46]
is set to "routed", there is no NAT.
When there's no NAT, there's no meaning to the HostPort field in a
port mapping (all the port mapping does is open the container's port),
and the HostIP field is only used to determine the address family.
So, check port bindings, and raise errors if fields are unexpectedly
set when the mapping only applies to a gateway_mode=routed network.
Zero-addresses are allowed, to say the mapping/open-port should be
IPv4-only or IPv6-only, and host ports are not allowed.
A mapping with no host address, so it applies to IPv4 and IPv6 when
the default binding is 0.0.0.0, may include a host port if either
uses NAT. The port number is ignored for the directly-routed family.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Without this change, if a port mapping did not specify a host address
and the network was IPv6-enabled, the same port would be allocated for
mappings from '0.0.0.0' and '::'. But, if the port mapping was specified
with explicit addresses even, for example:
-p 0.0.0.0:8080-8083:80 -p '[::]:8083-8080:80'
This change looks for port mappings that only differ in the host IP
address, and makes sure it allocates the same port for all of them. If
it can't, it fails with an error.
Signed-off-by: Rob Murray <rob.murray@docker.com>
Add bridge driver options...
com.docker.network.bridge.gateway_mode_ipv4=<nat|routed>
com.docker.network.bridge.gateway_mode_ipv6=<nat|routed>
If set to "routed", no NAT or masquerade rules are set up for port
mappings.
When NAT is disabled, the mapping is shown in 'inspect' output with
no host port number. For example, for "-p 80" with NAT disabled for
IPv6 but not IPv4:
"80/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "32768"
},
{
"HostIp": "::",
"HostPort": ""
}
Signed-off-by: Rob Murray <rob.murray@docker.com>
Use the bridge driver's iptables types to set up portmapping related
iptables rules - instead of using iptables.Forward, which is bridge
specific code in the iptables package.
Remove iptables.Forward() and its unit test, the bridge driver's
version is covered by TestAddPortMappings.
Remove hairpinMode from iptables.ChainInfo hairpinMode relates to bridge
driver specific behaviour, that is now implemented in the bridge driver.
Signed-off-by: Rob Murray <rob.murray@docker.com>
The bridge driver now does its own port-mapping, rather than using the
portmapper module (which ran as two completely separate instances, for
IPv4 and IPv6).
When asked for a mapping from any host address (0.0.0.0/0) with a range
of host ports, the same port will be allocated for IPv4 and IPv6, or the
mapping will fail with an error if that's not possible.
The bridge driver now manages its own port mappings. So, remove
linux-specific PortMapper code and make what's left Windows-only.
Also, replace the portmapper.userlandProxy interface with StartProxy().
Signed-off-by: Rob Murray <rob.murray@docker.com>
For a port mapping like '-p 8080-8083:80', when some non-docker process
is using a port in the range, try other ports in the range. And, don't
do that on live-restore.
Because the port mapping may fail on live-restore, leaving no ports
mapped for the endpoint - update the view of mapped ports shown in
'inspect' output. (The wrong mappings will still be shown in 'docker ps',
the container will be left running and connected to the network, it just
won't work. There's plenty of scope for better error handling here.)
Signed-off-by: Rob Murray <rob.murray@docker.com>
