Add a test-surface specification for the overlay network driver and the
Swarm service mesh: what each part is expected to do, as observed from
outside the daemon. Each expectation carries a tier for what a failing
test means, the vantage point it is observed from, and what its verdict
depends on: the operating system, a package's upgrade path, the docker
CLI, or none of them. The spec is a model of the daemon's behaviour.
Which expectations a test suite checks, and where, is the suite's
choice.
An expectation holds on a node in its baseline state unless it says
otherwise. For a daemon installed from a package, the baseline is a
clean install of a supported OS, a fresh install of the package, and
the setup that the product's installation documentation prescribes.
For any other daemon, it is a host that meets the host requirements
that each spec lists: the kernel facilities, programs and host
configuration that its expectations rely on.
It describes behaviour only, so it should change when the daemon's
behaviour does, or when a regression shows that an expectation was only
assumed.
Expectations that came up along the way but are not Swarm's go in stubs
beside it, so the Swarm spec can refer to them without owning them.
bridge-networking.md covers the bridge driver and the parts of the
daemon's firewall integration that are not specific to Swarm,
DOCKER-USER among them. ipvlan-networking.md covers the ipvlan driver.
container-dns.md covers name resolution that does not depend on the
network driver. network-common.md covers what every network driver must
do, and the Engine API requests for networks and endpoints. The
conventions all five follow are in a README.md of their own, so that a
new spec can use them without editing an existing one. The specs live
in daemon/libnetwork/docs/spec.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When the daemon started with the nftables backend after a release older
than 28.0.0 had run with iptables, NewCleaner deleted the jumps from
filter-FORWARD to DOCKER-FORWARD and DOCKER-ISOLATION-STAGE-1, then
removed Docker's chains. Those releases also jumped from FORWARD to the
filter table's DOCKER chain for each bridge ("-o docker0 -j DOCKER").
iptables refuses to delete a chain that a rule still jumps to, and
removeIPChains ignores the error. The cleaner's DelNetwork deleted the
per-bridge jump later, when the bridge driver restored the network, but
nothing tried to remove the chain again. The empty DOCKER chain
remained.
When the bridge driver has restored its networks and endpoints, call
the cleaner's new Done method, which removes the chains again.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When the daemon started with the nftables backend after a release older
than 28.0.1 had run with iptables, the iptables cleaner's DelNetwork
left the per-bridge rules those releases put in filter-FORWARD:
"-i docker0 ! -o docker0 -j ACCEPT" and the ICC rules. The iptables
backend deleted them in setupNonInternalNetworkRules and setIcc, but
only when adding a network's rules, and DelNetwork returned early for an
internal network.
Delete all the legacy per-bridge FORWARD rules in
deleteLegacyFilterRules, which the iptables backend and the cleaner call
for every network.
With the iptables backend, setupIPTables deletes the legacy rules after
calling setupNonInternalNetworkRules, which used to delete
"-i docker0 ! -o docker0 -j ACCEPT" before adding the DOCKER-FORWARD
rule that replaces it. Now, for a moment, both rules are present. Both
accept outgoing packets from the bridge.
Signed-off-by: Cory Snider <csnider@mirantis.com>
matchesType tests a network against only the first value of the type
filter. The values come from a map. Their order is random and changes
from one network to the next. A GET /networks request that filters on
both type=builtin and type=custom lists a random subset of the
networks. Before networks were filtered one at a time, the daemon
filtered the list once for each type value and listed every network
that matched any of them.
Match a network when its type is any of the filter's values.
integration-cli's TestDockerNetworkLsFilter checks this request but is
skipped in every run. Its OnlyDefaultNetworks requirement is met only
when GET /networks lists no networks at all. The daemon always lists
the predefined bridge, host and none networks. Port the test to
integration/network as TestNetworkListFilters, which uses the API
client. The new test checks the same id, name, type, label and driver
filter terms against two networks that it creates and the predefined
networks. It derives the expected results for type=custom and
driver=bridge from the unfiltered list, as the daemon can have other
networks. Remove the integration-cli test, its assertNwList helper and
the OnlyDefaultNetworks requirement. Nothing else uses them.
Add the combined type values to the filter unit test as well. Both
tests fail on the combined values without the fix.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Remove the redundant ErrorCoder type assertion and let the
recursive call handle unsupported error types.
Make all switch cases return explicitly.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add missing mappings for ErrFailedPrecondition (412) and
ErrResourceExhausted (429), aligning with containerd's errhttp
implementation.
Add table-driven tests covering the errdefs mappings, gRPC and
distribution errors, and wrapped and joined errors.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use errors.AsType to preserve the original JSON error details
when the error is wrapped, instead of replacing them with a
generic error containing only the message.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This error response was added in [moby@1421293], at which time the response
only contained a flat `error` field. This field was deprecated in favor of
an `errorDetails` struct in [moby@3043c26], but failed to update `FormatError`,
which continued to use the deprecated
`error` field.
Update `FormatError` to use the non-deprecated field, and update the error
message to be more descriptive. We're intentionally not preserving the actual
JSON marshal error, as it should be very unlikely to happen, and quoting the
error-message is not guaranteed to produce a JSON-compatible format.
[moby@1421293]: https://github.com/moby/moby/commit/14212930e40439d0cfc635d56e084736867f9e30
[moby@3043c26]: https://github.com/moby/moby/commit/3043c2641990d94298c6377b7ef14709263a4709
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a native nftables implementation of service load balancing which
achieves feature parity with the iptables firewall mode. This is
accomplished using a static nftables ruleset with dynamic data. NAT and
DSR modes are both supported. The backend server for each flow is chosen
randomly rather than round-robin like the iptables+IPVS implementation.
As the load balancer endpoint only sees one side of DSR flows we cannot
use netfilter's conntrack subsystem. And IPVS does not play well with
nftables -- nftables is missing an analogue for the `ipvs` xtables
matcher. DSR flows are made sticky by implementing one-sided connection
tracking directly in the nftables ruleset using maps that are updated in
the packet path.
A network's load-balancer tables have exactly one writer: each
loadBalancer records the data plane it wants, and the complete element set
is composed from every recorded contribution. nat-publish-port is an
interval map keyed on `l4proto . dport . numgen`, so the bucket space for
a published port has to be partitioned by a single owner -- and a service
whose published-port set changes has a separate binding, hence a separate
loadBalancer, per generation until the rolling update drains. Two of them
partitioning a shared port independently would emit overlapping intervals,
which nftables rejects. Composing them instead partitions the port once
over every backend that serves it, so a shared published port is
load-balanced across both generations. IPVS cannot do that: one fwMark
cannot select a different backend set per port.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The ingress ports need to be published through the bridge driver
when the virtual service is initialized, irrespective of the firewall
backend in use. Change the firewall-specific implementation to return a
flag signaling that the service was initialized and arrange to have the
caller publish the ports.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Factor the iptables-specific service mesh configuration out of the
common Linux service mesh implementation in preparation for adding an
nftables implementation.
Signed-off-by: Cory Snider <csnider@mirantis.com>
With the iptables firewall backend, the overlay driver programs iptables
rules which mark VXLAN datagrams of encrypted networks for encryption,
drop cleartext VXLAN datagrams received for those networks, and strip
the encryption mark from datagrams marked by local processes. A
firewalld reload can flush these rules. The node then sends the
network's VXLAN datagrams in cleartext and accepts forged cleartext
datagrams for it until the last local endpoint of the network leaves and
another joins, or the daemon restarts.
On firewalld reload, restore all rules which were previously programmed
by the overlay driver.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
v2: Fix YAML unmarshaler validation bypass
Quoted null and tilde scalars bypass UnmarshalYAML, allowing invalid
values to escape application-defined validation. In applications
accepting configuration from less-trusted sources, this can lead to
denial of service or shared configuration corruption.
Require implicit scalar resolution before treating untagged scalars
as null. Add regression tests covering custom validation, both quote
styles, explicit tags, and actual null values.
full diff: https://github.com/yaml/go-yaml/compare/v2.4.4...v2.4.5
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Paweł Gronowski