Add the builtin org.mobyproject.jobs.v0 extension, serving the Jobs API
on the daemon socket.
It is experimental and disabled by default; enable it with
extensions.enable (--extension-enable).
Run containers are driven through an Engine API client from the daemon
API point, and their exits observed through the container state point,
so restart-policy attempts do not count as exits.
Jobs persist through the storage point, and unnamed jobs are named
through the name-generator point.
Removing a run container also removes its anonymous volumes.
The extension host initializes before the daemon serves its API, so
activation reconciles runs and starts the scheduler in the background,
and the Jobs API answers Unavailable until then.
Shutdown releases blocked calls such as Wait, and waits for activation
and in-flight calls only until its deadline.
Authorization plugins are not supported yet, as the README states.
Integration tests cover a manual run, reconciliation after a killed
daemon, and the disabled extension.
A README next to the contract describes how to use it.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Resolve the runs an unclean stop leaves behind, after a crash or a
live-restore shutdown, before the scheduler re-arms.
A run whose start was never recorded is failed, as its outcome cannot
be trusted; a container it left behind is stopped first, so it cannot
overlap the job's next run; one whose ID was never recorded is found
by its name and removed.
A started run is re-attached to the exit watcher, which records an
exit from the downtime before enforcing a timeout re-armed from the
original start.
A job stuck running with a terminal run is returned to idle.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the lifecycle core of the jobs extension.
Registration is idempotent on a canonical spec hash, so formatting and
defaults do not change a job's identity, and a spec from a newer
contract fails validation on an older daemon.
Create-and-run fires the first run before anything else can reach the
new job.
Container specs with a negative StopTimeout are rejected, as runs must
be stoppable, and so is the host-dependent Local timezone.
Every fire goes through one chokepoint serialized per job, which
applies the concurrency policy and persists the run record before the
container is created, so a crash always leaves a discoverable run.
A fire is detached from its caller, as the daemon completes a
container start even when the request goes away.
A run whose start is unknown, because the start request failed or its
record could not be written, has its container stopped and keeps the
job occupied until the container exits.
Outcomes are persisted before auto-removal, and failed terminal or
idle writes are retried instead of leaving the job stuck; only the
job's latest run returns it to idle.
Cancels and timeouts always stop the container, even after the other
decided the outcome, so a failed stop is not final.
The scheduler is one goroutine over a min-heap, dispatching through
the same chokepoint.
On start, missed occurrences follow the missed-fires policy with at
most one catch-up, and an occurrence moved by a reschedule is kept.
A reschedule replaces exactly one occurrence, including one a tick
already consumed but has not fired yet.
A dispatched fire is dropped if its job was paused, resumed or removed
since, and the next-fire time is persisted from the scheduler's
current state after the fire is recorded, so a delayed fire cannot undo
a newer change and a crash cannot lose the occurrence.
A fire deferred by the queue policy keeps its occurrence recorded until
it runs, so a restart in between catches it up.
Removing a job keeps its run history by default, reachable by the
removed job's ID, and run containers are removed with their records.
Stored specs are compared as the running daemon canonicalizes them, so
jobs survive upgrades that change the container API types.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the five-field cron grammar of schedule triggers: numbers, lists,
ranges and steps.
Names, shortcuts and Quartz extensions are left to clients.
Day-of-month and day-of-week combine following Vixie's rule.
Occurrences follow the wall clock of the job's timezone: a time
skipped by a DST transition does not fire that day, and a repeated
one fires once.
The search is bounded by the 400-year Gregorian cycle, so impossible
dates never fire while sparse ones such as February 29 on a Sunday
still do.
Zone data is embedded, so timezones resolve on hosts without a zoneinfo
database, such as Windows.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the persistence layer of the jobs extension on top of the storage
point, with one record per job and per run.
Records are versioned, and damaged ones are quarantined at load with a
warning instead of failing the daemon; a backend that cannot list or
read records fails the load.
Iteration numbers are persisted before the runs they number, so they
stay monotonic across eviction, reloads and crashes.
Run history is capped per job by evicting the oldest terminal runs,
and is kept as a tombstone when its job is removed unless purged.
Dropped run records are reported to the owner, which removes their
containers.
A terminal run is immutable except for recording that its container
was removed.
Encoded records are bounded below the point's value limit, with
headroom for later state transitions.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Define the org.mobyproject.extension.jobs.api.v0 extension point: a
single-cardinality Jobs service with unary RPCs for registration,
execution, inspection, lifecycle, run history and wait.
The container definition travels as a container-create JSON body.
A trigger declaring no known kind is rejected, so a spec using a newer
trigger kind fails on an older daemon instead of registering as a
manual job.
Run logs are not proxied; consumers read the run container's logs.
Run history outlives its job by default, and a run container lives as
long as its run record.
Generated wire code is produced by mobyextgen (go generate
./extpoints/...) and committed.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Allow extensions to offer typed points through servicev0.Offer and leave
gRPC registration to the host.
The server registration list is empty for now; no points are offerable
yet.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
The containerd image store is not supported with userns-remap, so the
daemon disables the containerd-snapshotter feature and refuses to start
when it is explicitly enabled.
The containerd-migration feature was not covered by this, so a daemon
with both userns-remap and containerd-migration still switched to the
containerd image store.
Refuse to start in that case too, with the same kind of error as for
containerd-snapshotter.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Replace the CODECOV_TOKEN secret with GitHub Actions OIDC authentication.
Codecov supports tokenless uploads for public repositories, but
authenticated uploads are still needed to protect coverage results on
protected branches from unauthorized uploads.
Grant `id-token: write` at the job level to allow Codecov to request
a short-lived OIDC token. This permission does not grant repository
write access, and scoping it to the job avoids granting it to other
jobs added to the workflow in the future.
Coverage uploads from forked pull requests continue to use Codecov's
tokenless upload mechanism.
See https://github.com/codecov/codecov-action/blob/v7.1.1/README.md#using-oidc
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The getController method only returns the network's controller pointer,
without performing any additional operations.
Replace its callers with direct access to n.ctrlr and remove the
unnecessary accessor. Rename the ctrlr field to "controller" for
clarity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The controller pointer is initialized when a network is created and
is not modified afterward. Reading it therefore does not require
holding the network mutex.
Remove the unnecessary locking from getController.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Every daemon with an extension host opened an extra full-access API
socket at startup, even when no extension used the API.
The socket is now created when an extension first connects to the API.
The daemonapi client looks up the endpoint on its first connection, so
NewClient no longer takes a context.
The socket is served separately from the configured hosts, returns 503
until the daemon is ready, and closes after extensions shut down.
It is accessible only to the daemon user, or to Administrators and
LocalSystem on Windows.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Extensions had no supported way to call the Engine API.
The configured hosts may be socket-activated or TLS-only TCP.
Add the org.mobyproject.extension.daemon.api.v0 point.
It returns the address of a private unix socket where the daemon serves
its regular API handler.
daemonapiv0.NewClient helper resolves the point and returns a
*client.Client.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add a test-surface specification for the overlay network driver and the
Swarm service mesh: what each part is expected to do, as observed from
outside the daemon. Each expectation carries a tier for what a failing
test means, the vantage point it is observed from, and what its verdict
depends on: the operating system, a package's upgrade path, the docker
CLI, or none of them. The spec is a model of the daemon's behaviour.
Which expectations a test suite checks, and where, is the suite's
choice.
An expectation holds on a node in its baseline state unless it says
otherwise. For a daemon installed from a package, the baseline is a
clean install of a supported OS, a fresh install of the package, and
the setup that the product's installation documentation prescribes.
For any other daemon, it is a host that meets the host requirements
that each spec lists: the kernel facilities, programs and host
configuration that its expectations rely on.
It describes behaviour only, so it should change when the daemon's
behaviour does, or when a regression shows that an expectation was only
assumed.
Expectations that came up along the way but are not Swarm's go in stubs
beside it, so the Swarm spec can refer to them without owning them.
bridge-networking.md covers the bridge driver and the parts of the
daemon's firewall integration that are not specific to Swarm,
DOCKER-USER among them. ipvlan-networking.md covers the ipvlan driver.
container-dns.md covers name resolution that does not depend on the
network driver. network-common.md covers what every network driver must
do, and the Engine API requests for networks and endpoints. The
conventions all five follow are in a README.md of their own, so that a
new spec can use them without editing an existing one. The specs live
in daemon/libnetwork/docs/spec.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When the daemon started with the nftables backend after a release older
than 28.0.0 had run with iptables, NewCleaner deleted the jumps from
filter-FORWARD to DOCKER-FORWARD and DOCKER-ISOLATION-STAGE-1, then
removed Docker's chains. Those releases also jumped from FORWARD to the
filter table's DOCKER chain for each bridge ("-o docker0 -j DOCKER").
iptables refuses to delete a chain that a rule still jumps to, and
removeIPChains ignores the error. The cleaner's DelNetwork deleted the
per-bridge jump later, when the bridge driver restored the network, but
nothing tried to remove the chain again. The empty DOCKER chain
remained.
When the bridge driver has restored its networks and endpoints, call
the cleaner's new Done method, which removes the chains again.
Signed-off-by: Cory Snider <csnider@mirantis.com>
When the daemon started with the nftables backend after a release older
than 28.0.1 had run with iptables, the iptables cleaner's DelNetwork
left the per-bridge rules those releases put in filter-FORWARD:
"-i docker0 ! -o docker0 -j ACCEPT" and the ICC rules. The iptables
backend deleted them in setupNonInternalNetworkRules and setIcc, but
only when adding a network's rules, and DelNetwork returned early for an
internal network.
Delete all the legacy per-bridge FORWARD rules in
deleteLegacyFilterRules, which the iptables backend and the cleaner call
for every network.
With the iptables backend, setupIPTables deletes the legacy rules after
calling setupNonInternalNetworkRules, which used to delete
"-i docker0 ! -o docker0 -j ACCEPT" before adding the DOCKER-FORWARD
rule that replaces it. Now, for a moment, both rules are present. Both
accept outgoing packets from the bridge.
Signed-off-by: Cory Snider <csnider@mirantis.com>
matchesType tests a network against only the first value of the type
filter. The values come from a map. Their order is random and changes
from one network to the next. A GET /networks request that filters on
both type=builtin and type=custom lists a random subset of the
networks. Before networks were filtered one at a time, the daemon
filtered the list once for each type value and listed every network
that matched any of them.
Match a network when its type is any of the filter's values.
integration-cli's TestDockerNetworkLsFilter checks this request but is
skipped in every run. Its OnlyDefaultNetworks requirement is met only
when GET /networks lists no networks at all. The daemon always lists
the predefined bridge, host and none networks. Port the test to
integration/network as TestNetworkListFilters, which uses the API
client. The new test checks the same id, name, type, label and driver
filter terms against two networks that it creates and the predefined
networks. It derives the expected results for type=custom and
driver=bridge from the unfiltered list, as the daemon can have other
networks. Remove the integration-cli test, its assertNwList helper and
the OnlyDefaultNetworks requirement. Nothing else uses them.
Add the combined type values to the filter unit test as well. Both
tests fail on the combined values without the fix.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Remove the redundant ErrorCoder type assertion and let the
recursive call handle unsupported error types.
Make all switch cases return explicitly.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add missing mappings for ErrFailedPrecondition (412) and
ErrResourceExhausted (429), aligning with containerd's errhttp
implementation.
Add table-driven tests covering the errdefs mappings, gRPC and
distribution errors, and wrapped and joined errors.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use errors.AsType to preserve the original JSON error details
when the error is wrapped, instead of replacing them with a
generic error containing only the message.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This error response was added in [moby@1421293], at which time the response
only contained a flat `error` field. This field was deprecated in favor of
an `errorDetails` struct in [moby@3043c26], but failed to update `FormatError`,
which continued to use the deprecated
`error` field.
Update `FormatError` to use the non-deprecated field, and update the error
message to be more descriptive. We're intentionally not preserving the actual
JSON marshal error, as it should be very unlikely to happen, and quoting the
error-message is not guaranteed to produce a JSON-compatible format.
[moby@1421293]: https://github.com/moby/moby/commit/14212930e40439d0cfc635d56e084736867f9e30
[moby@3043c26]: https://github.com/moby/moby/commit/3043c2641990d94298c6377b7ef14709263a4709
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
