A restart after two or more reschedules replayed an occurrence that a
manual run had replaced: startup kept a persisted future occurrence
only when it was the one right after the upcoming occurrence.
Keep any persisted future occurrence when re-arming a schedule.
This also trusts an occurrence persisted while the clock ran ahead,
which cannot be told apart from repeated reschedules.
The test now reschedules through manual runs and a store reload.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Replace the fake clock injected into the manager with testing/synctest
bubbles, removing the test-only hook from production code.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the builtin org.mobyproject.jobs.v0 extension, serving the Jobs API
on the daemon socket.
It is experimental and disabled by default; enable it with
extensions.enable (--extension-enable).
Run containers are driven through an Engine API client from the daemon
API point, and their exits observed through the container state point,
so restart-policy attempts do not count as exits.
Jobs persist through the storage point, and unnamed jobs are named
through the name-generator point.
Removing a run container also removes its anonymous volumes.
The extension host initializes before the daemon serves its API, so
activation reconciles runs and starts the scheduler in the background,
and the Jobs API answers Unavailable until then.
Shutdown releases blocked calls such as Wait, and waits for activation
and in-flight calls only until its deadline.
Authorization plugins are not supported yet, as the README states.
Integration tests cover a manual run, reconciliation after a killed
daemon, and the disabled extension.
A README next to the contract describes how to use it.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Resolve the runs an unclean stop leaves behind, after a crash or a
live-restore shutdown, before the scheduler re-arms.
A run whose start was never recorded is failed, as its outcome cannot
be trusted; a container it left behind is stopped first, so it cannot
overlap the job's next run; one whose ID was never recorded is found
by its name and removed.
A started run is re-attached to the exit watcher, which records an
exit from the downtime before enforcing a timeout re-armed from the
original start.
A job stuck running with a terminal run is returned to idle.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the lifecycle core of the jobs extension.
Registration is idempotent on a canonical spec hash, so formatting and
defaults do not change a job's identity, and a spec from a newer
contract fails validation on an older daemon.
Create-and-run fires the first run before anything else can reach the
new job.
Container specs with a negative StopTimeout are rejected, as runs must
be stoppable, and so is the host-dependent Local timezone.
Every fire goes through one chokepoint serialized per job, which
applies the concurrency policy and persists the run record before the
container is created, so a crash always leaves a discoverable run.
A fire is detached from its caller, as the daemon completes a
container start even when the request goes away.
A run whose start is unknown, because the start request failed or its
record could not be written, has its container stopped and keeps the
job occupied until the container exits.
Outcomes are persisted before auto-removal, and failed terminal or
idle writes are retried instead of leaving the job stuck; only the
job's latest run returns it to idle.
Cancels and timeouts always stop the container, even after the other
decided the outcome, so a failed stop is not final.
The scheduler is one goroutine over a min-heap, dispatching through
the same chokepoint.
On start, missed occurrences follow the missed-fires policy with at
most one catch-up, and an occurrence moved by a reschedule is kept.
A reschedule replaces exactly one occurrence, including one a tick
already consumed but has not fired yet.
A dispatched fire is dropped if its job was paused, resumed or removed
since, and the next-fire time is persisted from the scheduler's
current state after the fire is recorded, so a delayed fire cannot undo
a newer change and a crash cannot lose the occurrence.
A fire deferred by the queue policy keeps its occurrence recorded until
it runs, so a restart in between catches it up.
Removing a job keeps its run history by default, reachable by the
removed job's ID, and run containers are removed with their records.
Stored specs are compared as the running daemon canonicalizes them, so
jobs survive upgrades that change the container API types.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the five-field cron grammar of schedule triggers: numbers, lists,
ranges and steps.
Names, shortcuts and Quartz extensions are left to clients.
Day-of-month and day-of-week combine following Vixie's rule.
Occurrences follow the wall clock of the job's timezone: a time
skipped by a DST transition does not fire that day, and a repeated
one fires once.
The search is bounded by the 400-year Gregorian cycle, so impossible
dates never fire while sparse ones such as February 29 on a Sunday
still do.
Zone data is embedded, so timezones resolve on hosts without a zoneinfo
database, such as Windows.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Add the persistence layer of the jobs extension on top of the storage
point, with one record per job and per run.
Records are versioned, and damaged ones are quarantined at load with a
warning instead of failing the daemon; a backend that cannot list or
read records fails the load.
Iteration numbers are persisted before the runs they number, so they
stay monotonic across eviction, reloads and crashes.
Run history is capped per job by evicting the oldest terminal runs,
and is kept as a tombstone when its job is removed unless purged.
Dropped run records are reported to the owner, which removes their
containers.
A terminal run is immutable except for recording that its container
was removed.
Encoded records are bounded below the point's value limit, with
headroom for later state transitions.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Define the org.mobyproject.extension.jobs.api.v0 extension point: a
single-cardinality Jobs service with unary RPCs for registration,
execution, inspection, lifecycle, run history and wait.
The container definition travels as a container-create JSON body.
A trigger declaring no known kind is rejected, so a spec using a newer
trigger kind fails on an older daemon instead of registering as a
manual job.
Run logs are not proxied; consumers read the run container's logs.
Run history outlives its job by default, and a run container lives as
long as its run record.
Generated wire code is produced by mobyextgen (go generate
./extpoints/...) and committed.
Co-authored-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Guillaume Lours <glours@users.noreply.github.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Load containerd.d/*.toml next to the daemon configuration file when
using embedded containerd.
Only the plugins, disabled_plugins, and required_plugins settings
apply.
Require version 4 or later and reject nested imports, so the merge
order follows the lexical order of the files.
Warn when drop-ins are present but dockerd uses a system or managed
containerd, which ignore them.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
When the host has no valid external nameservers (or /etc/resolv.conf is
missing/dangling), TransformForIntNS previously left ExtNameServers empty
and appended 'NO EXTERNAL NAMESERVERS DEFINED'. This caused the embedded
DNS resolver (127.0.0.11) in user-defined networks to have no upstream
nameservers, causing external queries to fail with SERVFAIL.
As already documented in TransformForIntNS:
'If no ExtNameServer addresses are found, use the defaults.'
Fallback to host loopback addresses (127.0.0.1, ::1) with HostLoopback: true,
matching host libc resolver fallback behavior when resolv.conf is missing.
Signed-off-by: arsalanyavari <arya48.yavari79@gmail.com>
TestBuildNetContainer failed with an empty /otherhost instead of the
expected "foobar" hostname on CI hosts with newer AppArmor policy.
The listener ran "nc -ll -p 1234 -e hostname".
With the stacked docker-default//&unconfined confinement seen in CI,
exec'ing /bin/hostname attached the host's "hostname" AppArmor profile,
which denied writing to the inherited socket:
```
apparmor="DENIED" operation="file_inherit" class="net" info="failed type match" error=-13 profile="hostname" comm="hostname" ... requested="send receive" denied="send receive"
```
The client then saw EOF and exited successfully with no output.
Move the test to integration/build as TestBuildNetworkModeContainer and
reply with shell builtins ("echo $HOSTNAME") so no binary named
hostname is exec'd.
Also wait until the listener answers with the expected hostname before
starting the build, instead of relying on a single connection attempt.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
The driver creates one network sandbox for each overlay network, and
initSubnetSandbox and setupSubnetSandbox add a subnet's bridge and VXLAN
link to it. Their names, and the "subnet sandbox join failed" error,
read as though each subnet had a sandbox of its own. Rename them to
initSubnet and setupSubnet, rename the subnet's sboxInit flag to
initDone, and report the error as a failure to join the network sandbox
for the subnet.
Signed-off-by: Cory Snider <csnider@mirantis.com>
The overlay driver kept the network namespaces of its network sandboxes
in the netns directory it was given, which the daemon also keeps the
namespaces of the controller's sandboxes in, and told them apart by the
"-" in their names. It cleaned up the network sandboxes an earlier
daemon left behind lazily. On the first join of any network's sandbox it
recorded the VNIs of their VXLAN links, so that it could destroy a stale
sandbox holding a VNI another network needed, and it destroyed a
network's stale sandboxes when the network was next initialized.
None of those sandboxes can be in use. A node cannot be in a Swarm with
live-restore enabled, so no container attached to an overlay network
outlives the daemon. Keep the network sandboxes in an "overlay"
subdirectory of the netns directory, and destroy everything in it when
the driver is first configured, along with the sandboxes earlier daemons
kept in the netns directory itself. Delete each stale sandbox's VXLAN
links before unmounting it: the kernel deletes them when it tears the
namespace down, but it does so asynchronously, and a new VXLAN link
cannot use their VNIs until it has.
A network's sandbox is destroyed before the network can be initialized
again, so the sandbox's key does not need the network's initialization
epoch to be unique. Name it "0-<network ID>". Earlier versions of the
driver walk the whole netns directory, the "overlay" subdirectory
included, and take a namespace with a "-" in its name to be a network
sandbox. They destroy it when a network needs one of its VNIs, or when
they initialize the network whose ID follows the "-", so they can still
clean up the sandboxes a newer daemon leaves behind. Their epochs start
at 1, so "0-" cannot clash with their names.
Signed-off-by: Cory Snider <csnider@mirantis.com>
A task on the ingress network needs rules in its own network namespace
that redirect each published port to its target port and filter all
other traffic addressed to its ingress endpoint. Adding them can fail,
for example when the kernel lacks a feature the rules use.
populateLoadBalancers logged that failure and let the join succeed. The
task then ran, and was reported as running, without serving its
published ports and with every port it listens on reachable from the
rest of the ingress network. The only trace was the daemon log.
Return the error instead, so the join fails and the task fails with the
reason. This applies to both firewall backends.
Signed-off-by: Cory Snider <csnider@mirantis.com>
Allow extensions to offer typed points through servicev0.Offer and leave
gRPC registration to the host.
The server registration list is empty for now; no points are offerable
yet.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Regenerate the protobuf bindings using Go 1.27.
The descriptor contents are unchanged. The generated diffs are limited to
the gzip-compressed raw descriptors embedded in the generated Go files.
Go 1.27 changed the `compress/flate` encoder to improve compression
performance. In this case, the new encoder marks the preceding DEFLATE block
as final instead of appending an empty final block, making each compressed
descriptor three or four bytes smaller.
Go 1.27 explicitly allows the exact encoded output to differ as a result of
this encoder change:
https://go.dev/doc/go1.27#compressflatepkgcompressflate
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The containerd image store is not supported with userns-remap, so the
daemon disables the containerd-snapshotter feature and refuses to start
when it is explicitly enabled.
The containerd-migration feature was not covered by this, so a daemon
with both userns-remap and containerd-migration still switched to the
containerd image store.
Refuse to start in that case too, with the same kind of error as for
containerd-snapshotter.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Replace the CODECOV_TOKEN secret with GitHub Actions OIDC authentication.
Codecov supports tokenless uploads for public repositories, but
authenticated uploads are still needed to protect coverage results on
protected branches from unauthorized uploads.
Grant `id-token: write` at the job level to allow Codecov to request
a short-lived OIDC token. This permission does not grant repository
write access, and scoping it to the job avoids granting it to other
jobs added to the workflow in the future.
Coverage uploads from forked pull requests continue to use Codecov's
tokenless upload mechanism.
See https://github.com/codecov/codecov-action/blob/v7.1.1/README.md#using-oidc
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The getController method only returns the network's controller pointer,
without performing any additional operations.
Replace its callers with direct access to n.ctrlr and remove the
unnecessary accessor. Rename the ctrlr field to "controller" for
clarity.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The controller pointer is initialized when a network is created and
is not modified afterward. Reading it therefore does not require
holding the network mutex.
Remove the unnecessary locking from getController.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Every daemon with an extension host opened an extra full-access API
socket at startup, even when no extension used the API.
The socket is now created when an extension first connects to the API.
The daemonapi client looks up the endpoint on its first connection, so
NewClient no longer takes a context.
The socket is served separately from the configured hosts, returns 503
until the daemon is ready, and closes after extensions shut down.
It is accessible only to the daemon user, or to Administrators and
LocalSystem on Windows.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Extensions had no supported way to call the Engine API.
The configured hosts may be socket-activated or TLS-only TCP.
Add the org.mobyproject.extension.daemon.api.v0 point.
It returns the address of a private unix socket where the daemon serves
its regular API handler.
daemonapiv0.NewClient helper resolves the point and returns a
*client.Client.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Paweł Gronowski