This makes
- UDP socket fd is owned by IO event source,
- open RAW socket fd just before sending first packet,
- set TOS and socket priority,
- use AF_UNIX soxket pair in the unit test and fuzzer, so the unit test
can now run by unprivileged user.
load_kexec_kernel() retries kexec_file_load() with an extracted kernel
(decompressed Image / ZBOOT PE / UKI) when the kernel rejects the image,
but it only does so when kexec_file_load() failed with ENOEXEC. On arm64
that retry never happens: arm64's image_probe()
(arch/arm64/kernel/kexec_image.c) returns -EINVAL on an ARM64_IMAGE_MAGIC
mismatch, whereas x86's bzImage64_probe() and the generic
kexec_image_probe_default() return -ENOEXEC. So `systemctl kexec` of a
UKI on arm64 skips the extraction path and falls back to the
/usr/sbin/kexec binary, which is no longer a dependency since e107c7ead0
("systemctl: replace kexec-tools dependency with direct kexec_file_load()
syscall") -- leaving kexec broken.
Accept EINVAL in addition to ENOEXEC. This is safe: the extraction in
kexec_maybe_decompress_kernel() re-gates on the actual file magic (MZ /
compression headers) and is a no-op returning 0 for anything else, so an
EINVAL that is not a format mismatch just falls through to the existing
fallback as before.
Fixing this in systemd (rather than only in the kernel) is appropriate:
systemd must keep working with already-shipped arm64 kernels whose
kexec_file_load() returns EINVAL for an unrecognized image magic.
Relates to: https://github.com/systemd/systemd/issues/28538
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
nvindex space is very scarce, let's hence introduce "priorities" on
nvpcrs, so that if we haven't got enough space for all we have some
control which ones get dropped and which ones kept.
Regression coverage for the soft-reboot linger bug fixed in
9f25feb4ed ("logind: keep lingering users at startup-time GC", #41789):
a lingering user's user@.service was started after a hardware reboot but
not after `systemctl soft-reboot`, because logind GC'd the user at
startup before user_start() ran.
On the first boot, enable lingering for the pre-existing testuser and
wait for its user@UID.service to come up. After the first soft-reboot
(which keeps the same persistent rootfs), assert the service is active
again — this is the regression check. Disable lingering again before the
nextroot switch in the second boot so the lingering user doesn't leak
into the later boots' minimal overlay rootfs.
Verified locally in a qemu/KVM VM: passes with the fix present, and with
the fix reverted the second-boot assertion times out (the lingering user
is GC'd in logind startup and user@UID.service never comes back),
confirming the test exercises the bug.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Traditionally, asynchronous programming in systemd has been achieved
using
sd-event along with the asynchronous interfaces of sd-bus and
sd-varlink.
This works well when the system is reacting to events and all code
triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback
can
use the stack as usual, declaring local state and ensuring proper
cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block
execution
within the callback, we can't directly invoke a long-running operation
and
wait for its result without introducing complexities. Instead, we need
to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along
as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope
may
be prematurely cleaned up. Therefore, the struct must store copies of
the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a
loop,
the complexity grows further. We must introduce additional shared state
to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must
track
the lifecycle of each long-running operation in the global Manager
struct,
ensuring proper cleanup even when stack unwinding can no longer manage
the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s
running
synchronously, but with the ability to yield control at predefined
points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while
waiting
for the long-running operation to finish and then "resume" the operation
once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in
a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve
the
handling of complex workflows, making the code easier to write and
maintain.
The implementation is based on ucontext.h's makecontext() (with a
fallback
to the venerable sigaltstack() approach on musl),
sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can
switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext()
because the
latter forcibly saves/restores a per context signal mask every time it
is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and
maintain
a per thread signal mask, which makes much more sense than having a per
fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used
until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a
deferred
event source which resumes the fiber when enabled. The deferred event
source
is oneshot by default so the fiber will run immediately until it yields
or
suspends. If it yields, the deferred event source is enabled again
(oneshot)
immediately. If it suspends, before it suspends, one or more event
sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for
completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or
any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is
cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the
fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup
to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them
behind
a generic futures interface to represent long-running operations,
regardless of
whether those operations are running on a fiber or not. Aside from
fibers, the
futures library (sd-future) will for example allow waiting for sd-event
sources
and doing sd-bus calls in the background as well. Fibers can suspend
until a
future is ready with sd_fiber_await() or by having the future wake up
the fiber
explicitly in its callback. A future always defaults to waking up the
current
fiber.
Each future kind plugs into the library by providing an sd_future_ops
vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it
via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to
that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from
a
fiber via sd_fiber_await() (synchronous-looking fiber code).
sd_fiber_await()
is itself built on a "wait future" that resolves when its target
resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber
callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future
impl's
cancel callback, which is responsible for tearing down its work and
ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from
sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new()
—
take a self-reference on their future so they outlive the caller's
scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds
would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal
sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an
unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the
following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of
it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
We include the header through src/include/override/fcntl.h, hence we should
have the latest copy of the header.
(E.g. compilation fails with musl-libc-1.2.5-6.fc44.x86_64 and older kernel
headers.)
Currently with FileDescriptorStorePreserve=yes the FD store is kept
around
regardless of what happens to a unit, which is useful in many cases. But
in
some cases, for example when complex services crash horribly, it's hard
to
reason about what was in the intermediate state, and it's better to
start
fresh.
Add a new 'on-success' option for the FileDescriptorStorePreserve=
setting
that keeps it around only for as long as the unit doesn't go to a
persistently
failed state.
This is especially useful in combination with LUO, where we don't want
to
keep around LUO sessions created by units that then proceeded to crash
and
burn, and might be in a bad state afterwards.
On 32-bit userspace running against a 64-bit kernel
BTRFS_IOC_GET_SUBVOL_INFO returns -ENOTTY: struct
btrfs_ioctl_get_subvol_info_args embeds four btrfs_ioctl_timespec
values, and that timespec struct (__u64 sec; __u32 nsec) packs to 12
bytes on i386 but 16 on x86_64 due to differing __u64 alignment.
sizeof(struct) is part of the ioctl cmd number via _IOR(), so the cmd
emitted by 32-bit userspace doesn't match the case label compiled by
the 64-bit kernel and the switch falls through to -ENOTTY.
btrfs already handles this exact class of bug for
BTRFS_IOC_SET_RECEIVED_SUBVOL via a btrfs_ioctl_timespec_32 struct
plus a _32 cmd alias in fs/btrfs/ioctl.c, but GET_SUBVOL_INFO (added
in 2018, four years after that fix) didn't get the same treatment.
Until a kernel patch lands the test can't exercise the ioctl on
32-bit, so convert TEST(info) to TEST_RET(info) and return
EXIT_TEST_SKIP with a clear message when -ENOTTY comes back. The
other tests in the file use ioctls that already have working compat
paths and remain unaffected.
Unfortunately, previously the path to test-ndisc-send has been wrong, so
some test cases have not been checked in our mkosi CIs. And two test
cases have been broken.
The test case `test_ndisc_redirect` was not updated when the logic in
networkd was changed by 9142bd5a8e. The
change itself should be OK. So, the test case is updated.
The test case `test_ndisc_mtu` was broken when the commit
32417c1723 is merged. The commit is not
correct, as we cannot set IPv6 MTU larger than interface MTU. So, the
offending commit is reverted.
core: add User,Group,SupplementaryGroups,Nice to varlink
Unit.StartTransient
This commit adds more writable fields to the
io.systemd.Unit.StartTransient
varlink method. With this its possible to set:
User,Group,SupplementaryGroups,Nice values.
Plus tests for them.
---
core: report unsupported service fields in varlink calls
Just like for the unsupported/bad exec_fields we should show
a message about what field is bad for service parameters. This
commit adds it using the same pattern. The JSON parser works in
fail-fast mode so we only display the first bad field (and
it depends on the parser what it finds first).
Add -fexcess-precision=standard so gcc inserts ISO C99 conformant
rounding at assignments, casts, and returns — without it, double values
on x87 happily stay at 80-bit extended precision across operations and
diverge from the SSE/x86_64 behavior, making strict equality comparisons
architecture-dependent.
The flag doesn't fully cover x87: per gcc PR#323
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=323), a function return
value carried in ST(0) can arrive at the caller still at 80-bit, so a
double that ought to compare equal to a same-magnitude literal picks up
extra mantissa bits and doesn't. Wrap fp_equal in volatile-double
temporaries to force a memory roundtrip — the only operation that
reliably truncates on x87 — so its callers get consistent results
regardless of how the operands were produced.
Add a TEST(fp_equal) case that exercises the previously-broken pattern:
a runtime 1.0/10.0 computed inside a noinline helper, returned across
the function ABI boundary, then compared against the literal 0.1.
Without the volatile truncation this assertion fails on 32-bit gcc.
The mock servers used to be driven out-of-band: each test created a
socketpair, forked a child, ran a hand-coded request/response script
against the raw fd, and sent SIGTERM to tear it down. That worked but
required pidref/process-util/signal plumbing in every test, two
distinct execution contexts that couldn't share state, and a JsonStream
attached to the mock side that pretended to be event-loop-driven while
actually being driven manually via blocking reads.
Now that JsonStream suspends when on a fiber, the mocks can live
inside the same process and event loop as the client. Each mock is
rewritten as an sd-fiber that runs alongside the client fiber: so the
mock fiber yields on I/O and the event loop schedules the client in the
meantime. Both sides progress cooperatively, no fork/SIGTERM/PID tracking,
no manual phase tracking.
Two cleanups fall out of the rewrite:
- A QMP_TEST(name, mock_fn) { ... } macro encapsulates the per-test
scaffolding (event loop, socketpair, mock fiber spawn, exit-on-idle
shim) and injects an already-connected QmpClient *client into the
test body. Each test now reads as a flat sequence of
qmp_client_call() invocations against that client.
- Repeated mock command/reply scripting is factored into
mock_qmp_expect(), mock_qmp_reply(), mock_qmp_expect_and_reply(),
mock_qmp_handshake(), and mock_qmp_query_status_running(). The
greeting JSON is built with sd_json_buildo() instead of being parsed
from a literal.
The file shrinks from 756 to 494 lines, mostly through deletions.
The synchronous qmp_client_call() pumps the event loop until its reply
arrives, pinning the parsed reply on c->current so it can hand out
borrowed pointers to the caller. That model only fits one in-flight
sync call: a second qmp_client_call() on the same client clears
c->current before issuing its own send, invalidating the first
caller's borrowed pointers. On a single-threaded event loop that was
fine, but with fibers two concurrent calls on the same client can
interleave through the pump (json_stream_wait() suspends the running
fiber) and trample each other.
To fix this, make qmp_client_call() detect when it's running on a fiber
whose event loop matches the client and transparently delegate to
qmp_client_call_suspend(), which makes use of a new QmpFuture to allow
multiple concurrent calls to qmp_client_call().
To make this work concurrently, we also change qmp_client_call() to
hand out references and copies of errors so that we don't have to store
the borrowed pointers we hand out in the QmpClient struct.
Add varlink_server_bind_fiber() and varlink_server_bind_fiber_many()
in varlink-util.{c,h} for registering a method handler that should
run on a dedicated fiber per dispatch. The fiber-bound methods live
in a separate s->fiber_methods map alongside the regular s->methods;
bind_internal()/bind_many_internal() are factored out so the regular
and fiber bind variants share their parsing/insertion code.
Registering the same method in both maps is rejected because the
dispatcher consults the regular map first and would otherwise
silently shadow the fiber binding.
varlink_dispatch_fiber() builds a VarlinkFiberData (refs to the
connection, parameters, and method name), spawns a fiber via
sd_fiber_new(), and makes the future floating so the fiber
self-manages its lifetime — neither the dispatcher nor the
connection has to track it. The fiber's priority is set to one
below the connection's quit event source so that on graceful
shutdown the fiber's exit handler fires (and runs its cleanup)
before varlink's quit_callback() closes the connection underneath
it; this is what lets a fiber-bound handler reply or flush its
sentinel on a still-open connection during shutdown.
The connection state transitions are reordered so they happen before
the fiber spawn rather than after the synchronous callback returns:
the fiber runs after dispatch has already moved past PROCESSING, which
matches the behaviour expected for a deferred reply (the fiber may
either reply immediately, or stash the connection and reply later, in
which case the post-callback logic treats it as a PENDING_METHOD).
Note that all the synchronous varlink APIs (sd_varlink_call() and friends)
already behave properly when on a fiber because they call json_stream_wait()
which calls ppoll_usec() which we already fixed to suspend when called from
a fiber.
The client/server varlink tests are migrated to fibers (threads → mock
server fibers on the same event loop) to exercise the new paths.
Two changes to teach sd-bus how to behave when called from a fiber, in
order of increasing depth:
2. sd_bus_call() now redirects to a new bus_call_suspend() helper when
the caller is a fiber whose event loop is the same one the bus is
attached to. The plain bus_poll() path serializes all bus traffic on
the slot's reply (only one method call can be in flight per
sd_bus*), which would defeat the point of running multiple fibers
against one bus. bus_call_suspend() builds on the async sd-bus API:
it wraps the call in a new BusFuture (sd-bus/bus-future.{c,h}) that
resolves when the reply or method-error arrives, lets the fiber
await that future, and surfaces the reply to the caller via
future_get_bus_reply(). Because the futures live on the event loop
rather than a per-bus slot, multiple fibers can drive concurrent
method calls against the same bus.
3. A new private SD_BUS_VTABLE_METHOD_FIBER flag dispatches a vtable
method handler on its own fiber, so handlers are free to use
sd_bus_call() against the same bus, sd_fiber_sleep(), loop_read(),
etc. without stalling the event loop for other connections or
handlers. The flag stays out of sd-bus-vtable.h (its bit value is
reserved there to prevent collisions) — the fiber runtime is a
systemd-internal implementation detail.
Lifecycle of fiber-dispatched handlers is tracked on the bus itself: a
new bus->fiber_futures set holds a ref to each in-flight handler.
bus_enter_closing() cancels every entry and process_closing() returns
with the bus still in CLOSING state until the set drains, so we can be
sure no fiber handler outlives the bus. bus_fiber_resolved() removes
the entry on completion. bus_free()'s assert(set_isempty()) makes the
invariant load-bearing.
Note that plain sd_bus_call() already works correctly on a fiber as it
calls ppoll_usec() which has already been modified to suspend when
running on a fiber.
To exercise these changes the existing thread-based client/server
sd-bus tests (test-bus-chat, test-bus-objects, test-bus-peersockaddr,
test-bus-server, test-bus-watch-bind) are migrated to fibers, and a
new test-bus-fiber is added that covers SD_BUS_VTABLE_METHOD_FIBER —
including handlers that issue nested sd_bus_call() on the same bus, the
cancel-on-close path, and concurrent dispatches across multiple fibers.
sd_event_run() blocks the calling thread on the event loop's epoll fd
until something happens. When the caller is a fiber, that's the wrong
behaviour: blocking the thread also stalls every other fiber and the
outer event loop driving them. The most common way to hit this is a
fiber that creates its own inner event loop (e.g. a server-style fiber
that wants to dispatch its own sources independently of whatever loop
the test or supervising fiber is running on) — with the existing
implementation the inner sd_event_run() would hold the thread while the
outer scheduler should be free to advance other fibers.
Add an event_run_suspend() variant in sd-event/event-future.c that
performs the same prepare/wait/dispatch dance, but when the fast path
finds nothing ready it (a) creates an IO future watching the inner
event loop's epoll fd on the *outer* event loop, (b) optionally creates
a time future for the timeout, and (c) suspends the fiber. When either
future fires the fiber is resumed and the prepare/wait/dispatch sequence
runs once more to actually dispatch what's pending. sd_event_run()
checks sd_fiber_is_running() and delegates to this variant when on a
fiber; profile_delays accounting is intentionally skipped on that path
since the underlying prepare/wait/dispatch primitives already account
for themselves.
Some helpers in src/basic — ppoll_usec_full() (used by fd_wait_for_event()),
loop_read(), loop_read_exact(), loop_write_full() and
pidref_wait_for_terminate_full() — block the calling thread. That's the
right behaviour outside a fiber but not inside one, where blocking the
thread also stalls every other fiber running on the same event loop.
Rewriting every caller to pick a fiber or non-fiber variant explicitly
would be a lot of churn and would split otherwise-shared code paths in
two.
Instead, the helpers detect at runtime whether they're running on a fiber
and dispatch to a suspending variant when they are. FiberOps in
fiber-ops.h holds five function pointers (ppoll, read, write, timeout,
cancel_wait_unref); a fiber_ops global constant is populated whenever we
enter a fiber with functions that delegate to suspending variants of common
syscalls. With this approach, the variants themselves stay in libsystemd
which is required because they make use of sd-event.
- loop_read()/loop_read_exact() take the fiber read hook on a fiber
unless the caller asked for a non-blocking attempt (do_poll=false) and
the fd is already non-blocking — in that case we fall through to read()
to preserve the existing return-EAGAIN-immediately semantic. The hook
itself suspends on EAGAIN until data is available, so neither the
do_poll knob nor the explicit fd_wait_for_event() retry loop are
needed on the fiber path.
- loop_write_full() likewise takes the fiber write hook on a fiber,
except when timeout=0 with an already-non-blocking fd (preserving the
fast-return-EAGAIN semantic). The fiber path runs inside a
FIBER_OPS_TIMEOUT() scope so the caller's timeout is honoured via a
deadline future, mirroring SD_FIBER_TIMEOUT() but reachable from
src/basic without pulling in sd-future.h.
- pidref_wait_for_terminate_full() polls the pidfd via fd_wait_for_event()
before each waitid() when either a finite timeout is set or we're on a
fiber, and requires pidref->fd >= 0 in those cases (returning
-ENOMEDIUM otherwise — extending the rule that already applied to
finite timeouts). The poll suspends the fiber via the ppoll hook above;
the subsequent waitid() doesn't block because the pidfd is already
signalled.
Add a family of sd_fiber_*() I/O wrappers that, when called from a
fiber, behave like blocking I/O from the caller's perspective but
yield to the event loop instead of blocking the thread:
sd_fiber_read / sd_fiber_write
sd_fiber_readv / sd_fiber_writev
sd_fiber_recv / sd_fiber_send
sd_fiber_connect
sd_fiber_recvmsg / sd_fiber_sendmsg
sd_fiber_recvfrom / sd_fiber_sendto
sd_fiber_accept
sd_fiber_ppoll
Most of them share a single helper, fiber_io_operation(), which when
invoked outside a fiber falls through to the underlying syscall
directly, preserving the regular blocking behaviour. Inside a fiber
the helper flips the fd to non-blocking (restoring its original mode
on return), tries the syscall once on the fast path, and on EAGAIN/
EWOULDBLOCK creates an sd-event-backed IO future via future_new_io(),
suspends the fiber, and retries the syscall once the event source
fires.
future_new_io() itself is added to sd-event/event-future.{c,h} as a
new IoFuture kind. It wraps sd_event_add_io() into an sd_future:
oneshot enable, EPOLLERR translated via SO_ERROR (suppressed for
non-sockets), and the fd duplicated with F_DUPFD_CLOEXEC to avoid
EEXIST when multiple sources watch the same descriptor.
Together these let fiber-using code write straight-line socket and
pipe I/O without bundling state into callbacks.
Traditionally, asynchronous programming in systemd has been achieved using
sd-event along with the asynchronous interfaces of sd-bus and sd-varlink.
This works well when the system is reacting to events and all code triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback can
use the stack as usual, declaring local state and ensuring proper cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block execution
within the callback, we can't directly invoke a long-running operation and
wait for its result without introducing complexities. Instead, we need to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope may
be prematurely cleaned up. Therefore, the struct must store copies of the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a loop,
the complexity grows further. We must introduce additional shared state to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must track
the lifecycle of each long-running operation in the global Manager struct,
ensuring proper cleanup even when stack unwinding can no longer manage the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s running
synchronously, but with the ability to yield control at predefined points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while waiting
for the long-running operation to finish and then "resume" the operation once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve the
handling of complex workflows, making the code easier to write and maintain.
The implementation is based on ucontext.h's makecontext() (with a fallback
to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the
latter forcibly saves/restores a per context signal mask every time it is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain
a per thread signal mask, which makes much more sense than having a per fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a deferred
event source which resumes the fiber when enabled. The deferred event source
is oneshot by default so the fiber will run immediately until it yields or
suspends. If it yields, the deferred event source is enabled again (oneshot)
immediately. If it suspends, before it suspends, one or more event sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them behind
a generic futures interface to represent long-running operations, regardless of
whether those operations are running on a fiber or not. Aside from fibers, the
futures library (sd-future) will for example allow waiting for sd-event sources
and doing sd-bus calls in the background as well. Fibers can suspend until a
future is ready with sd_fiber_await() or by having the future wake up the fiber
explicitly in its callback. A future always defaults to waking up the current
fiber.
Each future kind plugs into the library by providing an sd_future_ops vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from a
fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await()
is itself built on a "wait future" that resolves when its target resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future impl's
cancel callback, which is responsible for tearing down its work and ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() —
take a self-reference on their future so they outlive the caller's scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
Currently with FileDescriptorStorePreserve=yes the FD store is kept around
regardless of what happens to a unit, which is useful in many cases. But in
some cases, for example when complex services crash horribly, it's hard to
reason about what was in the intermediate state, and it's better to start
fresh.
Add a new 'on-success' option for the FileDescriptorStorePreserve= setting
that keeps it around only for as long as the unit doesn't go to a persistently
failed state.
This is especially useful in combination with LUO, where we don't want to
keep around LUO sessions created by units that then proceeded to crash and
burn, and might be in a bad state afterwards.
When the tracked IPv6 link-local address is removed, networkd clears
link->ipv6ll_address, but keeps DHCPv6, NDisc, and RADV running. These
engines keep using a stale source identity which affects the following:
- DHCPv6 client continues to send Solicit/Renew/Rebind from a nonexistent
source address.
- NDisc continues to send Router Solicitations from a nonexistent source
address. Router Advertisements cannot be received properly.
- RADV continues to advertise with a stale source address. This can lead
to downstream hosts configuring invalid routes.
- DHCP-PD prefixes remain configured without a valid upstream DHCPv6 path.
Added link_ipv6ll_lost() to stop IPv6 dynamic engines and related states:
- sd_dhcp6_client_stop()
- ndisc_stop() + ndisc_flush()
- sd_radv_stop()
This is called from address_drop() when the dropped address matches the
tracked IPv6LL. After clearing the tracked address, it scans for another
ready link-local address on the interface. If found, this is set as
link->ipv6ll_address and link_ipv6ll_gained() is called to restart the
engines with the new source identity.
This resolves the FIXME in address_drop().
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Then, this deprecates
- BindToInterface=
- RelayTarget=
- RelayAgentCircuitId=
- RelayAgentRemoteId=
settings in [DHCPServer] section.
These are gracefully translated as new settings.
