This partially reverts 13ce62caba.
Sometimes, the kernel passes non-safe characters in uevent files,
such as the UNIQ= property for USB serial devices.
Instead of rejecting the entire uevent file, let's only ignore the
specific key-value pairs that contain non-safe characters.
Fixes#43008.
(cherry picked from commit 8eede7122f)
When no prefix is specified, portablectl inspect first tries the
prefix derived from the image name. This keeps inspect aligned with
attach behavior.
If that lookup finds no matching units, retry with validated
PORTABLE_PREFIXES read from the image os-release. This makes inspect
work for images whose filename does not match their portable service
prefix.
Keep metadata error handling explicit so request-construction failures
are not logged twice, while sd_bus_call() failures still include the
inspect context.
Add a TEST-29-PORTABLE regression case for a directory image whose
name does not match the portable service prefix.
Fixes#37296.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
(cherry picked from commit 63b30468a5)
When a rate limit is armed, for example via StartLimitInterval=, and the
machine is suspended, the rate limit's "clock" is suspended as well,
since the elapse checks use CLOCK_MONOTONIC. This then causes unexpected
situations where a rate limit armed via StartLimitInterval=1h, followed
by a 10 hour suspend, would elapse after 11 hours total.
Let's avoid this by switching the rate limits to CLOCK_BOOTTIME, which
works the same as CLOCK_MONOTONIC, but accounts for the time spent in
suspend as well.
There's one slight concern when it comes to upgrade path - the old
"begin" value of the rate limit is stored as CLOCK_MONOTONIC, but after
upgrading systemd and serializing/deserializing the state it will be
suddenly compared against now(CLOCK_BOOTTIME), which might cause some
rate limits to elapse "prematurely". But this is just a one-time thing,
after which the rate limit timers should re-assess themselves.
Resolves: #42912
(cherry picked from commit 9ae50c78cc)
add_object_vtable_internal() inserts BusVTableMember entries into the
global vtable_methods/vtable_properties sets as it walks the vtable, but
only sets node_vtable.node (which gates the disconnect-time cleanup)
after the loop. A failure mid-loop thus left the already-inserted
members behind, dangling once the slot's interface string and the node's
path were freed, so a later registration reading those keys hit a
use-after-free. Remove this slot's members on the fail path.
The test asserts the failed registration leaves no member behind.
Follow-up for 19befb2d5f
(cherry picked from commit b7b1303209)
The CAN_HASH branch of bus_match_run() that iterates the array values of
an argNhas= match dispatches a callback per value but, unlike the leaf
and manual-iteration branches, does not re-check bus->match_callbacks_modified
afterwards. A callback that adds or removes a match (freeing the node)
would then have the next hashmap_get(node->compare.children, ...) read
freed memory. Add the same check the sibling branches already have.
Follow-up for 198b158f49
(cherry picked from commit 058b0de7bf)
netlink_slot_allocate() links the slot into nl->slots and takes an nl
reference, but sd_netlink_call_async() and netlink_add_match_internal()
hold it in a _cleanup_free_, so an error after allocation frees the slot
with a bare free()
Follow-up for ee38400bba
(cherry picked from commit 25c016f472)
"properties_off + 32 > n" is evaluated in 32-bit arithmetic, so a
properties_off near UINT32_MAX wraps but passes the check.
Follow-up for efbd4b3ca8
(cherry picked from commit 7003a9bf09)
The tag-match loop emits 6 instructions per tag into a fixed 512-entry
stack array without a bounds check (unlike the subsystem loop below),
so enough tags overflow the stack. Refuse with -E2BIG instead.
Follow-up for b1c097af8d
(cherry picked from commit 2f72db10d9)
Some state files were prefixed with the machine name which protected
against collisions when running concurrent machines in a systemd unit
with RuntimeDirectory= set. However, the socket files tpm.sock and
control were not, which caused a race. Prefixing these socket files
does not work because the path gets too long when both the subdirectory
and the socket file within it get unique identifiers.
Instead of prefixing all files, we can rather always create a
subdirectory and use simple names within in. This makes paths shorter
in the normal case and protects against races with RuntimeDirectory=
were instead of directly reusing RUNTIME_DIRECTORY we also create the
normal per-machine subdirectory. Since this here is about runtime state
it should not impact any running VMs/containers. The ssh-proxy looks
only at the normal case and does not support RUNTIME_DIRECTORY, so no
impact there as well.
(cherry picked from commit ff30101e81)
journal_file_read_tail_timestamp() is called unconditionally in
next_beyond_location() for every file on every iteration step,
resulting in O(N x files) volatile mmap reads. For large queries
like 'journalctl -n 1000000' this makes the command unusably slow
(~5 minutes on systems with many journal files).
Rather than suppressing the call entirely (which would make the
inotify path fully load-bearing for cross-boot ordering), rate-limit
it to at most once per second per file. This reduces the overhead
from O(N x files) to O(T x files) where T is the iteration time in
seconds, while still providing periodic refresh as a fallback for
any missed inotify events and keeping cross-boot ordering
reasonably fresh.
Embed a RateLimit struct in JournalFile for this purpose.
Measured improvement on a real system: 5:24 -> 2:39 (-51%) for
'journalctl -n 1000000'.
Fixes: #42521
Regression introduced in: #26355 (34af74946e)
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
(cherry picked from commit 8f8ab4bf98)
After walking a hash chain, the loop exits with p == 0. The error()
call for a tail_hash_offset mismatch passed p as the file offset,
printing 0000000000000000 instead of the actual last data object.
Pass 'last' instead, which holds the offset of the final chain entry.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
(cherry picked from commit 99b3bbb4c0)
If a varlink method takes a struct/object as a parameter, and it’s
marked as `SD_JSON_MANDATORY`, and it has an optional field inside it
which is *not* marked as `SD_JSON_MANDATORY`, we want to not require
that field to be set.
Previously, due to using the `merged_flags` from both the mandatory
struct and the optional field, `SD_JSON_MANDATORY` was effectively
always set on the optional field even if we didn’t want it. This
resulted in an error being emitted if the mandatory struct was provided
in a varlink call, but without the optional field.
Fix that by validating the field’s presence only against its own flags
and not also the flags of its parent.
Adds a unit test to prevent regressions.
(cherry picked from commit 0b3eeb1547)
journal_file_setup_data_hash_table() allocates s * sizeof(HashItem)
bytes for the hash table but then only zeroes s bytes, leaving 15/16 of
the entries uninitialized. This corrupts the hash chain in any newly
created journal file.
The adjacent journal_file_setup_field_hash_table() already uses the
correct size.
Signed-off-by: dongshengyuan <dongshengyuan@uniontech.com>
(cherry picked from commit a9e8f89592)
The binary catalog reader trusted two values straight from a (possibly
hostile) database: open_mmap() summed header_size + n_items *
catalog_item_size in uint64 with no overflow check, and find_id() added
the matched item's offset to the map base with no upper bound. Reachable
through sd_journal_get_catalog() with $SYSTEMD_CATALOG set, this let
catalog_get()/catalog_list() strdup() a string starting outside the
mapping. Guard the size math with MUL_SAFE/INC_SAFE and reject item
offsets that fall outside the file.
head_data_offset is declared as le64_t in journal-def.h, so it must be
assigned with htole64(p), not le64toh(p). All other le64_t field
assignments in this file (hash, next_hash_offset) consistently use
htole64().
On little-endian systems this makes no difference, but on big-endian
systems the field->data link would be stored with incorrect byte order,
corrupting journal file traversal.
Assisted-by: claude-opus-4-8 <noreply@anthropic.com>
When a slot is disconnected, previously we tried to remove the slot from
the hashmap with a wrong key. Hence, the pointer to the freed slot object
remained in the hashmap.
The cleanup destructor for VarlinkServerSocket only freed ss->address and
the struct, leaking ss->event_source. If sd_varlink_server_listen_address()
hits OOM at free_and_strdup() after the io-source was already armed, the
source stays registered in the event loop with userdata pointing at the
freed socket. Disable it before freeing; the call is a no-op when the
source was never armed, so the other freep sites are unaffected.
Follow-up for c14e841f31
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
When the accumulated trie key exceeds the fixed-size line buffer,
linebuf_get() returns NULL. trie_fnmatch_f() passed that NULL straight
into fnmatch() as the pattern, causing a SIGSEGV on a crafted hwdb.bin
(reachable now that recursion is capped rather than overflowing the
stack first). Treat the NULL like the other corruption checks and
return -EBADMSG.
Follow-up for 73fea38cf1
Fixes https://github.com/systemd/systemd/issues/42376
Co-developed-by: Claude Opus 4.8 <noreply@anthropic.com>
Many *at()-style helpers accept a directory fd that may be either a
regular, valid fd (>= 0) or one of the special AT_FDCWD/XAT_FDROOT
wildcard values, and open-code that check in their assertions. Add a
wildcard_fd_is_valid() helper for it and use it tree-wide.
Fixes a family of OOB reads in the trie walker where attacker-controlled
offsets are loaded from a (possibly hostile) hwdb.bin and added to
hwdb->map without first checking that the resulting pointer stays inside
the mapping. Crash sites:
- trie_fnmatch_f sd-hwdb.c:187 (#42340)
- trie_search_f sd-hwdb.c:233 (#42341)
- trie_children_cmp_f /bsearch sd-hwdb.c:94 (#42342)
- hwdb_add_property sd-hwdb.c:121 (#42343)
- trie_node_from_off (UBSAN) sd-hwdb.c:83
- trie_fnmatch_f stack overflow on cyclic children (CIFuzz finding)
Commit b45a897edc ("hwdb: reject out-of-bounds fnmatch prefixes")
plugged the first site only at the prefix_off-content level, but the
crash on the read of node->prefix_off itself still happens when node is
already OOB. This is the structural fix: a single hwdb_at() helper
validates that [off, off + size) lies inside the mapping, and
trie_node_from_off() / trie_string() return NULL on OOB; every caller
treats the result as nullable and surfaces -EBADMSG.
trie_fnmatch_f() additionally caps its recursion depth at 2048; without
the cap a corrupt trie whose child offsets form a cycle (or just a deep
linear chain) drives the parser into stack-overflow rather than
returning -EBADMSG (caught by CIFuzz address sanitizer on this branch).
Additions:
- A fuzz-hwdb libFuzzer harness that drives sd_hwdb_new_from_path,
sd_hwdb_get, sd_hwdb_seek, and sd_hwdb_enumerate on attacker bytes.
- Five hand-crafted unit tests in test-sd-hwdb.c (one per crash
bucket, plus the cyclic-trie recursion case) that build a malformed
in-memory hwdb.bin and assert -EBADMSG rather than SIGSEGV /
stack-overflow.
- Regression corpus files under test/fuzz/fuzz-hwdb/ pinning each
fixed bucket, including the CIFuzz stack-overflow reproducer.
Closes#42340.
Closes#42341.
Closes#42342.
Closes#42343.
Reported-by: AI-assisted libFuzzer campaign
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
Not the case on the sparc architecture. Replace with two translation
functions. The poll to epoll variant isn't used yet but will be used
when we add io-uring support to sd-event so we already add it here so
that it is in place.
Follow-up for cf4c65afa8
Ensure that fnmatch traversal doesn't go off the hwdb.bin in case of
a corrupted file
Originally reported on yeswehack.com as #YWH-PGM9780-70
For the unit test:
Co-developed-by: GitHub Copilot <github-copilot[bot]@users.noreply.github.com>
When a streaming method's callback returns 0 with no replies enqueued and
sd_varlink_set_sentinel() was passed NULL (POINTER_MAX), the framework
emits an empty parameters reply via sd_varlink_reply(v, NULL) as a clean
stream terminator. That call validated the empty parameters against the
method's output IDL — which always failed when the IDL declared any
mandatory output field, generating a spurious 'didn't pass validation'
warning even though the empty terminator is correct by construction.
Move the body of sd_varlink_reply() into a static varlink_reply_internal()
that takes a skip_validation flag. The public sd_varlink_reply() keeps
its existing behavior (validation enabled). Add a static
varlink_reply_terminator() helper that wraps
varlink_reply_internal(v, NULL, /* skip_validation= */ true), and route
varlink_dispatch_sentinel() through it for the synthetic empty terminator,
so both the synchronous dispatch path (varlink_dispatch_method()) and the
fiber path (varlink_fiber_entry()) skip validation in one place.
Without this, callers who want 'streaming method, possibly empty result'
semantics had to either fire a logically wrong 'NotFound'-style error
sentinel on empty (sysext.List, BootControl.ListBootEntries pattern) or
mark every output field NULLABLE just to placate the validator (the
io.systemd.Manager.EnqueueMarkedJobs approach), misrepresenting the
contract — exactly the tradeoff the FIXME in pcrlock.c flagged. With
this change, the empty terminator is invisible to the IDL validator and
the FIXME is no longer needed.
Co-developed-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Traditionally, asynchronous programming in systemd has been achieved
using
sd-event along with the asynchronous interfaces of sd-bus and
sd-varlink.
This works well when the system is reacting to events and all code
triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback
can
use the stack as usual, declaring local state and ensuring proper
cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block
execution
within the callback, we can't directly invoke a long-running operation
and
wait for its result without introducing complexities. Instead, we need
to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along
as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope
may
be prematurely cleaned up. Therefore, the struct must store copies of
the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a
loop,
the complexity grows further. We must introduce additional shared state
to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must
track
the lifecycle of each long-running operation in the global Manager
struct,
ensuring proper cleanup even when stack unwinding can no longer manage
the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s
running
synchronously, but with the ability to yield control at predefined
points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while
waiting
for the long-running operation to finish and then "resume" the operation
once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in
a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve
the
handling of complex workflows, making the code easier to write and
maintain.
The implementation is based on ucontext.h's makecontext() (with a
fallback
to the venerable sigaltstack() approach on musl),
sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can
switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext()
because the
latter forcibly saves/restores a per context signal mask every time it
is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and
maintain
a per thread signal mask, which makes much more sense than having a per
fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used
until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a
deferred
event source which resumes the fiber when enabled. The deferred event
source
is oneshot by default so the fiber will run immediately until it yields
or
suspends. If it yields, the deferred event source is enabled again
(oneshot)
immediately. If it suspends, before it suspends, one or more event
sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for
completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or
any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is
cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the
fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup
to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them
behind
a generic futures interface to represent long-running operations,
regardless of
whether those operations are running on a fiber or not. Aside from
fibers, the
futures library (sd-future) will for example allow waiting for sd-event
sources
and doing sd-bus calls in the background as well. Fibers can suspend
until a
future is ready with sd_fiber_await() or by having the future wake up
the fiber
explicitly in its callback. A future always defaults to waking up the
current
fiber.
Each future kind plugs into the library by providing an sd_future_ops
vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it
via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to
that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from
a
fiber via sd_fiber_await() (synchronous-looking fiber code).
sd_fiber_await()
is itself built on a "wait future" that resolves when its target
resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber
callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future
impl's
cancel callback, which is responsible for tearing down its work and
ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from
sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new()
—
take a self-reference on their future so they outlive the caller's
scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds
would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal
sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an
unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the
following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of
it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
Add varlink_server_bind_fiber() and varlink_server_bind_fiber_many()
in varlink-util.{c,h} for registering a method handler that should
run on a dedicated fiber per dispatch. The fiber-bound methods live
in a separate s->fiber_methods map alongside the regular s->methods;
bind_internal()/bind_many_internal() are factored out so the regular
and fiber bind variants share their parsing/insertion code.
Registering the same method in both maps is rejected because the
dispatcher consults the regular map first and would otherwise
silently shadow the fiber binding.
varlink_dispatch_fiber() builds a VarlinkFiberData (refs to the
connection, parameters, and method name), spawns a fiber via
sd_fiber_new(), and makes the future floating so the fiber
self-manages its lifetime — neither the dispatcher nor the
connection has to track it. The fiber's priority is set to one
below the connection's quit event source so that on graceful
shutdown the fiber's exit handler fires (and runs its cleanup)
before varlink's quit_callback() closes the connection underneath
it; this is what lets a fiber-bound handler reply or flush its
sentinel on a still-open connection during shutdown.
The connection state transitions are reordered so they happen before
the fiber spawn rather than after the synchronous callback returns:
the fiber runs after dispatch has already moved past PROCESSING, which
matches the behaviour expected for a deferred reply (the fiber may
either reply immediately, or stash the connection and reply later, in
which case the post-callback logic treats it as a PENDING_METHOD).
Note that all the synchronous varlink APIs (sd_varlink_call() and friends)
already behave properly when on a fiber because they call json_stream_wait()
which calls ppoll_usec() which we already fixed to suspend when called from
a fiber.
The client/server varlink tests are migrated to fibers (threads → mock
server fibers on the same event loop) to exercise the new paths.
Two changes to teach sd-bus how to behave when called from a fiber, in
order of increasing depth:
2. sd_bus_call() now redirects to a new bus_call_suspend() helper when
the caller is a fiber whose event loop is the same one the bus is
attached to. The plain bus_poll() path serializes all bus traffic on
the slot's reply (only one method call can be in flight per
sd_bus*), which would defeat the point of running multiple fibers
against one bus. bus_call_suspend() builds on the async sd-bus API:
it wraps the call in a new BusFuture (sd-bus/bus-future.{c,h}) that
resolves when the reply or method-error arrives, lets the fiber
await that future, and surfaces the reply to the caller via
future_get_bus_reply(). Because the futures live on the event loop
rather than a per-bus slot, multiple fibers can drive concurrent
method calls against the same bus.
3. A new private SD_BUS_VTABLE_METHOD_FIBER flag dispatches a vtable
method handler on its own fiber, so handlers are free to use
sd_bus_call() against the same bus, sd_fiber_sleep(), loop_read(),
etc. without stalling the event loop for other connections or
handlers. The flag stays out of sd-bus-vtable.h (its bit value is
reserved there to prevent collisions) — the fiber runtime is a
systemd-internal implementation detail.
Lifecycle of fiber-dispatched handlers is tracked on the bus itself: a
new bus->fiber_futures set holds a ref to each in-flight handler.
bus_enter_closing() cancels every entry and process_closing() returns
with the bus still in CLOSING state until the set drains, so we can be
sure no fiber handler outlives the bus. bus_fiber_resolved() removes
the entry on completion. bus_free()'s assert(set_isempty()) makes the
invariant load-bearing.
Note that plain sd_bus_call() already works correctly on a fiber as it
calls ppoll_usec() which has already been modified to suspend when
running on a fiber.
To exercise these changes the existing thread-based client/server
sd-bus tests (test-bus-chat, test-bus-objects, test-bus-peersockaddr,
test-bus-server, test-bus-watch-bind) are migrated to fibers, and a
new test-bus-fiber is added that covers SD_BUS_VTABLE_METHOD_FIBER —
including handlers that issue nested sd_bus_call() on the same bus, the
cancel-on-close path, and concurrent dispatches across multiple fibers.
sd_event_run() blocks the calling thread on the event loop's epoll fd
until something happens. When the caller is a fiber, that's the wrong
behaviour: blocking the thread also stalls every other fiber and the
outer event loop driving them. The most common way to hit this is a
fiber that creates its own inner event loop (e.g. a server-style fiber
that wants to dispatch its own sources independently of whatever loop
the test or supervising fiber is running on) — with the existing
implementation the inner sd_event_run() would hold the thread while the
outer scheduler should be free to advance other fibers.
Add an event_run_suspend() variant in sd-event/event-future.c that
performs the same prepare/wait/dispatch dance, but when the fast path
finds nothing ready it (a) creates an IO future watching the inner
event loop's epoll fd on the *outer* event loop, (b) optionally creates
a time future for the timeout, and (c) suspends the fiber. When either
future fires the fiber is resumed and the prepare/wait/dispatch sequence
runs once more to actually dispatch what's pending. sd_event_run()
checks sd_fiber_is_running() and delegates to this variant when on a
fiber; profile_delays accounting is intentionally skipped on that path
since the underlying prepare/wait/dispatch primitives already account
for themselves.
Some helpers in src/basic — ppoll_usec_full() (used by fd_wait_for_event()),
loop_read(), loop_read_exact(), loop_write_full() and
pidref_wait_for_terminate_full() — block the calling thread. That's the
right behaviour outside a fiber but not inside one, where blocking the
thread also stalls every other fiber running on the same event loop.
Rewriting every caller to pick a fiber or non-fiber variant explicitly
would be a lot of churn and would split otherwise-shared code paths in
two.
Instead, the helpers detect at runtime whether they're running on a fiber
and dispatch to a suspending variant when they are. FiberOps in
fiber-ops.h holds five function pointers (ppoll, read, write, timeout,
cancel_wait_unref); a fiber_ops global constant is populated whenever we
enter a fiber with functions that delegate to suspending variants of common
syscalls. With this approach, the variants themselves stay in libsystemd
which is required because they make use of sd-event.
- loop_read()/loop_read_exact() take the fiber read hook on a fiber
unless the caller asked for a non-blocking attempt (do_poll=false) and
the fd is already non-blocking — in that case we fall through to read()
to preserve the existing return-EAGAIN-immediately semantic. The hook
itself suspends on EAGAIN until data is available, so neither the
do_poll knob nor the explicit fd_wait_for_event() retry loop are
needed on the fiber path.
- loop_write_full() likewise takes the fiber write hook on a fiber,
except when timeout=0 with an already-non-blocking fd (preserving the
fast-return-EAGAIN semantic). The fiber path runs inside a
FIBER_OPS_TIMEOUT() scope so the caller's timeout is honoured via a
deadline future, mirroring SD_FIBER_TIMEOUT() but reachable from
src/basic without pulling in sd-future.h.
- pidref_wait_for_terminate_full() polls the pidfd via fd_wait_for_event()
before each waitid() when either a finite timeout is set or we're on a
fiber, and requires pidref->fd >= 0 in those cases (returning
-ENOMEDIUM otherwise — extending the rule that already applied to
finite timeouts). The poll suspends the fiber via the ppoll hook above;
the subsequent waitid() doesn't block because the pidfd is already
signalled.
Add a family of sd_fiber_*() I/O wrappers that, when called from a
fiber, behave like blocking I/O from the caller's perspective but
yield to the event loop instead of blocking the thread:
sd_fiber_read / sd_fiber_write
sd_fiber_readv / sd_fiber_writev
sd_fiber_recv / sd_fiber_send
sd_fiber_connect
sd_fiber_recvmsg / sd_fiber_sendmsg
sd_fiber_recvfrom / sd_fiber_sendto
sd_fiber_accept
sd_fiber_ppoll
Most of them share a single helper, fiber_io_operation(), which when
invoked outside a fiber falls through to the underlying syscall
directly, preserving the regular blocking behaviour. Inside a fiber
the helper flips the fd to non-blocking (restoring its original mode
on return), tries the syscall once on the fast path, and on EAGAIN/
EWOULDBLOCK creates an sd-event-backed IO future via future_new_io(),
suspends the fiber, and retries the syscall once the event source
fires.
future_new_io() itself is added to sd-event/event-future.{c,h} as a
new IoFuture kind. It wraps sd_event_add_io() into an sd_future:
oneshot enable, EPOLLERR translated via SO_ERROR (suppressed for
non-sockets), and the fd duplicated with F_DUPFD_CLOEXEC to avoid
EEXIST when multiple sources watch the same descriptor.
Together these let fiber-using code write straight-line socket and
pipe I/O without bundling state into callbacks.
Traditionally, asynchronous programming in systemd has been achieved using
sd-event along with the asynchronous interfaces of sd-bus and sd-varlink.
This works well when the system is reacting to events and all code triggered
by those events can run without blocking. In these scenarios, the global
Manager object is passed as userdata to the callback, and the callback can
use the stack as usual, declaring local state and ensuring proper cleanup via
_cleanup_. Control flow structures, such as loops, work as expected, and
everything runs smoothly.
However, challenges arise when the code needs to perform long-running
operations within these callbacks. Since the system cannot block execution
within the callback, we can't directly invoke a long-running operation and
wait for its result without introducing complexities. Instead, we need to
initiate the long-running task, register for completion with sd-event,
sd-bus, or sd-varlink, and provide a callback to be invoked when the
operation completes.
This callback, however, only receives a single userdata pointer, which
forces us to bundle all local variables into a struct and pass it along as
part of the callback. On top of that, after queuing the asynchronous
operation, the caller continues executing. As the caller's stack unwinds
when the function exits, the resources and state within the local scope may
be prematurely cleaned up. Therefore, the struct must store copies of the
local variables or ensure proper reference counting to prevent premature
resource cleanup.
When multiple long-running operations need to be initiated within a loop,
the complexity grows further. We must introduce additional shared state to
track the completion of all operations before we can run any code that
depends on their results.
Furthermore, since the daemon may be shut down at any time, we must track
the lifecycle of each long-running operation in the global Manager struct,
ensuring proper cleanup even when stack unwinding can no longer manage the
resources for us.
Fibers, or green threads, provide a more natural way of handling
asynchronous operations. By enabling cooperative multitasking within a
single thread, fibers allow us to write code that looks like it’s running
synchronously, but with the ability to yield control at predefined points,
such as when waiting for long-running tasks to complete.
With fibers, we can simplify the control flow by running asynchronous
operations within a fiber, allowing us to "pause" execution while waiting
for the long-running operation to finish and then "resume" the operation once
it's complete. This eliminates the need for multiple callback chains,
extensive state tracking, and the potential pitfalls of stack unwinding.
This commit introduces the ability to execute long-running operations in a
non-blocking manner while maintaining the simplicity and readability of
synchronous code. The fiber-based approach will significantly improve the
handling of complex workflows, making the code easier to write and maintain.
The implementation is based on ucontext.h's makecontext() (with a fallback
to the venerable sigaltstack() approach on musl), sigsetjmp()/siglongjmp()
and sd-event. ucontext.h provides us with alternate stacks that we can switch
between. We use sigsetjmp()/siglongjmp() instead of swapcontext() because the
latter forcibly saves/restores a per context signal mask every time it is called.
Using sigsetjmp()/siglongjmp(), we can avoid the unnecessary syscall and maintain
a per thread signal mask, which makes much more sense than having a per fiber
signal mask.
The default stack size is the same as a regular thread. Because we
use mmap() to allocate the stack, the memory won't actually be used until it
is paged in by the kernel, so we don't actually use 8MB per fiber.
To integrate fibers with the event loop, each fiber is assigned a deferred
event source which resumes the fiber when enabled. The deferred event source
is oneshot by default so the fiber will run immediately until it yields or
suspends. If it yields, the deferred event source is enabled again (oneshot)
immediately. If it suspends, before it suspends, one or more event sources
are registered with sd-event that will enable the deferred event source
(oneshot) to resume the fiber once the operation it is waiting for completes.
Yielding or suspending the fiber is done by calling sd_fiber_yield() or
sd_fiber_suspend() respectively. Both of these return zero on success or any
error value from the async operation that caused the fiber to resume.
This is also how fiber cancellation is implemented. When a fiber is cancelled,
sd_fiber_yield() and sd_fiber_suspend() will return ECANCELED when the fiber
is resumed, allowing the fiber to unwind its stack (which allows cleanup to
happen automatically) and finish.
Instead of having applications work directly with fibers, we hide them behind
a generic futures interface to represent long-running operations, regardless of
whether those operations are running on a fiber or not. Aside from fibers, the
futures library (sd-future) will for example allow waiting for sd-event sources
and doing sd-bus calls in the background as well. Fibers can suspend until a
future is ready with sd_fiber_await() or by having the future wake up the fiber
explicitly in its callback. A future always defaults to waking up the current
fiber.
Each future kind plugs into the library by providing an sd_future_ops vtable
(alloc, free, cancel, set_priority). The library treats the impl pointer
returned by alloc() as a black box. Future Implementations retrieve it via
sd_future_get_private().
A future starts in SD_FUTURE_PENDING and transitions exactly once to
SD_FUTURE_RESOLVED, carrying an integer result. Consumers can react to that
transition either by installing a one-shot callback with
sd_future_set_callback() (callback-style code) or by waiting on it from a
fiber via sd_fiber_await() (synchronous-looking fiber code). sd_fiber_await()
is itself built on a "wait future" that resolves when its target resolves;
sd_future_new_wait() exposes the same primitive directly so non-fiber callers
can chain futures without involving a fiber.
Cancellation is cooperative: sd_future_cancel() invokes the future impl's
cancel callback, which is responsible for tearing down its work and ultimately
resolving the promise with -ECANCELED. For fiber futures this is what
surfaces as the ECANCELED return from sd_fiber_yield()/sd_fiber_suspend()
mentioned above.
Fire-and-forget fibers — created by passing a NULL ret to sd_fiber_new() —
take a self-reference on their future so they outlive the caller's scope.
The self-ref is dropped when the fiber resolves. This floating mechanism
(sd_fiber_set_floating()) is restricted to fiber futures because they
uniquely guarantee resolution; allowing it for arbitrary future kinds would
risk silent leaks for kinds that may never resolve.
Note that fiber cleanup depends on the runtime operating normally. Each
fiber's _cleanup_-style cleanups live on the fiber's own stack and run
only when the fiber is resumed and allowed to unwind, which requires a
working event loop to drive it to completion. The exit event source
registered for top-level fibers ensures unwind on a normal sd_event_exit(),
but if the event loop itself terminates abnormally (e.g. an unrecoverable
allocation failure mid-dispatch) before all fibers have resolved, their
stacks never unwind and any resources they own leak.
The code lives in libsystemd as sd-future (not exported) for the following reasons:
- We may want to make this a public libsystemd API in the future
- The code can't live in src/basic as it makes heavy use of sd-event
- The code can't live in src/shared as sd-bus and sd-event make use of it
The log and log-context headers are updated with functions to allow
fibers to have their own log prefix and log context.
Previously, sd-dhcp-server can be run as a DHCP relay agent.
But, DHCP server and DHCP relay agent behave completely differently,
hence there is almost no code that can be shared between the two modes.
Let's split out the DHCP relay agent feature from sd-dhcp-server.
The new DHCP relay agent supports:
- multiple upstream and downstream interfaces,
- gateway address (giaddr field in DHCP message header) is configurable,
- supports more DHCP relay agent information sub-options,
- each interface has their own socket fd, and each socket is bound to
the interface, so that we can enable/disable each interface
safely without affecting other interfaces, and we can filter out any
unexpected packets from unmanaged interfaces,
networkd integration and test cases will be added later.
Drop the -fundamental suffix from src/fundamental/ headers in favor of names
that match their src/basic/ or src/shared/ counterparts (e.g.
macro-fundamental.h -> macro.h, assert-fundamental.h -> assert-util.h,
cleanup-fundamental.h -> cleanup-util.h). Rename src/basic/{btrfs,label}.{c,h}
to use the -util suffix to match the existing shared/btrfs-util and
shared/label-util siblings. Rename src/shared/mkdir-label.{c,h} to mkdir.{c,h}
and src/shared/tmpfile-util-label.{c,h} to tmpfile-util.{c,h} to match the
corresponding src/basic names.
This saves us from having to come up with separate names for files that do
the same thing across tiers, and it makes it easier to move stuff between
src/fundamental, src/basic and src/shared: consumers just #include "foo.h"
and pick up whichever tier their -I path resolves to first, so call sites
don't need to be updated when an API moves between layers.
Where a higher-tier wrapper exists (e.g. src/basic/macro.h wrapping
src/fundamental/macro.h), the wrapper uses an explicit "../fundamental/foo.h"
or "../basic/foo.h" relative include for the lower-tier header. We can't use
GCC's #include_next directive for this — when the wrapper is reachable both
via same-dir-as-source lookup and via -I (e.g. -Isrc/shared) for the
directory it lives in, #include_next advances by exactly one slot in libcpp's
internal directory chain and lands on the same physical directory it was
already in, never reaching the lower-tier sibling (see make_cpp_dir() in
gcc/libcpp/files.cc:1986).
To make sure the right headers are always picked up, the include directories
are reordered so that e.g. src/shared always takes priority over src/basic and
similar for the other directories.
Co-developed-by: Claude Opus 4.7 <noreply@anthropic.com>
- test-random-util is reworked to not use sqrt()
- pretty-print.c inlines ceil() so libm doesn't have
to be linked into libshared
- We add fno-math-errno to allow inlining of more math
functions by not requiring standard math functions to
set errno on invalid input.
exp10() has a symbol version > 2.34 on latest glibc. To allow
dropping our baseline required glibc runtime version to <= 2.34,
let's add our own version to prevent pulling in the newer symbol
from glibc.
Methods:
- `io.systemd.Job.List` — list all queued jobs or look up by `id`/`unit`
name, with streaming support. Uses context/runtime split: `JobContext`
(Unit, JobType) and `JobRuntime` (Id, State, Result,
ActivationDetails). Follows the same SELinux and parameter-conflict
patterns as `io.systemd.Unit.List`.
- `io.systemd.Job.Cancel` — cancel a specific job by ID, with SELinux
and polkit authorization.
- `io.systemd.Job.ClearAll` — cancel all pending jobs, with SELinux and
polkit authorization.
`message_skip_fields()` recursively processes D-Bus variant types in
message header fields with no depth limit. A crafted message with deeply
nested variants can cause unbounded recursion and overflow the stack.
Add a `depth` parameter checked against `BUS_CONTAINER_DEPTH` (128),
matching the limit already enforced by the public
`sd_bus_message_skip()` API. All recursive call sites pass `depth + 1`,
and the top-level caller in `message_parse_fields()` passes `0`.
Craft a raw D-Bus message with an unknown header field containing
BUS_CONTAINER_DEPTH+1 nested variants and verify that message parsing
rejects it with -EBADMSG rather than recursing until stack overflow.
message_skip_fields() recurses for each nested variant ('v') type in
D-Bus message header fields. A crafted message with deeply nested
variants (e.g., a variant containing a variant containing a variant...)
causes unbounded stack growth, leading to stack overflow and crash.
Add a depth parameter that increments on each recursive call and
rejects messages exceeding BUS_CONTAINER_DEPTH with -EBADMSG. This
matches the existing depth limits enforced elsewhere in the sd-bus
message processing (e.g., bus_message_enter_container).
Each dlopen_*() wrapper kept its dl handle as a file-scope
'static void *xxx_dl = NULL;' even though only the wrapper itself
ever referenced it. Move each one inside the corresponding function
so its scope matches its actual use, leaving the rest of each
translation unit free of the unused file-scope name.
In pcre2-util.c the assert(pcre2_dl) in pattern_matches_and_log()
becomes assert(sym_pcre2_match), which carries the same invariant
(pattern_compile_and_log() ran dlopen_pcre2()).
- **bpf-util: rename from bpf-dlopen, unify version-specific symbol
handling**
- **cryptsetup: dlopen libcryptsetup in tokens**
- **tree-wide: dlopen libpam in pam plugins**
- **test-bus-marshal: dlopen() glib and libdbus instead of linking
directly**
- **lock-util: Simplify timeout for lock_generic_with_timeout()**
- **color-util: simplify hsv_to_rgb, fix rgb_to_hsv negative-hue wrap**
- **tree-wide: Use our own macros instead of fabs()/fmax()/fmin()**
- **locale-util: dlopen() libintl instead of linking against it**
- **home: Use log2u64() over log2()**
- **meson: drop libdl, threads, and librt dependencies**
- **libc: Use dlsym() from a constructor instead of weak symbols**
- **libc: Make sure C23 versions of strtol(), sscanf() are not used**
Our baseline glibc is 2.34, which merged libdl, libpthread (the
dependency('threads') target), and librt into libc. Empty .so/.a stubs
remain for backward compatibility with old binaries, but new builds
resolve dl_*, pthread_*, mq_*, timer_*, etc. directly from libc.
On musl the same libraries are likewise empty stubs.
Drop the libdl, threads, and librt entries from every meson.build, and
remove the now-stale 'Libs.private: -lrt -pthread' from libudev.pc.in
since both flags resolve to empty link-time stubs on glibc 2.34+ and
musl.
Verified with readelf -d that libsystemd.so, libudev.so, and systemd no
longer carry DT_NEEDED entries for libdl/libpthread/librt.
Yu Watanabe