The Desktop hand-off window's stage text was published from the END mark of the
previous stage (`record_stage` guessed the next one from a table). A failed build
published nothing, but the code is committed so the update still restarts the
gateways, and the window kept the build/packaging message through the whole
restart (Greptile review on #144922).
Each message is now published where its stage begins: "Downloading updates"
after the backup, "Updating dependencies" before the completion's dependency
sync, "Building the apps" at the product build, and "Restarting Hermes services"
inside the fleet restart phase itself, so every caller (current completion and
historical takeover) gets it after a failed build too. `record_stage` is back
to origin/main's shape: no next-stage table, no failed-snapshot special case.
The Windows stage-relay test waits with tests.waiting.wait_until instead of two
hand-rolled sleep loops (the STAGE_CLI child imports it from the repo it already
puts on sys.path).
_current_checkout_sha() went through get_code_identity(refresh=True), which drops the cache and
re-runs the whole provenance walk (tag scan, rev-list, status, ...) only to compare SHAs. The
startup fleet hint calls it on every launch, so every `hermes` command paid a second full walk.
read_checkout_sha() answers the same SHA (stamp, else `git rev-parse HEAD`) and drops the cached
identity the same way, so updater callers that relied on the refresh still re-resolve.
The Windows resume merge appends its failed services to failed_or_stale_units
after the restart phase, and an aborted phase leaves entries there too; neither
should print the systemctl/launchctl "units were not restarted" advice. Comment
only, no behaviour change.
- A unit the restart phase recorded failed (a timed-out systemctl call whose job
systemd then finished) kept "Update incomplete — some units were not
restarted" and the gateway_restart follow-up even when the fleet probe showed
that unit's gateway serving the checkout. Verification now credits a
hermes-gateway* unit only on its OWN proof: its MainPID is a `current` fleet
row and was not running when the restart phase began (every scope where the
unit is loaded must agree). Another profile's current gateway never vouches
for it. The incomplete-units warning moves after the fleet probe and names
only the units still unproven; the receipt, marker and host restart stamp
follow the reconciled verdict.
- "↷ PID N: gateway process of another Hermes home — left alone" printed once by
the pre-swap pause and again by the restart phase (a fresh completion
process). Each notice is now recorded on the run's receipt, which the
completion request carries, and printed once per run.
A gateway unit whose stop outlives the updater's systemctl timeout (a cgroup
child stuck in D state makes systemd wait TimeoutStopSec for SIGTERM and again
for the final SIGKILL: ~181 s on a 90 s unit) broke both ends of `hermes update`:
- pause: `_disarm` -> `_stop_unit` timed out at 120 s although the drained
gateway was already gone; the pause rolled back with `systemctl start`, which
queued behind the still-running stop job and booted the gateway on the old
tree seconds before the git swap.
- restart: the post-drain `systemctl start` timed out the same way and the unit
was recorded failed ("Update incomplete"), while systemd finished the job and
brought the gateway up on the new code.
The client timeout cancels nothing in systemd. `run_unit_job` (one helper in
update_cmd_fleet_unit_budget, used by the pause stop/start, the post-drain
start and the blunt/catch-up restart) now waits for the unit's own job past a
timeout, bounded by 2 x TimeoutStopUSec (+ TimeoutStartUSec for a start) + slack
from the call, under an independent 900 s cap, and reports the settled state.
The pause fails only when the gateway is provably still running; a stop job
systemd is still finishing holds it. Calls that return in time are unchanged.
Folds the final-head review and /simplify-code pass on #141070:
- update_serve_obligations: defer_manual_serve and defer_unit_serve share
_defer_identified (pid/create-time guard, dead-pid discharge, reminder
write) instead of two copies; a unit row's failed-write warning says
"unit restart reminder", not "manual".
- update_cmd_fleet: a positive _is_serve_or_dashboard_unit name gate,
composed into _is_hermes_gateway_unit; _is_serve_unit uses it rather
than "gateway gate minus SIGUSR1 support". _SERVE_SKIP_REASON names
hermes-dashboard* units too.
- update_cmd_fleet_verify: SERVE_ONLY_REASON_PREFIX and the gateway
sentence are module constants, so the producer and the Desktop
predicate share one spelling; the test now feeds the producer's own
reasons to both scripts.
- Desktop hand-offs (posix.sh, windows.ps1): drop the gateway hint only
when EVERY gateway_restart line is serve/dashboard-only, so a serve-only
line next to a raw restart-phase error keeps it; case-sensitive in
PowerShell (-cmatch) like grep.
- main_dashboard._format_restart_commands replaces three copies of the
backticked " or " join.
- Docstrings: already_restarted_units covers declined and failed units.
Review of the stack found that classifying a unit's MainPID row as
systemd dropped the only cross-run reminder a stale system dashboard had:
on main the manual-serve row filed serve_restart_pending/<pid>.json, but
the fleet obligation records only gateway rows, so once the gateways were
current the stale unit was never mentioned again. Verification now files
the same per-incarnation reminder for a surviving systemd-owned
serve/dashboard, carrying its unit's restart command; startup prints that
command and clears it once the process exits.
Also from review:
- report_unaccounted_runtimes prints the owning unit's command instead of
the hardcoded user-scope hermes-serve hint when the unit is known.
- The follow-up still names a gateway a previous run left owed even when a
serve unit already made this run incomplete.
- Dashboards the cleanup stopped and could not bring back are worded as
stopped, not as still running pre-update code; failed serve units are
worded neutrally (the loop records them for several reasons).
- _is_serve_unit uses the fleet loop's own name gate, not a second regex.
- The PID -> (unit, scope) lookup and the scope -> restart command mapping
move to main_dashboard (_systemd_owner_for_pid, _systemd_restart_commands)
and gateway.py's service-managed refusal reuses them; an unknown scope
prints both concrete commands.
- Stale comment on _SERVE_SKIP_REASON updated; tests stub stale-survivor
signalling and exercise the real MainPID-vs-cgroup rule.
Refs #141021
`_collect_ledger_runtimes` knew launchd, desktop-ssh, desktop and
manual-serve. A unit-launched backend sets neither HERMES_SPAWN nor
HERMES_PARENT_PID, so a hermes-dashboard unit's MainPID read as
manual-serve and the update told the user to "ask their owner to
relaunch `hermes dashboard`" for a process systemd owns. The systemd
hint in `_warn_stale_serve_runtimes` keyed on supervisor == "systemd",
which this path never produced, and hardcoded a user-scope
hermes-serve.service.
Classify a ledger row whose PID is a unit's live MainPID (the same
ownership rule `_get_systemd_service_for_pid` already applies; cgroup
membership alone stays manual) as systemd, record the unit and scope,
and print that unit's exact restart command in the survivor warning.
The Desktop spawner check still wins over the unit check.
Refs #141021
The gateway /update watcher polls .update_exit_code every 2 s and reads an
empty file as exit 1. The fleet, completion and takeover writers and the
detached bash/Windows wrappers truncated it in place, so a poll in that window
told the user a successful update failed and cleared the markers. All writers
now write a temp file and rename it, like update_finish already did.
When the host update-restart record cannot be written, the arm falls back
to the invoking profile's fleet_restart_pending marker, which only that
profile reads. The fallback counted profiles with list_profile_names(),
which suppresses enumeration errors and returns ['default'], so an
unreadable profiles dir read as a single-profile install: the commit point
moved the tree while the other profiles saw no debt (review S2 residual).
The fallback is now one stdlib-only function,
update_host_obligation.arm_host_obligation, used by both the commit point
(update_cmd_fleet._write_fleet_restart_pending_marker) and the historical
updater's takeover (_update_takeover._arm_fleet_obligation), which had its
own copy with no profile check and no warning (kshitijk4poor P2). It
settles the debt with the marker only when a COMPLETE roster proves no
named profile exists; any error listing the roster or reading an entry's
identity/tombstone is unknown, so the arm returns False and says so. The
commit point refuses before the move; the takeover (tree already moved)
keeps the warning. The marker name has one definition.
When the host update-restart record could not be written,
_write_fleet_restart_pending_marker fell back to the invoking profile's
per-home marker and reported success. Every other profile of the install
reads only its own home, so it saw "no restart owed" for a tree the commit
point then moved. The per-home fallback now carries the debt only when the
install has no other profile; otherwise the arm returns False and
arm_commit_obligations refuses before the tree moves; the marker is still
left so a post-move arm keeps warning the invoking profile (review S2).
What: the host obligation record carries `owners` (one commit-point token per arming run:
install root, pid, uuid) and `found` (base64 of what its first owner found there).
write_host_obligation(owner=) adds the run to a same-SHA record instead of keeping it
byte-identical; disarm_commit_obligations releases the host record through
release_host_obligation, which removes only this run's token and, when no owner is left,
puts back what the first owner found (absent -> unlinked). A record whose owners do not name
the run (rewritten by another install's newer pull, or unparseable) stays untouched. The
non-host restores in disarm now fsync before os.replace (review C12, partial).
Why: the F22 disarm compared bytes, but a same-SHA arm kept the record unchanged, so install
A and install B arming the same SHA shared byte-identical bytes and A's failed run deleted the
debt B still owed through (review C2, probe review/commit/w2/probe_cas.py).
Test (RED at 5dfc384669e: record deleted): two installs arm the same SHA; A's disarm keeps the
record, the last owner's disarm puts back what the first owner found (nothing).
Health ratchet: update_cmd_fleet.py may only shrink vs main (2166 > 2165).
_systemd_restart_timeout (+ _SYSTEMCTL_RESTART_TIMEOUT_MAX) and
_service_restart_sec (+ _RESTART_SEC_UNITS) move verbatim to the new
sibling as systemd_restart_timeout / service_restart_sec; both late-import
update_cmd_fleet._systemctl, so the seam tests patch is unchanged. Callers
late-import the sibling; the unused update_cmd re-export of
_service_restart_sec is dropped (no shim). Behaviour is unchanged.
What: _write_fleet_restart_pending_marker reports whether a store took the restart and
arm_commit_obligations raises (pre-move refusal) when neither did; the arming snapshot treats
only FileNotFoundError as absence (any other read error refuses); disarm restores/unlinks a
record only while its bytes are still exactly what this run's latest arm left there.
Why: the tree could move with no restart owed; disarm replayed a stale snapshot over another
install's newer host record (shared per OS user) and deleted a record it merely failed to read.
Red (7847374318c update_cmd_commit.py + update_cmd_fleet.py): arm returned with both stores unwritable; disarm -> 'OLD' over 'OTHER-INSTALL'; read error -> record deleted.
Green: arm raises OSError; newer record kept; unreadable record survives; plain arm/disarm still restores.
After the run's receipt closed as `success`, three boundaries could still
write 1 to .update_exit_code (and the completion child could still answer
130), so the gateway watcher reported a failed update the receipt calls a
success:
- cmd_update's catch-alls (main.py, unchanged) call
_write_gateway_update_exit_code(False) for any escaping exception. The
writer now keeps 0 when this command's run is closed as a success
(update_receipt.committed_success: the receipt scope records the terminal
outcome it finalized, or adopted from the completion child).
- The parent adopts the child's terminal receipt and drops its stale
pre-child context as soon as the result is accepted, before the parent-
side Windows resume, so nothing after it can write that context.
- update_completion._finish: an interrupt that lands after verification
finalized the run answers 0 (the receipt's verdict); the 1 for a parked
stash (partial) or an interrupt before the run closed is unchanged, and
the status is written once the code is settled.
- update_finish (historical takeover): the same rule for its finally.
An interrupted run (receipt `interrupted`, exit 130) still writes 1, as
test_interrupt_after_child_success_demotes_gateway_marker_at_boundary pins.
Review: kshitijk4poor 5411222454 regression 3 / open thread F4.
What: an inventoried marker that owes an unmapped gateway (one the update stopped without a
profile it could name) now clears only when a local gateway no mapped owed identity explains is
running the checkout. The old guard only refused an all-external fleet.
Why: with `default` and an unmapped gateway both owed, `default` coming back current cleared the
whole marker, so the unmapped gateway stayed down with no warning and no owed restart. The new
rule subsumes the all-external refusal (no local row can be the successor).
Red: test_unmapped_stop_debt_is_not_settled_by_the_mapped_gateways_restart -> owes False with only
`default` current (review probe f7_mixed: marker cleared, pid 101 still down).
Green: owed while only `default` runs; settled once a further local gateway runs the checkout;
14 fleet/obligation files, 124 passed.
What: when the restart phase stops a gateway it cannot map to a profile, the owed obligation now
records it as an unmapped inventory row; _marker_owed_gateways turns that row into a sentinel that
only a live, current gateway of this checkout settles.
Why: the row-less obligation looked gateway-less at the next startup (an unmapped process leaves no
gateway_state.json), so the gateway-less discharge erased the debt while the gateway stayed down.
Red: pre-fix fleet/fleet_verify -> test_unmapped_stop_debt_survives_startup_until_a_current_gateway_runs fails (_update_owes_fleet_restart False, marker cleared).
Green: same test passes; reviewer probe unmapped_debt2 (with _pytest_owns_live_checkout stub) exits 0, inventory carries the unmapped row.
- A receipt store that refuses the terminal write after the commit point no
longer selects exit 1: update_receipt keeps the correlated terminal record
this process finalized (finalized_receipt) and the completion child answers
its parent from it; run_completion's correlation gate is unchanged.
- Restart failures (failed systemd unit, abort recovery, Windows service resume)
no longer write 1 to the gateway /update marker; the debt stays the
gateway_restart follow-up, the host obligation and the warning lines.
- TAIL_FOLLOWUPS includes profile_sync, so an owed profile sync keeps
source-completion-pending armed and withholds the install stamp.
The durable running latest.json carries the previous receipt's manual-serve rows
as `carried_manual_serves` (no plan yet), and the startup warning read only
plan.runtimes + pending_manual_serves: the warning vanished during a run and
after a run killed before its plan stage. Read carried rows too, and carry a
killed run's rows forward into the next run.
update_cmd_fleet.py had grown past the 2,000-line cap (2,230). The verify phase
(_verify_fleet_after_update, the settle-polled fleet snapshot, owed-inventory
recording, legacy-unit warning) is its own topic: it runs after the restart
phase and decides follow-up vs clear. It now lives in a sibling; restart-phase
helpers it uses are read through update_cmd_fleet so their test patches still
apply. update_cmd re-imports _verify_fleet_after_update from the new module.
Tests retarget their patches to where production reads. No behaviour change.
Review feedback (@kvnloo): when the post-restart fleet probe has no row for a
planned gateway's profile, the incarnation fallback silently does nothing and
the tripwire output reads identically to a genuinely missed restart. Log the
missing evidence at debug so the next report is diagnosable:
No post-restart gateway evidence for profile 'coder' (planned pid 76508 via
launchd) — reconciling on restart bookkeeping alone
Also make the call-site comment name the snapshot dependency the evidence
rests on (the same note #110343 carries at its wiring point).
`match_runtime_outcomes()` can only bridge a planned gateway runtime and the
restart bookkeeping by SERVICE NAME, and the two disagree whenever a service
serves a profile its name does not encode:
- macOS: the root-home LaunchAgent `ai.hermes.gateway` honours the install's
sticky active profile (`hermes profile use <name>`), so the process declares
`profile: coder` on its control socket while its label encodes the install
root. `_gateway_service_matches_profile("coder", "ai.hermes.gateway")` is
False for every naming rule, so the planned runtime is reported
`unaccounted` → `restart.incomplete` → exit 1 after a restart that verified
clean (fleet row: new PID, new SHA). The leftover `fleet_restart_pending`
marker then warns on every later invocation.
- Linux: a hash-suffixed unit for a custom HERMES_HOME has the same shape of
problem (#110238), which the open name-matching PRs only paper over for the
default profile.
Reconcile gateways on incarnation evidence too — the rule serve/dashboard
runtimes already use via `stale_serve_pids`: the post-restart fleet snapshot
(control-socket identity per profile) is passed in as `live_gateway_pids`, and
a planned PID that is gone while a gateway answers for the same profile counts
as `restarted`. Requiring a successor keeps the tripwire honest: no successor,
or the planned PID still answering, stays `unaccounted` and still exits 1.
`down` rows are excluded from the evidence — they report the pre-restart PID.
Fixes#110333
A profiles/<name> symlinked to a separate Hermes install is inventoried by the
update plan, left alone by the restart phase, and already classified
'external' by the post-restart fleet matrix. Reconciliation had no such
outcome, so the gateway ended 'unaccounted' and every hermes update exited 1.
Feed the fleet's verified external gateway pids into match_runtime_outcomes:
an untouched gateway in that set is 'external', reported informationally and
excluded from the exit-1 tripwire. No fleet evidence keeps it unaccounted.
Fixes#120240
Review minors on #126179:
(a) update_cmd_fleet._marker_only_restart_obsolete: a fleet whose every
row is EXTERNAL (served from another checkout root) fell through the
row loop and cleared the marker with zero evidence about this checkout.
For the SHA-less, inventory-less record it is now treated like an empty
fleet and stays armed.
(b) update_cmd_windows._hermes_holder_subcommand stripped only double
quotes before the inline-bootstrap check, so a shlex.join'ed
runtime_command (`python -I -c '<bootstrap>' dashboard`, the shape
launchd ProgramArguments take through main_dashboard) was never
inventoried. Strip both quote kinds, as _is_entry already does.
Follow-up trim of the #125964 salvage:
- Armer falls back to _current_checkout_sha() (the identity the reader compares the fleet
against) instead of a new receipt helper; at arm time the receipt has no post_update yet.
- Reader: the SHA-less, inventory-less record follows the existing inventory-less rule
(live fleet all current on the checkout) with no extra pid-liveness gate, matching the
invariant the SHA-bearing inventory-less record already has. The gatewayless branch stays
fail-closed for an SHA-less record (checkout_contains("") is not evidence).
- Drop the obligation_fields pid plumbing and update_receipt helper; keep 2 tests:
discharge on live-fleet evidence, and the armer recording the checkout SHA.
A no-op `hermes update` (pre == post SHA) whose head capture failed armed the
host update-restart record with `expected_sha: ""` and no inventory. The reader
bailed out on the empty SHA before the live-fleet reconciliation, so the
"gateways may still be serving pre-update modules" warning could never clear
even when every live gateway was current on the checkout.
Reader (hermes_cli/update_cmd_fleet.py):
- An inventoried obligation without its SHA stays fail-closed.
- An inventory-less, SHA-less record now falls through to the existing
live-fleet check once the process that armed it is gone (pid liveness via
pid_exists_stdlib), so a status call cannot discharge a restart phase that
is still running.
Armer (hermes_cli/update_cmd.py):
- When the completion request carries no SHA, fall back to the receipt's
post_update.sha (then pre_update.sha) instead of arming with "".
obligation_fields() now reports the arming pid alongside expected_sha.
Tests: discharge on all-current fleet, kept when the armer pid is alive, kept
on a stale fleet, kept with an inventory but no SHA; the discharge test fails
on the pre-fix reader with the issue's exact warning.
When the post-update dashboard respawn raised, the receipt derived the
runtime outcome from the incarnation liveness probe: the pre-update pid
being gone is exactly what a failed respawn looks like from the outside,
so the row read "restarted" and the overall receipt read "success".
The truth already existed on the producer side (the respawn path returns
the argvs that failed to spawn) and was dropped at the call site.
Thread it through: _restart_killed_backends now returns the PIDs whose
argv respawn was attempted and failed, _finish_dashboard_update_cleanup
returns them to both update paths, and match_runtime_outcomes checks
failed_respawn_pids before the incarnation probe — "we watched the spawn
fail" outranks "gone". Reuses the existing "failed" vocabulary (the unit
path already emits it) and escalates the receipt to "partial" like the
unaccounted-tripwire class. See #109290.
The systemd unit pass only listed and restarted hermes-gateway*/hermes-serve*
units, never hermes-dashboard*, although the runtime-outcome reconciliation
already credits hermes-dashboard{,-<profile>} unit restarts. A systemd-
supervised dashboard therefore stayed on pre-update code after every update:
the receipt recorded outcome 'deferred', nothing ever restarted it, and the
update still exited 0 (success). Same blind spot in the abort-recovery serve
pass, which enumerated hermes-serve* only.
Extend the unit list glob, the name gate and the recovery pass regex to the
hermes-dashboard* family (exact base unit or hyphenated profile, same strict
shape that rejects hermes-server.service). SIGUSR1 drain eligibility is
unchanged: dashboard units do not run gateway/run.py and keep going straight
to the blunt restart, like hermes-serve*.
Fixes#125297
Follow-up to the salvaged restart-wait commits:
- A drain or cron timeout of .inf now means "wait indefinitely" instead of
an OverflowError from the integer stop envelope, which crashed
`hermes gateway restart` and made `hermes update` silently fall back to
its 45s floor. The fleet "draining (up to Ns)" lines and the drain
progress report format the budget instead of int()-ing it, so an
unbounded wait no longer crashes them either (it did on main too).
- cron_drain_timeout is required: a 0.0 default meant "cron opted out",
the under-budget this fix exists to remove.
- Docstrings describe what the budget actually covers (PID exit, not
replacement startup).
- Tests assert the observer outlasts after-turn + the supervisor stop
envelope and that configured cron reaches the CLI wait, instead of
re-deriving the formula; the negative wording assertion on the pending
footer is dropped (change-detector).
The serve a remote Desktop spawns over SSH has no local spawner, so the
inventory read it as manual-serve: an update filed a manual-restart
reminder nobody on this host can discharge, reported the stale process
as unaccounted, and abort recovery could try an argv respawn without the
client's token file and owner nonce.
Classify it as desktop-ssh (using the canonical argv predicate, so rows
written before the ledger carried isolated are covered too) and treat it
like the local Desktop's own serve: skipped by the restart phase,
deferred to its client, never owed by abort recovery. A hand-started
serve --isolated stays manual-serve.
The fleet-restart-pending discharge parsed the marker inventory and
probed a gateway-less host inline (CC 36). _marker_owed_gateways now
turns the inventory into the owed set (raising ValueError, which the
caller's existing handler already maps to "keep the marker"), and
_discharge_gatewayless_marker owns the #118742 host probe. Every
verdict is unchanged; the caller is at CC 21.
- tools/browser_tool_install.py: keep pm-clean's frozen old-updater stub; main's
UTF-8 decode fix touched only the npx prefetch body it replaces.
- tests/hermes_cli/test_update_scoped_reconciliation.py: keep pm-clean's test
subset (catch-up rides the PM completion owner) and take main's gateway-less
host evidence (#120740): the updated seed that holds the host at a running
gateway, and the two gateway-less matrices for the source change that merged
cleanly into update_cmd_fleet.py.
* refactor(update): one predicate for serve rows outside the gateway matrix
The inventory branch of `_marker_only_restart_obsolete` inlined the rule for which
serve/dashboard rows the gateway matrix neither covers nor needs to (supervisor-owned,
or a manual serve handed to its own reminder). The inventory-less branch needs the same
rule for #118742, so it moves to `update_cmd_fleet_gatewayless.runtime_outside_gateway_evidence`
and both branches will read one definition. No behaviour change.
* fix(update): a Desktop-only host settles an inventory-less restart obligation
A host that runs no gateway (the Desktop app alone) can be left with an inventory-less
fleet-restart obligation: an updater that died before recording its inventory, or the
pre-inventory writer. With no owed set, the live gateway matrix is its only evidence, and
on that host the matrix is empty forever, so `_marker_only_restart_obsolete` never settled
and every later `hermes update` exited 1 with "gateways are still off the checkout code"
(#118742).
An empty fleet alone cannot tell that host from one whose gateway the dying update stopped,
so the inventory-less branch now asks the live host, never a historical receipt:
`host_owes_no_gateway_restart` settles only when no profile's `gateway_state.json` claims a
state other than stopped/startup_failed (a gateway that went away without a clean stop keeps
the obligation) and every live runtime sits outside the gateway matrix
(`runtime_outside_gateway_evidence`, shared with the inventory branch). HEAD must still
contain the pulled SHA (`checkout_contains`, same rule as #119367). Probe failures keep it.
Tests: the scoped-reconciliation matrix now holds its host at "the update stopped a gateway"
so it keeps pinning receipt independence; a new host-evidence matrix covers Desktop-only,
clean stop, carried commit, stopped gateway in a named profile, unclassified and
unidentified serves, a gateway row without fleet identity, and a diverged checkout. Two
manual-serve tests that assumed an empty fleet always stays pending now stub the live host
and assert the manual reminder survives the gateway obligation settling.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
---------
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
`_marker_only_restart_obsolete` bailed out with "a newer pull moved HEAD" whenever
`checkout_sha != expected_sha`, and otherwise held the fleet to `expected_sha` exactly.
A cherry-picked hotfix on top of the pulled SHA moves HEAD without any pull arming a
fresh obligation, so the recorded one could never discharge: every CLI start warned
and every no-op `hermes update` exited 1 through the armed veto in
`_pending_fleet_restart_needed`, while the gateway verifiably served HEAD.
The bail-out now fires only when HEAD no longer CONTAINS `expected_sha`
(`checkout_contains`, `git merge-base --is-ancestor`, fail-closed on any probe
failure), and the live fleet is held to the checkout SHA — the code it actually runs.
A gateway on genuinely stale code still keeps the obligation armed.
Closes#119367
`hermes update --yes` from a cron job inside the gateway takes the #100179
ancestor branch (`_drain_or_signal_gateway_for_update` -> self-restart request,
fire-and-forget) — the gateway can only restart after the updater exits. The
post-restart fleet matrix then saw that same gateway on the pre-update code_sha,
printed STALE + "Update not complete" and exited 1 on every nightly run (#119597).
The restart phase now records the pids that accepted a self-restart
(`_GatewayRestartOutcome.self_restart_pending_pids`, threaded from the systemd,
manual and launchd paths) and `collect_fleet_versions(self_restart_pending=)`
turns exactly those rows into `restart_pending` — rendered as "restart pending
(deferred until this process exits)" and excluded from the stale_or_down
verdict. Every other stale/down gateway keeps its verdict; the same pid without
the recorded acceptance is still STALE. The receipt keeps the row's old sha, so
the next `hermes update` / CLI startup hint still verifies the restart landed
via the live fleet (`_receipt_reports_stale_runtime` -> `_live_fleet_covers_receipt`).
Closes#119597
hermes-gateway*/hermes-serve* units, ai.hermes.gateway* LaunchAgents and
`gateway run` processes are account-wide namespaces shared by every Hermes
install on the box. The restart phase enumerated them by name, so a scratch
home's `hermes update` drained and restarted the account's real
hermes-gateway.service and SIGTERMed sibling installs' gateways (live: #93349
comment, 2026-09-23), then warned "Fleet version check returned no rows"
because none of those runtimes belonged to the updating home.
Ownership is now judged from what a runtime actually runs on, never from the
label: the live process environment (`_hermes_home_for_pid`), the unit's
declared Environment=HERMES_HOME, or the plist's pinned HERMES_HOME, compared
against the homes the update's plan inventories (the updating root and its
profiles/<name>). Foreign or unreadable ownership is named in the output and
left alone; it is not a failed restart. Applies to the systemd fleet loop, the
catch-up best-effort restart, the launchd derived-label loop, the manual
gateway sweep, and the pre/post-restart PID snapshots that drive the
fail-closed verdict.
Update stale test assumptions for PM install hints and terminal heartbeat,
keep Unix socket fixtures below AF_UNIX path limits, and remove timing and
host-process races from gateway tests. Read systemd units with BOM-tolerant
UTF-8 so the Windows footgun scan remains clean.
A pre-2026.7 `Restart=on-failure` system unit has no
RestartPreventExitStatus=78, so a permanent refusal restarted it ~180
times while the regenerated user units parked (#118282). The gateway
refreshes its USER unit at boot (`gateway run` -> refresh_systemd_unit_if_needed
(system=False)) but nothing ever rewrote an installed SYSTEM unit unless
the operator ran `sudo hermes gateway install|start|restart --system`.
The update-time fleet restart is the only routine contact with that unit:
as root it now regenerates the stale unit before the drain; without root
it names the exact repair command. A unit that already parks on 78 is
left alone. HERMES_HOME is restored after the refresh so the rest of the
update keeps running for the invoking profile.
Closes#118282
Every update route now finishes through update_completion._complete_selected,
which restarted the whole fleet unconditionally -- including the "Already up to
date" route that main sent through the pending-restart catch-up. Net effect:
each cron tick and each profile's `hermes update` drained and re-killed the one
multiplexed gateway.
Port the catch-up path's two live guards into the completion tail:
- host_restart_already_completed(checkout sha): a sibling profile attaches to
the restart this host already stamped (#95294); the restart phase now stamps
it via mark_host_restart_completed.
- every planned runtime AND every live fleet row current at the checkout sha
(#117051, d6b0d37ece). Both are required: the live matrix lists gateways
only, so a planned serve still on pre-update code keeps the restart.
The already-current route also arms the host obligation with the checkout sha;
an SHA-less arm replaced the standing record and wiped the restarted proof.
Host-scoped update-restart obligation (061195fac1 / 953b6f6f08 / 3be255eca6)
lands on the PM model: the obligation record, its readers and the legacy
per-home marker compat come in as-is. The catch-up restart path
(`_apply_pending_fleet_restart_catchup` / `_run_pending_fleet_restart`) is
retired here (the fleet restart rides the completion owner), so main's
per-host restart-once guard on that path is not carried; its unit→live
MainPID collapse IS ported into the live post-update systemd pass
(`_restart_systemd_gateway_units`), with the two collapse tests rewritten
against that function (red on the pre-port tree: `_unit_main_pid` absent).
Tests that exercised only the retired catch-up path are dropped.
Desktop: main's shared log-rotation planner replaces the inline constants
in main.ts; the merge keeps our machine-profile import beside its import.
utf-8 → utf-8-sig on the three new BOM-intolerant reads (footguns lint).
Review findings on the host-scoped update→restart obligation.
- update_cmd_fleet: an unwritable host state dir (read-only HERMES_GATEWAY_LOCK_DIR,
container UID that does not own $HOME) made write_host_obligation return False and the
caller ignored it, so an interrupted update left ZERO obligation — stale code, no
warning, no catch-up restart. The return is now propagated: the legacy per-home marker
(still read by every reader) carries it, and a host that can write neither says so.
- update_cmd_fleet::_obligation_fields: a PRESENT but unparseable/foreign-versioned host
record no longer falls through to the legacy marker; terms nobody can read cannot be
discharged by another record's terms.
- update_cmd_fleet::_restart_identity_sha: zip/pip/Docker installs resolve no checkout
SHA, so the restart-once stamp was "" and could never match — every profile's update
re-killed the one shared multiplexer. Falls back to the record's expected_sha, then the
receipt's post-update identity.
- update_host_obligation: any main_pid probe error is unproven identity (keep its own
restart), never an aborted restart pass.
- run_notifications: the online notice dedupes per home CHAT, so two served profiles
sharing one chat get one message (accounting stays per profile); transport resolution is
isolated per profile, so one broken adapter no longer starves the rest of the fan-out.
- run_adapters / run_profile_reconcile: _profile_configs is pruned with the served set, so
a failed or removed profile no longer owes a notice nothing can deliver and
.restart_pending.json is unlinked.
Tests cover the new format's own hazards: unwritable record dir, foreign-version record
with a legacy marker present, non-git install, shared home chat, broken adapter, pruned
config, plus a parity test for the duplicated host-state-dir resolver.
Teknium