hermes-agent/website/docs/developer-guide/cli-internals.md

sidebar_position, title, description

sidebar_position	title	description
15	CLI Internals	How hermes_cli is shaped: slash dispatch, config loaders, the skin engine, the transactional update pipeline, and process-identity rules

CLI Internals

Companion to hermes_cli/AGENTS.md (the rules) — this page holds the longer explanations.

Update pipeline

User-facing behaviour (receipts, --plan, snapshot modes) is in Updating; hermes_cli/AGENTS.md carries the short form of the rules.

Fleet-update campaign #91277 (Aug 2026). A PR that weakens a stage must answer for the failure class it guards. plan → snapshot → apply → restart-per-kind → verify → report

• Plan (update_inventory.py, hermes update --plan): read-only inventory — install kind, all profiles, every live gateway with supervisor + running code version. Deployment kinds are first-class: git updates in place; docker/nix/apt are NOT in-place-updatable and the updater reports the correct external command instead of fighting the deployment model.
• Snapshot (backup.py): pre-update quick snapshot for EVERY profile (the code swap + fleet restart touch all of them), each into its own state-snapshots/, identical file set, 1 GiB per-file cap, keep=1. Never add a partial/tiered snapshot set — mixed coverage creates torn-restore states across schema generations. Quick snapshots are FILE-LOSS RECOVERY (the per-profile cron-jobs safety net restores from them), NOT code-rollback insurance; --backup full mode owns rollback.
• Apply: git pull, or the Windows ZIP fallback — which fires ONLY when git itself failed (_should_zip_fallback_on_update_error, argv-classified; a dependency-install failure must never trigger a tree-clobbering re-download), REFUSES a dirty working tree (-uall + a pre-swap TOCTOU re-check — but classifies a !! line by whether the swap would destroy it: an ignored path under a root entry the ZIP does not ship (.bytecode-fingerprint, .hermes-bootstrap-complete, hermes_agent.egg-info/; tracked root entries stand in for the ZIP set before the download, the re-check gets the real one), a nested __pycache__/node_modules, or a _ZIP_PRESERVED_NESTED output is admitted; other ignored files under shipped dirs still block), and grafts the live nested build outputs (_ZIP_PRESERVED_NESTED: apps/desktop/{release,dist,node_modules,build}, hermes_cli/web_dist, ui-tui/{dist,node_modules,packages/hermes-ink/dist}, web/node_modules, scripts/whatsapp-bridge/node_modules) into the staged swap by hardlink (the GitHub source ZIP has none of them; without the graft the swap deletes them). Post-swap, the Desktop rebuild decision also trusts the build stamp under HERMES_HOME, so an install that already lost its artifacts in an earlier update is rebuilt instead of "forgotten" (#90495).
• Restart-per-kind: systemd and launchd restarts are FLEET-WIDE within the updating install (every hermes-gateway* unit / ai.hermes.gateway* LaunchAgent whose home is the updating root or one of its profiles/<name>), drain-first (SIGUSR1), with per-unit/per-label failure isolation. Restarting only the invoking profile's service leaves siblings on stale sys.modules until they crash — the largest dupe-PR cluster in the repo's history came from that bug. The fleet is bounded by HOME, not by namespace: hermes_cli/update_fleet_scope.py judges every unit/label/process by the home it actually runs on (live environ, unit Environment=, plist HERMES_HOME), and a runtime of another HERMES_HOME on the same account — a sibling install, the real hermes-gateway.service seen from a scratch home — is named and left alone, never restarted (#93349).
• Verify: gateways stamp code_sha/code_version into gateway_state.json on every runtime-status write (gateway/status.py); the updater compares each live gateway against the fresh checkout and prints a fleet version matrix. After the commit point (the tree moved) nothing fails hermes update (exit 0 unless the code was rolled back): a provably-stale or stopped gateway, a failed build/launcher/config migration/bytecode sweep/channel adoption/Windows resume each print ⚠, land in the receipt as outcome: "success" with followups: [{step, reason}], and keep their own obligation armed (source-completion-pending for the tail, the fleet-restart obligation for gateways) so the next launch or hermes update — including "Already up to date" — retries them. The owed-fleet-restart CLI startup warning is what keeps a mixed-version fleet from ever looking healthy. A completion process that could not be started (temporary directory, request file, spawn) or died without a correlated result (OOM, SIGKILL) is the same kind of debt: an owed completion follow-up with the source-update tail re-armed, exit 0. Exit 2 (refused/concurrent) and exit 1 (nothing committed / rolled back) keep their meaning; the one committed run that exits 1 is an autostash whose restore conflicted and stays parked: the receipt is partial with a user_action, since nothing retries what only the user can re-apply (#122557).
• Report: every run writes a machine-readable receipt to the ROOT home's logs/update_receipts/ (never a sticky profile's; update.log likewise) — latest.json pointer; steps, skips WITH reasons, restart outcome, plan, fleet snapshot, followups. The receipt is on disk as outcome: "running" from the start and refreshed at each stage boundary; the next update marks a running record whose processes are gone interrupted and says so. Readers (hermes logs update, debug bundle, dashboard status, pm receipts) use the root home too. A dependency-sync failure in the completion bootstrap is a dependencies follow-up (A6), and Ctrl-C after the commit point finalizes interrupted (exit 130), never failed. Before a source swap, the parent captures plan/snapshots/receipt and its Windows pause token. update_completion.py runs new-code PM preparation with site initialization disabled, then selected-Python builds, maintenance, scans/restarts and verification. Git/current/ZIP share this owner; never reload or purge modules to continue in the old interpreter. The parent keeps the lock, waits, and accepts only a correlated terminal result. cmd_update still finalizes early failures and missing/killed-child outcomes; PM refusal data survives the handoff. See website/docs/developer-guide/source-update-completion.md. A begun-but-unwritten receipt is a bug.
• Nothing runs pulled code in the pre-pull interpreter. This tree finishes updates through update_completion.run_completion / _update_takeover.py. hermes_cli/update_handoff.py and hermes_cli/update_serve_obligations.py are the FROZEN COMPAT SURFACE for releases that lazily import those module names from the NEW tree after the checkout swap. Keep their public names importable and behavior-preserving. They call into _old_updater.stop_for_relaunch → _run_child. Removing a name bricks every release mid-update. see tests/compat/old_updater_surface.json.

Process-scan coordination between updater, serve/dashboard, and gateway is being replaced by a gateway-owned control socket (#92091); scans are the fallback layer for old/crashed processes — read #92091 before adding any heuristic. Process identity rules (never argv substrings; canonical matchers; parser-derived flag sets; never blanket-exclude gateway ancestors, #87594): root AGENTS.md and website/docs/developer-guide/cli-internals.md.

The systemd blunt-restart fallback waits for the unit's TimeoutStopUSec plus TimeoutStartUSec, with 15 seconds of client-side slack. It reads the target unit in the same manager scope as the restart; both the initial attempt and retry use this budget, including the catch-up restart after an interrupted update. A start after a graceful drain uses only the start budget plus slack. A missing, unparseable, or infinite phase limit falls back to 90 seconds for that phase, keeping unattended updates bounded. Timing out the systemctl client does not cancel the manager's transaction. Custom multi-command stop chains or EXTEND_TIMEOUT_USEC can still outlast this estimate; a real timeout remains an incomplete restart, and successful commands still require the existing service-health and fleet-version verification. Raw numeric *USec values are microseconds, while formatted values use systemd's fixed units, including days, weeks, months and years. The combined timeout is capped below the native signed 32-bit millisecond poll limit (with rounding headroom), so exceptionally long unit limits cannot overflow subprocess polling. Zero/unknown/infinite phase limits use the bounded fallback. This does not change active-turn drain settings.

Nous sign-in

Every Nous sign-in is the plain device-code login. Sign-in completion is one function, settle_after_upgrade, called once per completion: it moves a config on the welcome route to the account's host and the tier's recommended default (models.recommended_nous_default_model, shared with GET /api/model/recommended-default).

The shared flow, states, and copy live in anon_sign_in.py; CLI rendering lives in anon_sign_in_cli.py. anon_auth.py keeps identity and settlement, and re-exports the sign-in API. The flow resolves identity and persistence collaborators through anon_auth at call time to preserve module-attribute monkeypatch seams.

The sign-in itself is one composition: anon_auth.run_sign_in() yields SignInStates (Code, Waiting, Completed, Cancelled, TimedOut, Failed, AlreadySignedIn). It requests a device code, polls the token endpoint on one deadline (an expired code is TimedOut), saves the account, runs settle_after_upgrade exactly once, and never lets a persist or settle failure escape as an exception: it becomes Failed. It works with or without a free tier on disk. Completed.email comes from the access token's email claim; Completed.handoff carries the guest handoff's result lines (what moved, what to reconnect) and Completed.handoff_status the latest record as fields (guest_handoff.wire_status: the public view plus those lines), for a surface that renders the result itself. Every state carries its own .copy (the chat form, which never contains a raw exception, a URL or a hermes verb) and .copy_terminal, so no caller maps a reason to a string. cancelled() is checked at least once a second inside the token poll and once more before the save; a cancel always wins and nothing is saved. TLS verification comes from _resolve_verify, so HERMES_CA_BUNDLE and friends apply. scope is entered only around the precondition and persist blocks, never across a yield or a network wait, because run_in_executor does not carry contextvars. replace_account=True lets a signed-in install sign in again (the desktop's "connect another account"); otherwise a signed-in install gets AlreadySignedIn before any network. upgrade_guest (hermes auth upgrade), the CLI and TUI /login, the gateway /login and the desktop start route are renderers over it; a surface that needs the cancel check and the save to be atomic passes persist_guard. hermes model and hermes auth add nous keep _login_nous (it carries --portal-url, --ca-bundle, --insecure and the shared-store import); that is the same plain device login, and it prints the handoff result the same way. Every sign-in that saves an account (run_sign_in, _login_nous, hermes auth add nous, the billing step-up, the tool-setup login) calls settle_after_upgrade itself once its locks are released; the handoff runner never settles.

Guest handoff after a sign-in

When any writer saves a Nous account over a free-tier identity, hermes_cli/guest_handoff.py hands that identity's connected apps to the account: POST /api/anonymous/claim {"token": "<anon_...>"} with the account's own Bearer token. Nothing asks the user first; the result is shown afterwards (hermes auth status nous; the desktop reads it from the sign-in poll's handoff_status and, while a move is still retrying, from the GET /api/providers/oauth/nous/handoff?since=&wait= long-poll, which answers once the record's updated_at changes). Sign-in code does not call it:

• note_replacement runs at the two lowest write seams (auth._store_provider_state and auth_nous._write_shared_nous_state) under the caller's auth-store locks and only writes a pending record. Records live in guest_handoff.json beside the shared Nous store, keyed by a hash of the anon_ credential and bound to the destination account, so profiles sharing one free-tier identity share one record and one claim.
• Only a sign-in changes which identity the shared store holds. Every _write_shared_nous_state caller declares renewal: a token refresh (renewal=True) never replaces another identity there (identity = the guest's anon_ credential, or the account's JWT sub), so a refresh of one profile's account never replaces a guest another profile minted. An empty store still takes a refresh, which keeps profiles that share one account's rotating refresh token in step.
• drain runs the claim outside every lock, bounded to about 10 s (each record, lock wait, token read and request counts against the budget): after a shared-store write, when a guest_handoff.deferred() block exits (a surface that saves under its own lock, such as the desktop poller, wraps the save in it), and after a free-tier reconcile. It only moves apps; config completion (settle_after_upgrade) stays with the sign-in that saved the account.
• Retries left by any process are finished by a retry worker that every long-lived Nous process starts (start_nous_auth_keepalive starts it, guest or not). Each tick runs every due record under its owning profile's home and secrets, so a named profile's move retries from a process launched as another profile.
• Every attempt is reserved in one locked read-modify-write that checks the record is still due, inside its retry window, and bound to the account whose Bearer is about to be sent (the token's sub must equal the record's destination). Apps never go to any other account.

Answers: claimed records what moved (and refreshes the account token only when cohort_carried; an answer without connectors reads as unknown, never "nothing moved"); unknown_token is Gone, read as "probably moved" after an earlier attempt (a lost success); already_retired is Gone too; a 401 refreshes the account token once. Only codes no later attempt can change are final (client_not_allowed, account_locked, invalid_request). Every other refusal, unknown codes included, and every transport failure retries with backoff (Retry-After honoured), the slowest rung repeating, until 13 days after the record was made: the service reaps an idle free-tier identity, apps included, after 14. A record an older build failed on a transient code re-arms on the next run (a sign-in, hermes auth status, hermes auth upgrade, the retry worker). A Claimed record is never reopened, and user copy never shows a server code.

Process identity: never infer it from argv substrings

The bug class behind ~10 fleet-update issues (#90778, #87594, #78089, #76129, #91964, ...): classifying a process by "serve" in cmdline or similar. kanban --preserve-cache contains "serve"; a flag VALUE can equal a subcommand (-m dashboard serve); truncated cmdlines hide the real subcommand. Rules:

• Use the canonical matchers: gateway.status.looks_like_gateway_command_line (gateway run), hermes_cli.update_cmd._hermes_holder_subcommand (top-level subcommand of any Hermes argv). Never hand-roll token scans.
• Flag sets must be DERIVED from the parser (_holder_value_flags() introspects build_top_level_parser()), never hand-written lists — they drift.
• Never blanket-exclude ancestors from process scans: when /update runs as the gateway's child, a gateway ancestor must stay visible to the pause machinery (#87594). Exclude interactive ancestry, carve out gateway-shaped ancestors.
• Match on FULL cmdlines; truncate only at display time (#78089).
• Before adding any new scan heuristic, read #92091 — the gateway control socket replaces scans as the primary coordination mechanism; scans are the fallback layer for old/crashed processes.

Skin engine — what skins customize

Element	Skin key	Used by
Banner panel border / title / section headers / dim / body	colors.banner_border, banner_title, banner_accent, banner_dim, banner_text	banner.py
Response box border	colors.response_border	cli.py
Spinner faces (waiting / thinking)	spinner.waiting_faces, spinner.thinking_faces	display.py
Spinner verbs / wings (optional)	spinner.thinking_verbs, spinner.wings	display.py
Tool output prefix / per-tool emojis	tool_prefix, tool_emojis	display.py → get_tool_emoji()
Agent name / welcome / response label / prompt symbol	branding.agent_name, welcome, response_label, prompt_symbol	banner.py, cli.py

Built-in skins (_BUILTIN_SKINS in hermes_cli/skin_engine.py): default (classic gold/kawaii), ares (crimson/bronze with custom spinner wings), mono (grayscale), slate (cool blue). Add a built-in as a dict entry {"name", "description", "colors", "spinner", "branding", "tool_prefix"}. User skins are ~/.hermes/skins/<name>.yaml with the same keys, activated with /skin <name> or display.skin: <name>; the full YAML template is in the Skins & Themes user guide.

Profiles: multi-instance support

Hermes supports profiles — fully isolated instances, each with its own HERMES_HOME (config, API keys, memory, sessions, skills, gateway). For single-profile commands (hermes -p x <cmd>), _apply_profile_override() in hermes_cli/main.py sets HERMES_HOME before any module imports, so every get_hermes_home() reference scopes to the active profile. The multiplex gateway and the Desktop/dashboard serve backend serve several profiles from one process instead: the active profile is a contextvar override bound per activity, os.environ["HERMES_HOME"] stays the launch profile's, and a module-level constant derived from the home freezes to that launch profile (see Gateway Internals § Multiplexed profiles). Profile operations are HOME-anchored (_get_profiles_root() returns Path.home() / ".hermes" / "profiles", not get_hermes_home() / "profiles") so hermes -p coder profile list sees all profiles regardless of which one is active — intentional. Profile-safe coding rules are in the root AGENTS.md; multiplex secret-scope rules in gateway/AGENTS.md.