A managed worker (every Kanban card, --safe-mode and --ignore-user-config
chats) built AIAgent from the frozen model.base_url with no api_mode, and
resolved the route only to fill a missing key. The in-process turn takes
api_key, base_url and api_mode from route_runtime(), so a route whose
api_mode or endpoint only resolution knows failed only in the worker:
opencode-go Responses models fell back to chat_completions (400), and
Bedrock, which stores no base_url, landed in us-east-1 where eu./apac.
inference profiles do not exist.
worker_route() runs the same route_runtime() call with the bound launch
key (if any) and returns api_key, base_url and api_mode for the agent. A
bound launch key still wins; on a fallback entry nothing is carried, as
before (#139155). Resolution stays in the worker's own profile home: no
new secret path.
Refs #143547
Co-authored-by: Kazantcev Andrey <45011689+heckad@users.noreply.github.com>
Co-authored-by: Gekiere Aurélien <11493426+aurel282@users.noreply.github.com>
On Linux psutil's create_time() is the cached /proc/<pid>/stat start ticks
plus boot_time(), and psutil 7.2.2 re-reads /proc/stat btime on every call.
WSL2 re-anchors btime after host sleep, so the owner's read of a live
worker's birth drifted from the one the worker reported in its hello, and
the exact-equality fences (reservation, worker.adopt/persist, delegation
dispatch, the launcher signal and the Kanban owner receipt) refused it:
every Kanban card, cron run and subagent ended permission_denied.
worker_birth() reads the kernel start ticks since boot
(create_time(monotonic=True)) on Linux and keeps process_birth() elsewhere;
every managed-worker identity fence now uses it. The spawn ledger keeps
process_birth() so its persisted epoch births still match after an update.
An owner_admitted receipt written before this change reads as another
process: its reclaim pauses, the fail-safe direction.
Refs #143547
Co-authored-by: Sebastian Hänisch <5960303+TKCen@users.noreply.github.com>
agent/managed_worker.py::RESULT_FIELDS dropped usage_window_message, so a
window refusal on a managed-worker session reached the owner as a plain
billing failure and the error card fell back to the generic out-of-credits
card. The marker is a bounded scalar like failure_resets_at; usage_limit
(nested lists) still does not cross the worker frame, so that route shows the
window card without the suggested-model buttons.
Most failed turn results (agent/turn_recovery._failed_turn_result and the
other early exits) name no provider or model, so the descriptor the
authority built for message.complete had no auth_kind / api_key_env and the
Desktop's auth card offered no action. The TUI path reads both from the
agent; the authority now does too.
- run_turn_runner: the projection records the agent's provider beside its
model (a result that names one still wins).
- managed_worker: the worker's result frame carries the agent's provider and
model unless the result already names them.
A turn that failed on the session authority completed with status 'error'
and no error_surface, so the Desktop showed the generic card (with a Retry
that fails again against a full free tier) instead of free_tier_at_capacity,
rate_limit with its reset time, auth, and so on.
- run_turn_runner: the projection keeps free_tier, failure_retryable and
failure_resets_at, the fields build_error_surface_from_result reads.
- session_settlement_recovery.completion_payload: every failed completion
builds the descriptor from its result; a gateway refusal's prebuilt one
(session_cron) still wins.
- managed_worker: the worker result frame carries the same fields, with
free_tier validated like billing_block.
Since #106742 every Kanban card runs in a gateway managed worker, and
ef52a09a55 removed the delegation toolset there: a delegated child needs
its own owner-registered execution (review r4224751560 / r4230769955), and
_open_child_session_db refused with worker_child_registration_required.
The rule stays; the missing owner side is built:
- child.reserve (gateway/session_worker_child.py): a worker asks on its own
fenced receipt stream. The owner verifies the parent's claim, mints the
child's session row (source subagent, parent lineage, parent profile,
_delegate_from marker) and a kind='child' execution bound to the parent's
verified pid/birth, the worker's fresh secret and the connection
principal, under the parent's id prefix. Caps come from the root
session's frozen policy (delegation.max_concurrent_children,
max_spawn_depth); a refusal or draining owner is a receipt, so the
parent's journal never sticks.
- ChildSessionStore (agent/runtime_session_child.py): the child persists
through ordinary worker.persist under its own claim, writes only its own
row, never binds the process delegation ledger, finishes on close.
- Retirement: every terminal transition of an execution (finish, loss,
settlement) retires its live children and ends their rows; an owner
restart retires old-epoch children (never adoptable) while the root alone
becomes unknown.
WORKER_DISABLED_TOOLSETS is gone because the only child path in a worker
now goes through the reservation.
Fixes#141793
On macOS psutil shifts create_time() by abs(INIT_BOOT_TIME - boot_time()),
using a boot-time reference frozen when psutil was imported in the
calling process. After a clock step (NTP, sleep/wake) a long-lived
gateway owner reads a different birth for a fresh worker than the worker
reports itself, so verify_worker_identity's exact match refused every
managed worker with permission_denied until a gateway restart.
hermes_cli.process_identity.process_birth reads the unadjusted kernel
start (create_time(monotonic=True)) on macOS and plain create_time()
elsewhere (Linux and Windows apply no per-process reference). Every site
that writes or checks a worker birth now uses it: the worker's hello,
the reservation, the worker.adopt claim, the delegation dispatch check,
the verified-interpreter signal, and the Kanban owner binding with its
reclaim check. The match stays exact, so a forged birth is still refused.
#140612 undoes the shift arithmetically (birth - diff). psutil applies
abs(diff), so for a backward step that doubles the error instead of
removing it, and the other birth sites stay exact on the drifting value.
The new test emulates both step directions on every platform.
Refs #134889 (KB-6)
Refs #140580
Refs #140719
Local sessions that execute out of process (safe mode, config-only, the custom-provider opt-in) settled message.complete without the billing block or session usage an in-process local turn carries, so their viewers never showed the billing banner or the turn-end status bar.
The worker's result frame now carries billing_block as a bounded flat record, validated field by field at the owner boundary. The owner marks the turn's result local_viewer, and since the worker exits with its turn, it reads the session-cumulative counters from the session row the worker's usage writes just committed. The real-subprocess finished-worker test asserts the billing block, failure reason and usage reach completion_payload.
WorkerRPC.close() had no production caller and RuntimeSessionStore.close() does not close the RPC, so the owner socket lived until process exit. execute() now closes it after store.close() in its finally, on success and on a start failure; a failing close is suppressed so it never replaces the turn result or the start-failure exit.
Refs #134889 (KB-3)
Refs #139162
Refs #139011
worker_api_key() walked only the frozen session snapshot's fallbacks on a
primary AuthError. For an ordinary session the owner sends the refreshed
profile chain in construct_v1, so an expired primary (e.g. a lapsed nous
login) with a fallback added after session creation now failed to start,
where the key-less agent used to reach that fallback at init.
Walk the chain the agent will actually get: construct_v1's when the owner
sent one, the frozen snapshot's otherwise (older owner). A resolved fallback
still yields api_key=None, so its key is never paired with the primary
endpoint; when no entry resolves the primary AuthError is re-raised and the
Kanban start exit (75/78) is unchanged.
Refs #134889 (KB-5)
Refs #139162
Refs #139155
A managed worker launched without a launch key (every Kanban run) handed AIAgent
api_key=None. On the anthropic_messages wire only native Anthropic recovers a
missing key, so a custom provider (MiniMax and other Anthropic-protocol hosts)
sent `Authorization: Bearer ` and died as a transient APIConnectionError.
Move the policy-route resolution out of _resolve_policy_agent_runtime into
session_worker_construct.route_runtime and call it from both the in-process
turn and the worker's agent construction. The worker resolves inside its own
profile environment (the owner's _worker_env), so it never reads another
profile's key, and a key from a fallback entry is never paired with the
primary provider. A bound launch key still wins. Resolution errors stay inside
_construct_agent, so Kanban start failures keep their exit mapping.
Refs #134889 (KB-5)
Refs #139162Fixes#139155
Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
`chat -q` / `-z` over the session authority: the finite viewer detaches on the first prompt,
so a clarify question parked the turn until agent.clarify_timeout (an hour by default) with
nobody able to answer. Main's single-query CLI answered immediately with
_single_query_clarify_callback; finite approvals were already headless
(_is_single_query_approval_context reads the same finite flag).
The in-process turn (_clarify_callback_sync) and the managed worker (WorkerControls.clarify)
now return the same reply, byte-identical to main's: outcome "undelivered" with the
single-query notice. Interactive turns are unchanged.
(cherry picked from commit 6dc1ff23401f70cbeb44a6a00051bf6679fe4fb4)
Mechanism: WorkerControls._ask waited a fixed Event.wait(3600) for the owner's
clarify answer, bypassing agent.clarify_timeout / legacy clarify.timeout, which
tools.clarify_gateway defines as the timeout for every surface (<= 0 = unlimited).
A configured 7 s held the worker and the session FIFO for an hour; unlimited was
truncated to an hour.
Fix: resolve the timeout through tools.clarify_gateway.get_clarify_timeout(), which
reads the worker's frozen config snapshot (load_config returns it once the worker
policy is bound), and map a nonpositive value to an unbounded wait.
Test: tests/gateway/test_managed_worker_clarify_timeout.py (red on r14-base: the
1 s config still waits; 0 / -1 pass 3600 instead of an unbounded wait).
Review: andrexibiza 4243963765
Mechanism (N84 follow-up): the canonical Stop now hard-cancels in-process
turns (76196ee5c56), but a managed worker (Kanban, opt-in CLI sessions)
received the same Stop as agent.interrupt(): soft, so an auto-compression or
summary call in the worker kept running. Both the live-agent path
(WorkerControls.stop) and the Stop latched during agent construction were soft.
Mechanism (B1-11 follow-up): publish_execution no longer reads the ledger
(4e4ce2693e), but execute_managed still ran check_approval_generation
(db.get_session + a decode of every pending admission) once per worker frame
on the owner loop.
Fix: both worker Stop sites call request_hard_interrupt (managed workers run no
background delegations, delegate_task is off there, so no
interrupt_for_session). The frame loop compares authority.executing, set with
the claim's stamp and cleared when execution returns, the same fence
publish_execution uses; a stale generation still raises stale_generation.
test_managed_worker_finished_exit now stamps the claim like the drain does.
Test: tests/gateway/test_managed_worker_hard_stop_frame_fence.py is red on
r13-integ1 (soft stops; 19 ledger reads for 20 worker frames).
Review: dokterdok N84
Review: ahrazzle B1-11
Every Kanban run (and an opted-in CLI session) now executes in a managed worker, which builds
its AIAgent from the frozen session config. On main a Kanban run was `hermes chat -q`, whose
HermesCLI._init_agent passes provider_routing (only/ignore/order/sort/require_parameters/
data_collection), openrouter.min_coding_score, agent.service_tier and a static fast tier's
request overrides. The managed constructor passed none of them, so a worker silently routed
to providers the profile excluded (data_collection: deny included) and dropped priority
processing. Same class as the checkpoint limits (85b0db45ba9): a constructor input the
in-process path resolves from config that the worker's construction never forwarded; a diff
of the two AIAgent(...) keyword sets left only these (the rest ride construct_v1 or are
in-process-only display/callback wiring).
Fix: session_worker_construct.routing_agent_kwargs(config, model, provider, base_url) reads
them from the frozen config exactly like the CLI, and the worker spreads it into AIAgent.
Test: tests/gateway/test_managed_worker_routing_kwargs.py (fails on r11-base: every routing
kwarg is None).
Review: andrexibiza 27 (class sweep: managed-worker construction inputs)
(cherry picked from commit 07d8ebe691a2e428b1663b195f4d32d58dada38c)
A managed worker relied on cooperative shutdown: on owner death its control
pipe hits EOF and WorkerControls interrupts the agent, but a tool that ignores
the interrupt kept the worker (and whatever the tool spawned) running
indefinitely after the gateway that owned the session was gone.
POSIX: the owner now spawns the worker with start_new_session (it leads its
own process group), and a control-pipe EOF/OSError (owner gone; a closing
owner signals the worker itself) arms a non-daemon timer that, after
ORPHAN_GRACE_SECONDS (10s) for cooperative shutdown, SIGKILLs the worker's
own group and exits. Tools that open their own session (terminal, background
processes adopted across turns) are deliberately outside it. The kill runs
only when the worker leads its group. No preexec_fn (spawned from a worker
thread). Windows: the worker exits itself after the same grace; the gateway
process is already in a kill-on-close job (attach_self_to_kill_on_close_job),
which the worker inherits.
Test (real processes; red on r11-base agent/managed_worker.py: the orphaned
worker outlived its owner):
tests/gateway/test_managed_worker_parent_death.py::test_worker_and_its_descendants_die_when_the_owner_is_killed
Review: andrexibiza 29
Review: andrexibiza 5479692359-P3.g
(cherry picked from commit a994e46a6daab7e6bb9479380d8fbb8a57d5d304)
The in-process turn passes all four checkpoint arguments to AIAgent through
gateway.run._checkpoint_agent_kwargs (enabled, max_snapshots,
max_total_size_mb, max_file_size_mb). The managed worker forwarded only
checkpoints_enabled=policy.checkpoints_enabled, so a worker-run session
ignored configured limits and used run_agent's 20/500/10 defaults.
The translation now lives in gateway.session_worker_construct
(checkpoint_agent_kwargs, re-exported from gateway.run under its old name so
gateway/run.py shrinks) and the worker applies it to the session's frozen
config snapshot (policy.config(), which already carries the --checkpoints
fold into checkpoints.enabled). Same values in both execution modes.
Test (red on r11-base: two snapshots kept instead of one):
tests/gateway/test_managed_worker_checkpoint_limits.py::test_managed_worker_trims_checkpoints_to_the_configured_max_snapshots
Review: andrexibiza 27
Review: andrexibiza 5479692359-P3.b
Review: kshitijk4poor P3-managed-checkpoints
(cherry picked from commit 85b0db45ba95f9bb44d2574490574baf34d807e4)
A managed worker parent could select delegate_task, but the child path
always raised worker_child_registration_required: a delegated child needs
its own owner-registered execution (scope, outbox, epoch fence), and a
single-turn, opt-in managed worker has no way to reserve one. The model was
offered a tool whose only outcome was a refusal.
The worker now constructs its agent with the delegation toolset disabled
(WORKER_DISABLED_TOOLSETS). Disabled toolsets are subtracted last, so the
tool_search bridge and a restored tools[] pin drop delegate_task as well;
the refusal in _open_child_session_db stays as the backstop. In-process
(default) sessions are unchanged. Owner-registered child executions for
managed workers are a separate design (DECISION NEEDED in the lane notes).
Review: https://github.com/NousResearch/hermes-agent/pull/106742#discussion_r4230769955
(cherry picked from commit 9332c3f893f3a959a80bbfc2ae114e769c87de11)
RuntimeSessionStore refused any on-disk journal whose scope differed from
the constructor scope, before adopt() could ever see the authenticated
replacement assignment. After owner replacement a lost-ACK receipt
(sequence 1) therefore stayed stranded: the old epoch was fenced
(stale_epoch) and the verified new epoch was refused (outbox_scope_mismatch).
The constructor now takes the separately authenticated worker.adopt result
(adopted=). A differing journal is reconciled only when that proof names the
same execution/generation/target and the requested epoch, every other scope
field is identical, the epoch only moves forward, and a target change is
explained by a pending compression.publish. The pending entries replay
under the journal's own target with the adopted epoch, so the owner resolves
the original receipt (exact digest) instead of writing a second one, and the
constructor then requires the replayed scope to equal the adopted scope.
Without a matching proof the outbox is refused exactly as before.
agent.managed_worker.execute passes the adoption it already verified.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#discussion_r4230769936
(cherry picked from commit 7995f199993375ec1b99af16da51f2412ebf4590)
Shell hooks and outbound webhooks are callbacks on the process-local plugin manager. The owner
registers them (gateway startup; --accept-hooks consent via accept_launch_hooks), but a managed
turn's tools and model calls run in the worker child, whose plugin manager had none: only the
Kanban path called shell_hooks.register_from_config (bind_worker_context), and no worker ever
registered outbound webhooks. A probe on r6-base recorded 1 shell-hook fire + webhook deliveries
per turn in process and 0/0 for the same turn managed.
gateway/session_policy.register_worker_hooks registers both from the session's frozen config in
the worker, before MCP/tool discovery and agent construction:
- consent = the session's --accept-hooks launch flag (request_json) or the Kanban attempt's
context (always consents, as before); otherwise register_from_config resolves the usual
HERMES_ACCEPT_HOOKS / hooks_auto_accept / existing allowlist, with no TTY prompt in the child;
- bypass sessions (config-only/safe) register nothing, matching accept_launch_hooks; the safe
policy also refuses inside both registrars.
The Kanban-only call in bind_worker_context is folded into this single worker seam (Kanban
workers now also get outbound webhooks, as the classic one-shot CLI worker registered both).
Red/green (tests/gateway/test_managed_worker_hooks.py):
- test_managed_turn_fires_shell_hooks_and_webhooks_in_the_worker (real daemon, loopback model,
real `sh` hook + loopback webhook receiver, sessions created with accept_hooks=True): against
the previous commit `'managed': {'shell_pids': [], 'webhooks': []}` vs in-process 1 fire +
pre_llm_call delivery; green: both fire, and the managed hook's parent pid is the worker, not
the owner.
- test_worker_hook_registration_follows_the_session_policy: consent sources and bypass refusal.
tests/gateway/test_session_kanban.py stays green.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5462648005
The in-process constructor passes ephemeral_system_prompt=combined_ephemeral (the gateway
prompt from agent.system_prompt / display.personality / channel overrides, plus the frozen -s
skills blocks) and prefill_messages=runner._prefill_messages. agent/managed_worker.execute passed
only policy.skills_prompt and no prefill, so a managed turn silently dropped the configured
instruction and few-shot examples while still completing.
construct_v1 (the owner-resolved construction inputs introduced for the fallback chain) now also
carries:
- ephemeral_system_prompt: TurnRunner's combined prompt for a LOCAL turn, resolved on the owner
through runner._get_system_prompt_for_channel under the owning profile scope, joined with the
frozen skills_prompt exactly as run_turn_runner does. It stays API-time only (the agent appends
it at request build), so the persisted system prefix is unchanged;
- prefill_messages: runner._prefill_messages, as passed in process.
Bypass sessions never read the profile: config-only/safe carry no configured prompt and no
prefill (safe mode additionally drops prefill in agent_init); their frozen launch skills still
apply. The child validates the closed shape; an older owner's frame constructs as before.
Red/green (tests/gateway/test_managed_worker_construct.py):
- test_managed_turn_sends_configured_instruction_and_prefill (real daemon, loopback model,
agent.system_prompt + prefill_messages_file): against the previous commit the managed request
has `'system_mark': [False], 'prefill': []` while in-process has the marker and both prefill
messages; green: identical placement in both, and the stored system prompt row contains no
marker in either mode.
- test_construct_inputs_follow_the_session_mode extended: ordinary resolves prompt+prefill,
config-only/safe never call the profile readers, malformed fields are refused.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5461749257
The in-process constructor (gateway/session_execution._build_fresh_agent) passes
fallback_model=runner._refresh_fallback_model(); agent/managed_worker.execute never supplied it,
and _init_fallback_chain builds only from that parameter. An ordinary managed custom-provider
session with a healthy `fallback_providers` backup therefore failed its turn when the primary
was quota-refused, where the same turn in process fell back and completed.
The owner now resolves the mode-appropriate chain per turn
(gateway/session_worker_construct.construct_inputs) and carries it in the hydrated request as
`construct_v1` beside `turn_v1` (BOOTSTRAP_FIELDS stays closed). The child validates the closed
shape and passes it to AIAgent.
- ordinary: runner._refresh_fallback_model() under the owning profile's scope (same per-turn
refresh + per-home last-known-good semantics as in process);
- config-only (--ignore-user-config): the frozen explicit snapshot's chain, never the profile;
- safe mode: none (agent_init already forces fallback_model={} under the safe policy; the owner
does not resolve one either). Bypass workers never reload the user profile.
Red/green (tests/gateway/test_managed_worker_construct.py):
- test_managed_turn_falls_back_like_the_in_process_turn (real daemon, loopback primary 429
insufficient_quota + loopback backup): on r6-base the managed receipt is
`{'failed': True, 'completed': False, 'model': None}` with 0 backup requests while in-process
completes on backup-model; green: both complete with final_response BACKUP_OK on backup-model.
- test_construct_inputs_follow_the_session_mode: ordinary uses the runner refresh, config-only
and safe never call it, the child refuses malformed construct_v1 (red: module absent).
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5461749257
The owner already hydrates the session's creation-time config into the bootstrap, but
bind_bypass_policy returned without binding anything for ordinary (non-bypass) sessions.
discover_profile_mcp -> tools/mcp_tool_config._load_mcp_config -> load_config() then read the
LIVE config.yaml, so editing an already-selected MCP server's transport after session creation
changed what an existing session's next fresh worker connected to.
bind_worker_policy now binds the snapshot for every worker before any config import:
- ordinary sessions bind it through the new safe_worker_policy._bind_worker_config, which
makes worker_config_snapshot() return the frozen config while safe_worker_enabled() stays
False, so plugins, MCP, hooks and providers keep their ordinary behaviour (ignore_user_config
is NOT set on ordinary sessions);
- bypass sessions bind exactly as before.
Two consequences of the now-frozen readers are handled at their root:
- save_config in a frozen worker rebases the caller's edit (desired vs the snapshot it loaded)
onto the LIVE raw file (hermes_cli/config_worker_save.py), so an `always` approval or a
connector enable lands only its own change instead of writing the snapshot back over every
profile edit made since the session was created;
- the owner overlays the live `command_allowlist` onto the hydrated snapshot of ordinary
sessions (gateway/session_managed_worker._worker_config): it is approval state, so a grant an
earlier worker persisted (or one the operator removed by hand) reaches the next worker, as it
did when the child read the live file.
Red/green (tests/gateway/test_managed_worker_frozen_config.py, real daemon + real stdio MCP):
- test_existing_session_keeps_frozen_mcp_transport_after_profile_edit: on r6-base
`assert ['echo_LIVE_B'] == ['echo_FROZEN_A']` (existing session followed the edit); green:
the existing session sees FROZEN_A, a session created after the edit sees LIVE_B.
- test_ordinary_worker_binding_freezes_readers_but_keeps_customizations: load_config /
read_raw_config / load_user_config_effective / _load_mcp_config all serve the snapshot, the
plugin still executes, safe_worker_enabled() is False, and a worker save_permanent_allowlist
keeps the live file's later edits (red on r6-base: no ordinary binding exists).
Safe/config-only controls: tests/agent/test_safe_worker_policy.py and
test_managed_worker_bypass.py unchanged and green (renamed binding import only).
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5461749257
execute_managed spawned the worker with stderr=DEVNULL. The worker dup2s its ordinary
stdout onto stderr and logs 'managed worker failed' with the full traceback there, so every
worker crash, provider error echo and tool print vanished: an operator had no trace of why a
managed turn went unknown/failed.
The owner now pipes the child's stderr and drains it on a dedicated thread (an undrained
pipe would block the child) into <profile home>/logs/managed-worker.log
(gateway/session_managed_worker_log.py):
- secrets: tracebacks can carry exception text with credentials, so every line passes
redact_sensitive_text(force=True) (regardless of security.redact_secrets) AND an exact
scrub of this worker's own launch api_key and assignment scope secret before it is written;
- private: created 0600 (O_CREAT 0o600 + fchmod) under the owning profile's home, never
the launch profile's;
- bounded: 1 MiB cap with one rotated copy (managed-worker.log.1); a line longer than 64 KiB
is redacted, cut to 4 KiB and marked [truncated], its remainder discarded;
- the drain never stops reading on a write failure (one warning), owns and closes the pipe at
EOF, and ManagedWorker.close() joins it after the child is reaped.
Authorities without an absolute home (test doubles) keep DEVNULL. The worker-side comment in
agent/managed_worker.py::main now describes the sink. The SimpleNamespace authorities in
test_managed_turn_contract and test_managed_spawn_cancellation gain the profile_id every
real authority carries (fixture shape only, no assertion changed). The pre-existing unused
`# noqa: F401` on the hermes_bootstrap import is dropped (ruff.strict RUF100 in the touched
file; matches every other hermes_bootstrap import site).
Red on r6-base: tests/gateway/test_managed_worker_stderr_log.py
test_worker_stderr_reaches_a_private_redacted_profile_log -> FileNotFoundError (no log)
test_worker_log_is_bounded_and_scrubs_the_workers_exact_secrets -> ModuleNotFoundError
Green after; adjacent managed/worker/Stop suite: 30 files, 74 passed after the fixture fix.
Live probe with the real `python -m agent.managed_worker` fed a bootstrap
carrying a synthetic sk-proj key and assignment secret: log mode 0o600, contains
'managed worker failed' + the ValueError traceback, neither secret present.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5462648005
On main, HERMES_SINGLE_QUERY_SESSION made agent/session_source.py label finite
non-interactive runs `oneshot`, which keeps scripted `hermes chat -q`, `-Q`,
`--oneshot` and `hermes -z` runs out of the TUI, Desktop and dashboard
pickers while `hermes -c` / `sessions list` still see them (sessions.md).
Through the gateway the client always created `source='cli'`, and the owner
did not accept `oneshot`, so every scripted run surfaced as a resumable chat.
- The client creates finite runs (the same `oneshot` predicate the view uses
to wait for one admission) with `source='oneshot'`; an explicit `--source`
still wins, and a gateway that does not advertise `oneshot` gets `cli`.
- The owner maps `oneshot` to the CLI surface (SURFACES, create contract,
generated TS/OpenRPC); the frozen label survives the per-turn peer refresh
(previous commit), so the row stays `oneshot`.
- Every owner-side picker path skips it: session.list (previous commit, via
INTERNAL_LISTING_SOURCES) and the profiles.list `last_session` roster
preview. Desktop's sidebar (SIDEBAR_EXCLUDED_SOURCES) and the dashboard /
console / TUI listings already exclude `oneshot` and now see it.
- Managed workers / worker delegation treat `oneshot` as a CLI session.
Live: real `hermes chat -q` and `hermes -z` through a fake-home gateway and a
loopback stub are stored as `oneshot` in state.db and absent from
session.list, the console filter and the Desktop sidebar filter;
`chat -q --source tui` (an explicit label) stays visible.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5456384314
`hermes chat -q --source tool` exited 1 with "Gateway does not support source
'tool'": runtime.describe advertised only cli/tui/gui/acp, and build_policy
refused any other label. The parser help and sessions.md still document
`--source tool` for third-party integrations, which main stored as `tool` and
kept out of every human session picker.
- session_policy.SURFACES maps `tool` to the CLI surface; describe now
advertises SURFACES, and the canonical create contract (+ generated TS /
OpenRPC) admits it.
- The per-turn peer refresh (`_record_gateway_session_peer` ->
session_a2a.storage_source) rewrote every LOCAL row's source to the transport
name `local`, so even a frozen `tool` (or cron / kanban / bot_room) label
was lost after the first turn. Producer labels from the frozen receipt now
survive; interactive cli/tui/gui/acp rows keep `local` as before.
- session.list (TUI switcher feed) skips INTERNAL_LISTING_SOURCES, like the
TUI/console/dashboard listings on main.
- `tool` sessions behave as CLI ones elsewhere: managed-worker opt-in checks
the CLI platform, the worker builds a `cli` agent, worker delegation accepts
the row.
Live: real `hermes chat -q --source tool` through a fake-home gateway and a
loopback stub now exits 0 and state.db stores `tool`; the row is absent from
session.list and the console/Desktop-sidebar filters.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5456384314
These four launch options had no working path on the canonical gateway. Classic
`hermes chat` refused them and pointed users at `hermes --tui ...`. The Ink TUI sent
them in session.create, but CREATE_FIELDS rejected them (4001 invalid_params at the
canonical contract), and canonicalRequest threw before the session existed.
localCreationOptions read the documented, exported HERMES_ACCEPT_HOOKS=1, so with
that variable set no TUI session could be created at all. HERMES_TUI_PASS_SESSION_ID
was read by nothing and was silently dropped. On main the in-process TUI gateway and
cli.py applied all four, and tests/hermes_cli/test_cli_preloaded_skills.py covered
the -s preload. That test was deleted on the branch with no replacement.
All four options are now creation fields of the session that names them, applied
by the owner:
- skills: rendered once with build_preloaded_skills_prompt (off the owner loop,
only for a new session; a retried request_id resumes the frozen route) and stored
in the frozen LocalSessionPolicy.skills_prompt. The in-process turn appends it to
the ephemeral prompt and the managed worker passes it as ephemeral_system_prompt,
matching the CLI/TUI startup preload. A typo among good names is logged and
skipped, as on main. When no name resolves, creation is refused with unknown_skill.
A bypass (--safe-mode / --ignore-user-config) launch refuses -s because it must
not read the profile.
- checkpoints: folded into the frozen config's checkpoints.enabled, which the
in-process agent already reads through _checkpoint_agent_kwargs(policy config).
The managed worker now passes it too.
- pass_session_id: frozen in the request and passed to AIAgent (in-process and
managed), so "Session ID: <id>" rides the system prompt.
- accept_hooks: consents to the profile's configured shell hooks (per-user
allowlist, off-loop) and registers them on the owner, like the in-process
--accept-hooks. It is a no-op on a bypass launch, so an exported
HERMES_ACCEPT_HOOKS=1 cannot brick --safe-mode.
Classic `hermes chat` is a gateway client too: it forwards the same fields, plus
HERMES_ACCEPT_HOOKS=1 when that variable is exported, instead of refusing them.
The Ink TUI maps HERMES_TUI_SKILLS to a list and the flags to booleans, and the
pipe bootstrap stops leaking HERMES_TUI_CHECKPOINTS/HERMES_TUI_PASS_SESSION_ID into a
spawned daemon. The canonical contract (CanonicalCreateParams plus the generated
TS/OpenRPC) advertises the new fields.
Live proof: a real `python -m gateway.run` daemon on a fake home, a real WS
session.create per field, the real Ink mapping via tsx with HERMES_ACCEPT_HOOKS=1
exported, and a real turn against a loopback model. The skill body and
"Session ID: <id>" reach the model's system prompt, a write snapshots into the
checkpoint store (no snapshot without the flag), and the hook command lands in
the allowlist. A real `hermes chat -Q -q ... -s X --checkpoints --pass-session-id
--accept-hooks` exits 0 with the skill in its prompt. On the base, all of these
were refused with invalid_params, Ink threw, and classic chat exited 2.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5456384314
The managed worker sent only final_response/failed/interrupted, and the
owner committed that with an empty `usage`. `prompt.receipt
include_result=True` (what `hermes -z --usage-file` and API callers read)
therefore reported no tokens, cost, model, provider, api_calls or exit
reason for a managed turn, although the worker had billed inference.
The worker's result frame now carries the bounded accounting fields of
run_conversation (scalars only, text capped), and the owner validates them
(accept_result) and commits the same projection the in-process gateway
turn commits: prompt/completion tokens as input/output in both the result
and `usage`. Transcript copies (messages) stay out of the frame.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#discussion_r4211047387
A managed worker sent tool.start/tool.complete with only tool_call_id and
name, and the owner forwarded that reduced shape. Ink keys its trail on
tool_id/context, ACP renders paths/output from args/result and marks the
call failed from is_error/result, and API observers (Responses streaming,
runs SSE) got no tool callbacks at all. A failing tool in a managed session
therefore showed as a nameless or completed row.
The worker now forwards the callback arguments the in-process turn
publishes from (executor-redacted display args, the result and its failure
verdict), and the owner builds the event with the ordinary turn's payload
builder (_tool_lifecycle_payload / _tool_complete_payload, including
/verbose result_text) and notifies the admission's API observers through
publish_api_tool_event, exactly as TurnRunner does. One shape, one builder.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#discussion_r4213427126
Code-health ratchet on PR #106742 (BLE001/S110/HX006), no behaviour change:
- narrow per-row JSON skips: hosted_room_attachments.reconcile_room_events
(ValueError/TypeError/AttributeError/RecursionError), hermes_state_gateway
routing-entry lookups (ValueError/TypeError/RecursionError)
- approval_context: only the gateway import is optional (ImportError);
finite_turn_required() is a ContextVar read that cannot fail
- deliberate boundaries keep their behaviour and now log the traceback:
profile park (exc_info), session-store init (module logger, same
"gateway.run" name, still WARNING), a2a profile dispatch (local log;
outbound reply stays redacted), readiness probe and session_search
profile open (debug), Windows control-pipe thread hand-off (debug; start()
re-raises), eval harness top level (logger.exception)
- managed_worker.main: traceback goes to the worker's own stderr (the owner
spawns it with stderr=DEVNULL) via a non-propagating handler, never the
channel or profile log files; the owner still gets only
managed_worker_failed
- evals/shared_gateway: timeout=30 on git rev-parse/diff calls
(cherry picked from commit dfc56cdd5ab875bd14b921de71be4a5bc8afc57f)
The one-shot worker's credential mapping (main 89096ba56c / 5fd24f47fd:
_single_query_exit_code(None, credentials_rate_limited=..., credentials_terminal=...))
moved to hermes_cli/turn_exit.turn_exit_code, but the managed path never
passed the flags: a provider quota wall or revoked login during agent
construction crashed the worker with no worker_result, so the dispatcher's
submitter exited 1 and the card spent its failure budget (or retried into
a dead credential) instead of EX_TEMPFAIL requeue / EX_CONFIG park.
- turn_exit.credential_failure_flags(exc): the CLI's predicates
(is_rate_limited_auth_error / AuthError.relogin_required), walking the
cause chain because agent construction wraps the resolver error.
- session_kanban.record_start_failure + managed_worker._construct_agent:
book the mapped exit before the crash; run_worker_turns maps exceptions
the same way. Unknown failures keep exit 1.
Also: web_routers/display.py annotated _open_rfb with Path without a
module-level import (only a function-local one); import it at module scope.
(cherry picked from commit 7363a92d60f312d8670cbd44115b910262d13a68)
Main's #127760 changed clarify_tool to call callback(questions) and expect
{answers, outcome, notice?}. The branch's out-of-process tool worker still exposed
the old clarify(question, choices, multi_select) callback, so every clarify in a
managed worker raised TypeError (reported to the model as a tool error) and no
owner prompt was ever published — the live daemon tests waited forever for it.
WorkerControls.clarify now asks one owner prompt per question over the existing
frame protocol (unchanged on the wire), stops at the first interrupted or unanswered
question (outcome cancelled / timed_out) and maps an empty answer to a skip, like
the in-process gateway callback. The two daemon tests' fake models now emit the
questions[] tool shape main made mandatory.
(cherry picked from commit 2fb94c03911a4957ed0bd2bea52607a25f3457d2)
retire_agent stopped the memory provider directly, whose short drain cancels
an LLM-bound boundary extraction queued during the turn. Give the memory
manager the same bounded flush_pending(timeout=10) head start the CLI exit
path (hermes_cli/cli_shutdown.py) uses, so worker-executed turns do not lose
extraction.
(cherry picked from commit eb10ae818fd385df4dd1834719bcbf268b393a9a)
_worker_env decided whether to scrub the launch environ from the process-wide
multiplex flag. A worker spawned for another profile while that flag read False
(a transient toggle, or a swallowed set_multiplex_active failure) inherited the
launch process env byte-for-byte: NOUS_API_KEY, launch .env names and
vault-sourced values. Probe (fake HOME, flag off): before the fix env=None, so the
whole launch environ was inherited; after it, no LAUNCH-* values reach the worker.
The scrub now keys on _is_routed_home(owner), as served_profile_child_env already
does, and a routed owner also gets the _scrub_credentials belt for credentials no
.env or source snapshot recorded. The launch profile's own worker in a
single-profile gateway still inherits byte-for-byte.
outbox_dir turned execution_id into a path with no validation. A legacy-imported
admission id is arbitrary text, so ../ (POSIX) or ..\ / drive spellings (Windows)
could name a directory outside worker-outboxes. Names are now limited to
[A-Za-z0-9_-]{1,200}; real ids are 'admission-worker:<uuid hex>'.
(cherry picked from commit 693691f2a6498a6e733d781777dbde0f4c854267)
A fresh PM install failed its first 'hermes chat' turn (install-e2e HEAD ->
NEXT: 'The admitted turn failed', no model request). The managed worker and
the kanban worker client are spawned as 'sys.executable -m <module>' with a
rebuilt child env, which strips the Hermes-owned PYTHONPATH that dependency
activation exported; the store interpreter then has no third-party packages.
Both entries now import hermes_bootstrap first, like every other Hermes entry
point, so the child selects the committed environment itself.
- hermes_state_worker_compression.publish_on_connection: select the parent's tool_names like the
direct store does (d956f0ae57 added the column to the child row INSERT; the worker verb's own
parent SELECT missed it and raised IndexError -> 'internal error' on every worker compaction).
- agent/managed_worker: join the auto-title thread before the close-time flush. The title bills
through the worker store from a daemon thread; landing after execution.finish is a
stale_generation persist that failed the flush and lost the worker (safe-mode CLI exit 1).
- Desktop steer: a refused/failed redirect keeps the draft in the canonical queue when the server
owns it, the local queue otherwise, or the composer when no queue exists yet (#68927 composed
with the canonical-queue admission path); test re-expressed on the queue contract.
- tests/tools/test_bot_mode_dm: drop main's two _run_local_turn decode tests (retired subprocess
lane; delivery goes through session_bot.deliver here).
a2a.forward binds the exact peer into the frozen policy, but the submission
carried text only, so memory providers saw the peer's turns as the local
principal with is_bot false. Derive the author from the policy when no
producer stamped one, and carry it in the managed-worker bootstrap frame so
run_conversation receives it out of process too. Attribution only; it
grants nothing.
The owner discovers profile MCP servers at startup, but its registry never
reaches a fresh worker interpreter; the worker built AIAgent from toolset
names alone, so a configured stdio MCP tool was missing from both the direct
definitions and the deferred tool_search catalog the model received. The
worker now runs discover_mcp_tools under its restored policy (filtered to
the frozen toolset selection, no interactive OAuth) before agent tool
discovery snapshots agent.tools. Safe mode is unchanged: its bound policy
makes discover_mcp_tools return [] and spawns nothing.
Finding: F25.
AIAgent.close() runs _close_task_resources, which kills every process the
turn's task ids own; the managed worker called it at every admission
boundary, so a `terminal(background=true)` process reported as started was
dead by the time the result settled. A settled admission is a turn
boundary, not the end of the session: the worker now retires its agent the
way the owner's cached in-process agent does between turns
(shutdown_memory_provider + release_clients), leaves the session's
processes, sandbox and browser alive, and the next admission's worker
adopts them from the profile's process checkpoint so process_manage can
poll and kill them.
Finding: F24.
Compose legacy adoption, producer admission, busy/group controls, API structured
content and terminal result projection with native image staging. Keep worker
completion on the canonical finalization path and preserve provider-qualified
ownership in profile fixtures.
The initial integrated invariant run exposed a raw-identity fixture; align its
owned row with authenticated_subject rather than weakening runtime isolation.
execution_id 'admission-worker:<hex>' was used verbatim as a directory name; ':'
is illegal on Windows, so RuntimeSessionStore.__init__ died with WinError 267 after
adoption (windows-latest run 34367805448). Derive the outbox dir via outbox_dir().
Popen.pid is not the worker on Windows: the uv venv python.exe is a launcher
whose child (or grandchild) is the real interpreter, so a worker asserting
scope['pid'] == os.getpid() refused every managed/safe-mode execution there.
The worker now introduces itself first ({type:'hello', pid, birth, ancestors});
the OWNER (session_worker_reservation.verify_worker_identity) proves the reported
pid is alive, has that create_time, and descends from the reserved handle within
MAX_LAUNCHER_HOPS (or is the handle: the POSIX chain-length-0 case), and only then
binds the adoption digest to the REAL worker pid+birth. Siblings, forged births,
and the owner's own pid are refused. ManagedWorker.close signals the verified
interpreter as well as the handle so an exited launcher cannot orphan its worker.
managed_policy routes any frozen ignore_user_config/safe_mode session to
execute_managed (the custom-provider opt-in stays); the bootstrap frame carries
both bools (validated as exact bools with safe→ignore) and the worker's
bind_bypass_policy freezes them plus the explicit config via
_bind_safe_worker_policy before the owner RPC pulls in hermes_cli.config /
providers / plugins, and before RuntimeSessionStore or run_agent.
Invariants: (1) safe and config-only policies select the managed worker while
ordinary policies stay in-process, and the frame validates the flags; (2) in a
fresh interpreter the bound worker returns the frozen config from
load_config/read_raw_config (helper thread too), never opens the profile YAML,
and safe suppresses plugin discovery while config-only still executes plugins.
