A managed worker (every Kanban card, --safe-mode and --ignore-user-config
chats) built AIAgent from the frozen model.base_url with no api_mode, and
resolved the route only to fill a missing key. The in-process turn takes
api_key, base_url and api_mode from route_runtime(), so a route whose
api_mode or endpoint only resolution knows failed only in the worker:
opencode-go Responses models fell back to chat_completions (400), and
Bedrock, which stores no base_url, landed in us-east-1 where eu./apac.
inference profiles do not exist.
worker_route() runs the same route_runtime() call with the bound launch
key (if any) and returns api_key, base_url and api_mode for the agent. A
bound launch key still wins; on a fallback entry nothing is carried, as
before (#139155). Resolution stays in the worker's own profile home: no
new secret path.
Refs #143547
Co-authored-by: Kazantcev Andrey <45011689+heckad@users.noreply.github.com>
Co-authored-by: Gekiere Aurélien <11493426+aurel282@users.noreply.github.com>
An exact session.create retry was checked only against the names the
stored, filtered client_env kept, so a retry that added a name the rules
keep was accepted, and an identical overlapping create could be refused
at commit.
The rules in force when the session is committed (behaviour names, the
built-in toolchain names and configured extras, prefixes and secret
substrings) are now stored with its policy as plain data
(client_env_rules_json). A retry's complete client_env, filtered by those
rules, must equal the stored map, both at staging and in the
transactional commit: adding a name the rules keep, dropping a stored name
or changing a value is refused, and a name the rules drop is ignored. The
losing create's own map is carried only in memory on its staged receipt,
never written; a receipt that has lost it (a plain dict copy, for
example) is refused at commit (invalid_params) rather than judged by its
own filtered map. A session created without client_env still retries
only without one, and the policy identity leaves the new field out when
it is empty, so existing fingerprints don't change.
Refs #134889 (LC-8)
Refs #139523
On main each `hermes` process scanned ./.hermes/plugins in its own cwd
when HERMES_ENABLE_PROJECT_PLUGINS was set. Every session now runs in the
one gateway, which discovers plugins once from its own directory, so a
project's plugins silently never load.
The plugins guide now says so in a Project-local plugins section. When a
CLI or TUI session starts with a forwarded HERMES_ENABLE_PROJECT_PLUGINS
in a folder whose ./.hermes/plugins has entries, the gateway logs a line
to gateway.log and session.info carries it as config_warning: the TUI
already renders that at session start, and the classic CLI now prints it.
A folder that can't be listed skips the notice; it never fails
session.create or session.info. Nothing is loaded; loading project
plugins in a plugin host is left for later.
Host-specific client values also need proof that the client is on this
host. A connection is native_owner only when it redeemed a ticket minted
over the gateway's private control socket, from loopback with no Origin
header; session.create passes that as client_local, and a client can't
send it (an unknown field). Without it only HERMES_EPHEMERAL_SYSTEM_PROMPT
and HERMES_INFERENCE_PROVIDER from client_env apply: the session keeps the
gateway's toolchain, prefill file and project-plugins check
(client_env_remote on its policy, left out of the policy identity when
false), and its config_warning says the client's environment was not
applied because the client is not attached to this gateway locally.
Refs #134889 (LC-4)
Refs #134889 (LC-8)
Refs #134889 (T3)
Refs #139523
On main each `hermes` process read HERMES_EPHEMERAL_SYSTEM_PROMPT,
HERMES_PREFILL_MESSAGES_FILE and HERMES_INFERENCE_PROVIDER from its own
shell. Since the one gateway runs every session, the values exported by
whichever client auto-started it became daemon-wide (messaging and cron
included), and every later client's exports were ignored.
The classic CLI and the Ink TUI now send an allowlisted `client_env` map
(plain strings) with session.create; the classic CLI takes the
allowlist's extra names from the profile's effective config (managed
layer included), as the gateway reads them, and a safe-mode or
ignore-user-config launch reads no profile. The gateway filters the map
again against the session profile's allowlist, freezes it with the
session's policy (outside launch_request, so viewers never see the
values) and binds it for that session's turns only: the ephemeral
prompt, the prefill file and the provider request read the session's
value, in-process and for a managed worker. Desktop sends nothing and
keeps the gateway's environment, as does a session created before this
change.
The auto-started daemon no longer inherits these variables (nor
HERMES_ENABLE_PROJECT_PLUGINS); a gateway the operator starts keeps its
own environment. Clients send the field only when runtime.describe lists
it.
An exact retry of a session created before this change returns its
original session; the retry and commit comparisons and the policy identity
treat an absent client_env as before. A retry is judged against the
committed, filtered map: every stored name must come back with its stored
value, and names dropped at creation don't count, whatever the allowlist
is now. Fresh creates use the current rules, and the unfiltered map is
never stored.
Refs #134889 (LC-8)
Refs #139523
Gateway-owned sessions (in-process turns and managed workers alike) got
the runner's prefill examples, loaded once when the daemon started from
the launch profile: a prefill_messages_file path or file edited after
that never reached a new session. On pre-merge main every `hermes chat`
process read the file once at startup from its own profile.
session_prefill() resolves the path from the session policy's
creation-time config, relative to the session's own profile home. The
file is read by the creation attempt's commit (commit_local_session),
before its write transaction; on the async create paths that runs in
tracked_write's worker thread, not on the owner's event loop. The
snapshot is carried with that creation attempt. It is held in the owning
authority's memory only when that attempt's commit creates the session.
An overlapping retry or a conflicting create never replaces it, and a
refused or failed create (for example a retry of a deleted session) holds
nothing. Other sessions cannot evict it, an edit after creation reaches
only new sessions, and deleting the session releases it. A session this
owner did not create (restored after a gateway restart, or a new branch)
reads the file once on first use and keeps it; main's CLI likewise
re-read it on every start. The in-process agent build and
construct_inputs for managed workers share the snapshot, so both send the
same examples. Bypass sessions still never load the profile's file. The
HERMES_PREFILL_MESSAGES_FILE env precedence is unchanged.
Related open PR, not used: #124630 (re-reads the active profile on every
agent build for multiplexed messaging turns, against the pre-split
layout; does not read the session's frozen config).
Refs #134889 (CFG-7)
Refs #139523
A gateway-owned session's in-process turn (and its /bg side agent) took
provider_routing from the runner's daemon-start snapshot of the launch
profile, so an edit after `hermes gateway` started never reached a new
session, and a multiplexed profile's session ran the launch profile's
routing. On pre-merge main every `hermes chat` process read it once at
startup from its own profile.
The turn now takes provider_routing from the session policy's config,
frozen at creation from the session's own profile home: a new session
sees the current value, an open one keeps what it started with, and the
in-process turn matches the managed worker (routing_agent_kwargs already
read the same frozen config). Messaging turns without a session policy
keep the runner copy, as main's gateway did.
Related open PRs, not used: #32088 (re-reads routing from config every
turn on the old gateway/run.py) and #124630 (re-reads the active
profile per turn for multiplexed messaging turns, against the pre-split
layout); neither reads the session's own frozen config.
Refs #134889 (CFG-6)
Refs #139523
resolve_runtime_provider takes no config argument; it reads the session's
frozen config only through the surrounding frozen_runtime_config(frozen)
scope. Say so where it is called, so the call is not moved out of the `with`.
Review suggestion on #139614 (kshitijk4poor)
A canonical `session.mutate` model pick switched from the session's frozen
creation route (`old.provider`, `old.base_url`) beside the key resolved for
the route it runs now. A following session (Bot Chat, `follow_profile_config`)
whose profile moved to another endpoint therefore validated the pick against
the old endpoint with the new endpoint's key. Resolve the current model,
endpoint and key from one resolution of the session's own route (never its
fallback), under one read of the profile, as main's switch did from the live
runtime; the provider stays the configured name, so a pick on a named
`providers:` entry keeps its launch key. When the session's own route cannot
authenticate, switch from the configured provider alone instead of letting
the AuthError escape the RPC as `internal error` (#138879).
The turn resolver and the managed-worker bootstrap had the sibling: the launch
key a keyed alias pick bound was still sent after a profile change superseded
the pick and moved the route to another endpoint. A launch key now applies only
while the effective route is the one it was bound with (`route_launch_key`).
Fixes#138879
(cherry picked from commit fc8b1cf51718059a0e01b1bb01a5de902d984b3d)
worker_api_key() walked only the frozen session snapshot's fallbacks on a
primary AuthError. For an ordinary session the owner sends the refreshed
profile chain in construct_v1, so an expired primary (e.g. a lapsed nous
login) with a fallback added after session creation now failed to start,
where the key-less agent used to reach that fallback at init.
Walk the chain the agent will actually get: construct_v1's when the owner
sent one, the frozen snapshot's otherwise (older owner). A resolved fallback
still yields api_key=None, so its key is never paired with the primary
endpoint; when no entry resolves the primary AuthError is re-raised and the
Kanban start exit (75/78) is unchanged.
Refs #134889 (KB-5)
Refs #139162
Refs #139155
A managed worker launched without a launch key (every Kanban run) handed AIAgent
api_key=None. On the anthropic_messages wire only native Anthropic recovers a
missing key, so a custom provider (MiniMax and other Anthropic-protocol hosts)
sent `Authorization: Bearer ` and died as a transient APIConnectionError.
Move the policy-route resolution out of _resolve_policy_agent_runtime into
session_worker_construct.route_runtime and call it from both the in-process
turn and the worker's agent construction. The worker resolves inside its own
profile environment (the owner's _worker_env), so it never reads another
profile's key, and a key from a fallback entry is never paired with the
primary provider. A bound launch key still wins. Resolution errors stay inside
_construct_agent, so Kanban start failures keep their exit mapping.
Refs #134889 (KB-5)
Refs #139162Fixes#139155
Co-authored-by: liuhao1024 <liuhao1024@users.noreply.github.com>
Every Kanban run (and an opted-in CLI session) now executes in a managed worker, which builds
its AIAgent from the frozen session config. On main a Kanban run was `hermes chat -q`, whose
HermesCLI._init_agent passes provider_routing (only/ignore/order/sort/require_parameters/
data_collection), openrouter.min_coding_score, agent.service_tier and a static fast tier's
request overrides. The managed constructor passed none of them, so a worker silently routed
to providers the profile excluded (data_collection: deny included) and dropped priority
processing. Same class as the checkpoint limits (85b0db45ba9): a constructor input the
in-process path resolves from config that the worker's construction never forwarded; a diff
of the two AIAgent(...) keyword sets left only these (the rest ride construct_v1 or are
in-process-only display/callback wiring).
Fix: session_worker_construct.routing_agent_kwargs(config, model, provider, base_url) reads
them from the frozen config exactly like the CLI, and the worker spreads it into AIAgent.
Test: tests/gateway/test_managed_worker_routing_kwargs.py (fails on r11-base: every routing
kwarg is None).
Review: andrexibiza 27 (class sweep: managed-worker construction inputs)
(cherry picked from commit 07d8ebe691a2e428b1663b195f4d32d58dada38c)
The in-process turn passes all four checkpoint arguments to AIAgent through
gateway.run._checkpoint_agent_kwargs (enabled, max_snapshots,
max_total_size_mb, max_file_size_mb). The managed worker forwarded only
checkpoints_enabled=policy.checkpoints_enabled, so a worker-run session
ignored configured limits and used run_agent's 20/500/10 defaults.
The translation now lives in gateway.session_worker_construct
(checkpoint_agent_kwargs, re-exported from gateway.run under its old name so
gateway/run.py shrinks) and the worker applies it to the session's frozen
config snapshot (policy.config(), which already carries the --checkpoints
fold into checkpoints.enabled). Same values in both execution modes.
Test (red on r11-base: two snapshots kept instead of one):
tests/gateway/test_managed_worker_checkpoint_limits.py::test_managed_worker_trims_checkpoints_to_the_configured_max_snapshots
Review: andrexibiza 27
Review: andrexibiza 5479692359-P3.b
Review: kshitijk4poor P3-managed-checkpoints
(cherry picked from commit 85b0db45ba95f9bb44d2574490574baf34d807e4)
The in-process constructor passes ephemeral_system_prompt=combined_ephemeral (the gateway
prompt from agent.system_prompt / display.personality / channel overrides, plus the frozen -s
skills blocks) and prefill_messages=runner._prefill_messages. agent/managed_worker.execute passed
only policy.skills_prompt and no prefill, so a managed turn silently dropped the configured
instruction and few-shot examples while still completing.
construct_v1 (the owner-resolved construction inputs introduced for the fallback chain) now also
carries:
- ephemeral_system_prompt: TurnRunner's combined prompt for a LOCAL turn, resolved on the owner
through runner._get_system_prompt_for_channel under the owning profile scope, joined with the
frozen skills_prompt exactly as run_turn_runner does. It stays API-time only (the agent appends
it at request build), so the persisted system prefix is unchanged;
- prefill_messages: runner._prefill_messages, as passed in process.
Bypass sessions never read the profile: config-only/safe carry no configured prompt and no
prefill (safe mode additionally drops prefill in agent_init); their frozen launch skills still
apply. The child validates the closed shape; an older owner's frame constructs as before.
Red/green (tests/gateway/test_managed_worker_construct.py):
- test_managed_turn_sends_configured_instruction_and_prefill (real daemon, loopback model,
agent.system_prompt + prefill_messages_file): against the previous commit the managed request
has `'system_mark': [False], 'prefill': []` while in-process has the marker and both prefill
messages; green: identical placement in both, and the stored system prompt row contains no
marker in either mode.
- test_construct_inputs_follow_the_session_mode extended: ordinary resolves prompt+prefill,
config-only/safe never call the profile readers, malformed fields are refused.
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5461749257
The in-process constructor (gateway/session_execution._build_fresh_agent) passes
fallback_model=runner._refresh_fallback_model(); agent/managed_worker.execute never supplied it,
and _init_fallback_chain builds only from that parameter. An ordinary managed custom-provider
session with a healthy `fallback_providers` backup therefore failed its turn when the primary
was quota-refused, where the same turn in process fell back and completed.
The owner now resolves the mode-appropriate chain per turn
(gateway/session_worker_construct.construct_inputs) and carries it in the hydrated request as
`construct_v1` beside `turn_v1` (BOOTSTRAP_FIELDS stays closed). The child validates the closed
shape and passes it to AIAgent.
- ordinary: runner._refresh_fallback_model() under the owning profile's scope (same per-turn
refresh + per-home last-known-good semantics as in process);
- config-only (--ignore-user-config): the frozen explicit snapshot's chain, never the profile;
- safe mode: none (agent_init already forces fallback_model={} under the safe policy; the owner
does not resolve one either). Bypass workers never reload the user profile.
Red/green (tests/gateway/test_managed_worker_construct.py):
- test_managed_turn_falls_back_like_the_in_process_turn (real daemon, loopback primary 429
insufficient_quota + loopback backup): on r6-base the managed receipt is
`{'failed': True, 'completed': False, 'model': None}` with 0 backup requests while in-process
completes on backup-model; green: both complete with final_response BACKUP_OK on backup-model.
- test_construct_inputs_follow_the_session_mode: ordinary uses the runner refresh, config-only
and safe never call it, the child refuses malformed construct_v1 (red: module absent).
Review: https://github.com/NousResearch/hermes-agent/pull/106742#pullrequestreview-5461749257
unsupportedpastels