mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-06 05:17:15 +00:00
emozilla
4bc2efbeb3
A press reviewer ran Qwen 27B on a 64 GB RTX Spark and saw single queries take hours. The
causes, and what each part of this change does about them:
- llama-server reads a long prompt for minutes before the first token. Local requests now
always stream and ask for prompt-reading progress, so the stall timers see the server
working and the router notices a hang-up within seconds. They no longer fall back to a
non-streamed request, which the router can't cancel. The status line shows how much of
the prompt the server has read.
- A laptop that sleeps mid-request froze the request, and the stall timer cut it on wake.
Stall timers for a server on the same machine now skip time the process was stopped
(sleep or Modern Standby). Background-call watchdogs run on the awake clock.
- After repeated stalls the breaker failed every later message at once, even after the local
server recovered. At turn start, a local server that answers its health check gets one more
attempt; one more stall shuts the breaker again.
- On Windows unified memory, the GPU carve-out is invisible to Windows, and the model plus
llama-server's default 8 GiB prompt cache overran commit. Free commit now narrows the launch
window (never refusing a model the pool can hold), caps how many models the router holds at
once, and sizes the prompt cache, split across those models.
- Hermes's status reads named the model, and the router loads any model a request names, so
Eject and the idle unload reloaded the model at once. Status reads pass autoload=false; an
unloaded model counts as idle, and the context probe reads the router's launch args.
- The smart-approval verdict has a 16-token budget, and a thinking model spent all of it
thinking (45 of 45 verdicts came back empty). The verdict asks for no thinking unless the
operator set auxiliary.approval.reasoning_effort.
- Compression's 600 s limit counted the time the server spent reading a 150K-token
conversation, which alone can exceed it on a laptop GPU. Prompt reading now pushes the fence
deadline back (CompressionCommitFence.extend_deadline), only while the server reports newly read
tokens, and never past a fence the host closed or one a commit holds. The agent and gateway
hygiene waits read the moved deadline after every slice.
- Desktop keep-awake defaults to "while working". When a timed sleep interrupts a turn with
keep-awake off, the next wake offers it once per computer; the offer survives the next prompt.
(cherry picked from commit 518ce272b9)