mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-06 05:17:15 +00:00
emozilla
4bc2efbeb3
A press reviewer ran Qwen 27B on a 64 GB RTX Spark and saw single queries take hours. The
causes, and what each part of this change does about them:
- llama-server reads a long prompt for minutes before the first token. Local requests now
always stream and ask for prompt-reading progress, so the stall timers see the server
working and the router notices a hang-up within seconds. They no longer fall back to a
non-streamed request, which the router can't cancel. The status line shows how much of
the prompt the server has read.
- A laptop that sleeps mid-request froze the request, and the stall timer cut it on wake.
Stall timers for a server on the same machine now skip time the process was stopped
(sleep or Modern Standby). Background-call watchdogs run on the awake clock.
- After repeated stalls the breaker failed every later message at once, even after the local
server recovered. At turn start, a local server that answers its health check gets one more
attempt; one more stall shuts the breaker again.
- On Windows unified memory, the GPU carve-out is invisible to Windows, and the model plus
llama-server's default 8 GiB prompt cache overran commit. Free commit now narrows the launch
window (never refusing a model the pool can hold), caps how many models the router holds at
once, and sizes the prompt cache, split across those models.
- Hermes's status reads named the model, and the router loads any model a request names, so
Eject and the idle unload reloaded the model at once. Status reads pass autoload=false; an
unloaded model counts as idle, and the context probe reads the router's launch args.
- The smart-approval verdict has a 16-token budget, and a thinking model spent all of it
thinking (45 of 45 verdicts came back empty). The verdict asks for no thinking unless the
operator set auxiliary.approval.reasoning_effort.
- Compression's 600 s limit counted the time the server spent reading a 150K-token
conversation, which alone can exceed it on a laptop GPU. Prompt reading now pushes the fence
deadline back (CompressionCommitFence.extend_deadline), only while the server reports newly read
tokens, and never past a fence the host closed or one a commit holds. The agent and gateway
hygiene waits read the moved deadline after every slice.
- Desktop keep-awake defaults to "while working". When a timed sleep interrupts a turn with
keep-awake off, the next wake offers it once per computer; the offer survives the next prompt.
(cherry picked from commit 518ce272b9)
65 lines
2.4 KiB
Python
65 lines
2.4 KiB
Python
"""llama.cpp context-window probes (sibling of ``model_metadata``)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from itertools import pairwise
|
|
from typing import Optional
|
|
from urllib.parse import quote
|
|
|
|
|
|
def _json_object(resp) -> dict:
|
|
"""The response body when it is a JSON object, else an empty dict."""
|
|
try:
|
|
body = resp.json()
|
|
except ValueError:
|
|
return {}
|
|
return body if isinstance(body, dict) else {}
|
|
|
|
|
|
def llamacpp_context(client, server_url: str, model: str) -> Optional[int]:
|
|
"""llama.cpp's RUNTIME n_ctx: /props for a loaded model, else the ``--ctx-size`` the router will
|
|
launch it with (its /models entry), so a lazily-loaded model doesn't fall to a family catch-all.
|
|
The /props read passes autoload=false: the router loads any model a request names, and a context
|
|
probe must not undo an Eject or an idle unload."""
|
|
import httpx
|
|
|
|
from agent.model_metadata import _positive_int
|
|
|
|
for props_path in (f"/props?model={quote(model)}&autoload=false", "/props"):
|
|
try:
|
|
resp = client.get(f"{server_url}{props_path}")
|
|
except httpx.HTTPError:
|
|
return None
|
|
if resp.status_code != 200:
|
|
continue
|
|
settings = _json_object(resp).get("default_generation_settings")
|
|
n_ctx = _positive_int(settings.get("n_ctx")) if isinstance(settings, dict) else None
|
|
if n_ctx is not None:
|
|
return n_ctx
|
|
return router_launch_ctx(client, server_url, model)
|
|
|
|
|
|
def router_launch_ctx(client, server_url: str, model: str) -> Optional[int]:
|
|
"""``--ctx-size`` from the router's launch args for *model* (GET /models), or None."""
|
|
import httpx
|
|
|
|
from agent.model_metadata import _model_id_matches
|
|
|
|
try:
|
|
resp = client.get(f"{server_url}/models")
|
|
except httpx.HTTPError:
|
|
return None
|
|
if resp.status_code != 200:
|
|
return None
|
|
entries = _json_object(resp).get("data")
|
|
for entry in entries if isinstance(entries, list) else []:
|
|
if not isinstance(entry, dict) or not _model_id_matches(str(entry.get("id", "")), model):
|
|
continue
|
|
status = entry.get("status")
|
|
raw_args = status.get("args") if isinstance(status, dict) else None
|
|
args = [str(a) for a in raw_args] if isinstance(raw_args, list) else []
|
|
for flag, value in pairwise(args):
|
|
if flag in ("--ctx-size", "-c") and value.isdigit() and int(value) > 0:
|
|
return int(value)
|
|
return None
|