mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-11-06 05:17:15 +00:00
emozilla
4bc2efbeb3
A press reviewer ran Qwen 27B on a 64 GB RTX Spark and saw single queries take hours. The
causes, and what each part of this change does about them:
- llama-server reads a long prompt for minutes before the first token. Local requests now
always stream and ask for prompt-reading progress, so the stall timers see the server
working and the router notices a hang-up within seconds. They no longer fall back to a
non-streamed request, which the router can't cancel. The status line shows how much of
the prompt the server has read.
- A laptop that sleeps mid-request froze the request, and the stall timer cut it on wake.
Stall timers for a server on the same machine now skip time the process was stopped
(sleep or Modern Standby). Background-call watchdogs run on the awake clock.
- After repeated stalls the breaker failed every later message at once, even after the local
server recovered. At turn start, a local server that answers its health check gets one more
attempt; one more stall shuts the breaker again.
- On Windows unified memory, the GPU carve-out is invisible to Windows, and the model plus
llama-server's default 8 GiB prompt cache overran commit. Free commit now narrows the launch
window (never refusing a model the pool can hold), caps how many models the router holds at
once, and sizes the prompt cache, split across those models.
- Hermes's status reads named the model, and the router loads any model a request names, so
Eject and the idle unload reloaded the model at once. Status reads pass autoload=false; an
unloaded model counts as idle, and the context probe reads the router's launch args.
- The smart-approval verdict has a 16-token budget, and a thinking model spent all of it
thinking (45 of 45 verdicts came back empty). The verdict asks for no thinking unless the
operator set auxiliary.approval.reasoning_effort.
- Compression's 600 s limit counted the time the server spent reading a 150K-token
conversation, which alone can exceed it on a laptop GPU. Prompt reading now pushes the fence
deadline back (CompressionCommitFence.extend_deadline), only while the server reports newly read
tokens, and never past a fence the host closed or one a commit holds. The agent and gateway
hygiene waits read the moved deadline after every slice.
- Desktop keep-awake defaults to "while working". When a timed sleep interrupts a turn with
keep-awake off, the next wake offers it once per computer; the offer survives the next prompt.
(cherry picked from commit 518ce272b9)
58 lines
2.1 KiB
Python
58 lines
2.1 KiB
Python
"""Clocks that stop while the computer sleeps.
|
|
|
|
A laptop that sleeps in the middle of a request freezes Hermes and the local model server
|
|
together. Timers measured on the wall clock or on Python's ``time.monotonic`` on Windows keep
|
|
counting through the sleep, so every stall watchdog fires the moment the machine wakes and kills
|
|
work that was about to continue. Watchdogs that judge "no progress" use these clocks instead.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import time
|
|
from collections.abc import Callable
|
|
|
|
_SLEEP_THRESHOLD_S = 5.0
|
|
|
|
|
|
def _windows_unbiased_clock() -> Callable[[], float] | None:
|
|
"""The unbiased interrupt time in seconds, or None where kernel32 lacks it."""
|
|
try:
|
|
import ctypes
|
|
|
|
query = ctypes.windll.kernel32.QueryUnbiasedInterruptTime
|
|
except (AttributeError, OSError):
|
|
return None
|
|
|
|
def clock() -> float:
|
|
value = ctypes.c_ulonglong(0)
|
|
query(ctypes.byref(value))
|
|
return value.value / 10_000_000
|
|
|
|
return clock
|
|
|
|
|
|
#: Seconds the machine has spent awake, from an arbitrary start. Windows: the unbiased interrupt
|
|
#: time, which excludes sleep and hibernation. Linux and macOS: ``time.monotonic``, which already
|
|
#: stops during suspend there.
|
|
awake_seconds: Callable[[], float] = (
|
|
(_windows_unbiased_clock() if sys.platform == "win32" else None) or time.monotonic)
|
|
|
|
|
|
class SleepWatch:
|
|
"""Reports how long this process was stopped since the last check, for a caller that checks
|
|
every ``interval`` seconds: the wall time past one interval. That covers a machine that slept
|
|
and a process that Modern Standby froze, without relying on how any clock treats either.
|
|
Gaps under five seconds (clock sync, scheduling noise) don't count.
|
|
"""
|
|
|
|
def __init__(self, interval: float) -> None:
|
|
self._interval = interval
|
|
self._wall = time.time()
|
|
|
|
def slept(self) -> float:
|
|
"""Seconds stopped since the previous call, or 0.0."""
|
|
wall = time.time()
|
|
stopped, self._wall = wall - self._wall - self._interval, wall
|
|
return stopped if stopped >= _SLEEP_THRESHOLD_S else 0.0
|