From 4bc2efbeb37ac459ee502a76c4e73e1829a033ea Mon Sep 17 00:00:00 2001 From: emozilla Date: Wed, 14 Oct 2026 15:13:14 -0400 Subject: [PATCH] fix(local): keep long local-model turns alive on laptops A press reviewer ran Qwen 27B on a 64 GB RTX Spark and saw single queries take hours. The causes, and what each part of this change does about them: - llama-server reads a long prompt for minutes before the first token. Local requests now always stream and ask for prompt-reading progress, so the stall timers see the server working and the router notices a hang-up within seconds. They no longer fall back to a non-streamed request, which the router can't cancel. The status line shows how much of the prompt the server has read. - A laptop that sleeps mid-request froze the request, and the stall timer cut it on wake. Stall timers for a server on the same machine now skip time the process was stopped (sleep or Modern Standby). Background-call watchdogs run on the awake clock. - After repeated stalls the breaker failed every later message at once, even after the local server recovered. At turn start, a local server that answers its health check gets one more attempt; one more stall shuts the breaker again. - On Windows unified memory, the GPU carve-out is invisible to Windows, and the model plus llama-server's default 8 GiB prompt cache overran commit. Free commit now narrows the launch window (never refusing a model the pool can hold), caps how many models the router holds at once, and sizes the prompt cache, split across those models. - Hermes's status reads named the model, and the router loads any model a request names, so Eject and the idle unload reloaded the model at once. Status reads pass autoload=false; an unloaded model counts as idle, and the context probe reads the router's launch args. - The smart-approval verdict has a 16-token budget, and a thinking model spent all of it thinking (45 of 45 verdicts came back empty). The verdict asks for no thinking unless the operator set auxiliary.approval.reasoning_effort. - Compression's 600 s limit counted the time the server spent reading a 150K-token conversation, which alone can exceed it on a laptop GPU. Prompt reading now pushes the fence deadline back (CompressionCommitFence.extend_deadline), only while the server reports newly read tokens, and never past a fence the host closed or one a commit holds. The agent and gateway hygiene waits read the moved deadline after every slice. - Desktop keep-awake defaults to "while working". When a timed sleep interrupts a turn with keep-awake off, the next wake offers it once per computer; the offer survives the next prompt. (cherry picked from commit 518ce272b922ab24048696b3b2d322633ef7e6d1) --- agent/auxiliary_client.py | 91 +++--- agent/auxiliary_managed_local.py | 44 +++ agent/auxiliary_stream_watchdog.py | 14 +- agent/chat_completion_helpers.py | 92 +++--- agent/chat_completion_local.py | 102 +++++++ agent/chat_completion_stream_monitor.py | 55 +++- agent/conversation_compression.py | 77 ++--- agent/conversation_compression_deadline.py | 79 +++++ agent/local_prompt_progress.py | 123 ++++++++ agent/model_metadata.py | 22 +- agent/model_metadata_llamacpp.py | 64 +++++ agent/review_idle_queue.py | 6 +- agent/sleep_clock.py | 57 ++++ agent/turn_context.py | 4 + .../electron/keep-awake-controller.test.ts | 112 ++++++++ .../desktop/electron/keep-awake-controller.ts | 110 +++++++ apps/desktop/electron/main.ts | 65 ++--- apps/desktop/electron/power-save.test.ts | 8 +- apps/desktop/electron/power-save.ts | 15 +- apps/desktop/electron/preload.ts | 7 + apps/desktop/src/global.d.ts | 1 + apps/desktop/src/i18n/ar.ts | 1 + apps/desktop/src/i18n/ar_boot.ts | 8 +- apps/desktop/src/i18n/de_notices.ts | 10 +- apps/desktop/src/i18n/en_notices.ts | 10 +- apps/desktop/src/i18n/es_notices.ts | 10 +- apps/desktop/src/i18n/fr_notices.ts | 10 +- apps/desktop/src/i18n/ja_notices.ts | 10 +- apps/desktop/src/i18n/ru_notices.ts | 10 +- apps/desktop/src/i18n/types_notices.ts | 7 +- apps/desktop/src/i18n/zh-hant.ts | 1 + apps/desktop/src/i18n/zh-hant_boot.ts | 7 +- apps/desktop/src/i18n/zh_notices.ts | 9 +- apps/desktop/src/main.tsx | 3 + apps/desktop/src/store/keep-awake.test.ts | 33 ++- apps/desktop/src/store/keep-awake.ts | 30 +- apps/desktop/src/store/notifications.ts | 18 +- gateway/run_turn_hygiene.py | 21 +- hermes_cli/local_runtime/bootstrap.py | 21 +- hermes_cli/local_runtime/capabilities.py | 4 +- hermes_cli/local_runtime/endpoint.py | 12 + hermes_cli/local_runtime/estimator.py | 4 + hermes_cli/local_runtime/hardware.py | 100 +++++-- hermes_cli/local_runtime/load_progress.py | 6 +- hermes_cli/local_runtime/presets.py | 29 +- hermes_cli/local_runtime/supervisor.py | 13 +- hermes_cli/web_routers/local_models.py | 4 +- locales/_keys.desktop.json | 3 + .../agent/test_compression_prompt_reading.py | 154 ++++++++++ tests/agent/test_local_prompt_progress.py | 269 ++++++++++++++++++ tests/agent/test_review_idle_queue.py | 2 +- tests/agent/test_stream_wait_notice.py | 3 +- .../test_hygiene_wait_moved_deadline.py | 46 +++ tests/hermes_cli/test_load_progress.py | 6 +- .../test_local_abandoned_requests.py | 7 +- tests/hermes_cli/test_local_launch_budget.py | 2 + tests/hermes_cli/test_local_models_routes.py | 2 +- .../test_local_residency_admission.py | 2 +- .../hermes_cli/test_local_router_autoload.py | 80 ++++++ tests/hermes_cli/test_local_runtime.py | 2 +- .../test_local_windows_commit_budget.py | 146 ++++++++++ tests/tools/test_smart_approval_reasoning.py | 45 +++ tools/approval_smart.py | 15 + website/docs/user-guide/configuration.md | 4 +- website/docs/user-guide/desktop.md | 2 +- 65 files changed, 1994 insertions(+), 335 deletions(-) create mode 100644 agent/auxiliary_managed_local.py create mode 100644 agent/chat_completion_local.py create mode 100644 agent/conversation_compression_deadline.py create mode 100644 agent/local_prompt_progress.py create mode 100644 agent/model_metadata_llamacpp.py create mode 100644 agent/sleep_clock.py create mode 100644 apps/desktop/electron/keep-awake-controller.test.ts create mode 100644 apps/desktop/electron/keep-awake-controller.ts create mode 100644 tests/agent/test_compression_prompt_reading.py create mode 100644 tests/agent/test_local_prompt_progress.py create mode 100644 tests/gateway/test_hygiene_wait_moved_deadline.py create mode 100644 tests/hermes_cli/test_local_router_autoload.py create mode 100644 tests/hermes_cli/test_local_windows_commit_budget.py create mode 100644 tests/tools/test_smart_approval_reasoning.py diff --git a/agent/auxiliary_client.py b/agent/auxiliary_client.py index 1685f867fc..79aee83b9b 100644 --- a/agent/auxiliary_client.py +++ b/agent/auxiliary_client.py @@ -24,6 +24,8 @@ from types import SimpleNamespace from typing import Any, Callable, Dict, List, NamedTuple, Optional, Tuple, TYPE_CHECKING, Union from urllib.parse import urlparse, parse_qs, urlunparse +from agent.local_prompt_progress import ( + PromptReadingClock, current_prompt_reading_hook, prompt_progress, prompt_reading_hook, with_prompt_progress) from agent.error_classifier import ( _BILLING_PATTERNS, _OVERLOADED_PATTERNS, @@ -31,6 +33,7 @@ from agent.error_classifier import ( is_reasoning_field_rejection, is_reasoning_required_rejection, ) +from agent.auxiliary_managed_local import _is_managed_local_endpoint from agent.auxiliary_reasoning_floor import remember_reasoning_floor, with_reasoning_floor from agent.auxiliary_structured_output import _route_key, _without_structured_output_format, remember_structured_output_rejection from agent.codex_headers import ( @@ -411,12 +414,13 @@ def aux_progress_hook(hook): def _current_aux_stream_deadline() -> Optional[float]: - """The waiting host's absolute monotonic deadline, if one is installed.""" - return getattr(_aux_stream_deadline, "value", None) + """The waiting host's absolute monotonic deadline, if one is installed (read live from a callable).""" + value = getattr(_aux_stream_deadline, "value", None) + return value() if callable(value) else value @contextlib.contextmanager -def aux_stream_deadline(deadline: Optional[float]): +def aux_stream_deadline(deadline: float | Callable[[], float | None] | None): """Publish the host's absolute ``time.monotonic()`` deadline to the stream consumer. ``None`` is a passthrough; re-entrant-safe. Host->worker return leg of the progress hook: @@ -430,7 +434,7 @@ def aux_stream_deadline(deadline: Optional[float]): session that compression never managed to shrink. See #99692. """ previous = getattr(_aux_stream_deadline, "value", None) - _aux_stream_deadline.value = deadline if isinstance(deadline, (int, float)) else previous + _aux_stream_deadline.value = deadline if isinstance(deadline, (int, float)) or callable(deadline) else previous try: yield finally: @@ -458,7 +462,8 @@ def _run_protected_sync_provider_call(callback: Callable[[dict[str, Any]], Any], progress_hook = getattr(_aux_progress, "hook", None) dispatch_hook = getattr(_aux_dispatch, "hook", None) provider_response_hook = getattr(_aux_provider_response, "hook", None) - host_deadline = _current_aux_stream_deadline() + host_deadline = getattr(_aux_stream_deadline, "value", None) + reading_hook = current_prompt_reading_hook() # #99692: the stream is consumed on the daemon below, and thread-locals do not cross that boundary — an # owner-thread-only deadline would leave the fix inert on exactly the path large-session compression # takes (protected call + hard-cancel source installed). @@ -472,7 +477,7 @@ def _run_protected_sync_provider_call(callback: Callable[[dict[str, Any]], Any], aux_progress_hook(progress_hook), _aux_thread_local_hook(_aux_dispatch, dispatch_hook), _aux_thread_local_hook(_aux_provider_response, provider_response_hook), - aux_stream_deadline(host_deadline), + aux_stream_deadline(host_deadline), prompt_reading_hook(reading_hook), aux_interrupt_protection(cancel_check=cancel_check), ): outcome["result"] = callback(kwargs) @@ -7003,43 +7008,6 @@ def _client_streams_internally(client: Any) -> bool: return isinstance(client, (CodexAuxiliaryClient, AnthropicAuxiliaryClient, BedrockAuxiliaryClient)) -_MANAGED_LOCAL_STATE_TTL_S = 15.0 -_managed_local_cache: tuple[float, str] = (0.0, "") - - -def _managed_local_netloc() -> str: - """host:port of the managed local llama-server ("" when none), read with a short TTL from - the supervisor state file provider resolution also uses (exact match).""" - global _managed_local_cache - now = time.monotonic() - ts, cached = _managed_local_cache - if now - ts < _MANAGED_LOCAL_STATE_TTL_S: - return cached - try: - from hermes_cli.local_runtime.supervisor import state_path - - raw = state_path().read_text(encoding="utf-8-sig") - base = str((json.loads(raw) or {}).get("base_url", "")) - netloc = urlparse(base).netloc.lower() - except Exception: - netloc = "" - _managed_local_cache = (now, netloc) - return netloc - - -def _is_managed_local_endpoint(base_url: Optional[str]) -> bool: - """True when *base_url* targets the llama-server this Hermes manages.""" - if not base_url: - return False - managed = _managed_local_netloc() - if not managed: - return False - try: - return urlparse(str(base_url)).netloc.lower() == managed - except Exception: - return False - - def _provider_requires_stream(provider: str, base_url: Optional[str]) -> bool: """Providers that only accept streaming (non-stream = 400): Tencent Copilot, any ``auxiliary.stream_only_base_urls`` substring, and the managed local llama-server @@ -7111,11 +7079,15 @@ def _create_with_progress( return _create_with_progress_once(client, retry_kwargs, task, force_stream=force_stream) -def _stream_request_plan(kwargs: dict[str, Any]) -> tuple[dict[str, Any], str, float]: - """(stream kwargs, model name, total ceiling) for a streamed re-aggregation.""" +def _stream_request_plan(kwargs: dict[str, Any], client: Any = None) -> tuple[dict[str, Any], str, float]: + """(stream kwargs, model name, total ceiling) for a streamed re-aggregation. Requests to the + managed llama-server also ask for prompt-reading progress, which keeps the watchdogs fed + through a long prefill.""" stream_kwargs = dict(kwargs) stream_kwargs["stream"] = True stream_kwargs["stream_options"] = {"include_usage": True} + if _is_managed_local_endpoint(str(getattr(client, "base_url", "") or "")): + stream_kwargs = with_prompt_progress(stream_kwargs) return (stream_kwargs, str(kwargs.get("model") or ""), _aux_stream_total_ceiling(kwargs.get("timeout"))) @@ -7149,7 +7121,7 @@ def _create_with_progress_once( if not _client_streams_internally(client): _notify_aux_provider_response() return response - stream_kwargs, model, total_ceiling = _stream_request_plan(kwargs) + stream_kwargs, model, total_ceiling = _stream_request_plan(kwargs, client) try: chunks = client.chat.completions.create(**stream_kwargs) except Exception as exc: @@ -7184,7 +7156,7 @@ def _aggregate_chat_stream( ``_is_timeout_error`` matches) past *total_ceiling* or the *no_progress* windows (#100501).""" from agent.auxiliary_stream_watchdog import _close_chunk_stream, consume_chat_stream acc = _ChatStreamAccumulator( - model=model, total_ceiling=total_ceiling, host_deadline=_current_aux_stream_deadline()) + model=model, total_ceiling=total_ceiling, host_deadline=_current_aux_stream_deadline) try: consume_chat_stream(chunks, acc, no_progress) finally: @@ -7200,8 +7172,10 @@ class _ChatStreamAccumulator: """Shared per-chunk accumulation so sync and async aggregation cannot drift.""" def __init__(self, model: str = "", total_ceiling: Optional[float] = None, - host_deadline: Optional[float] = None): + host_deadline: float | Callable[[], float | None] | None = None): self._started = time.monotonic() + # llama-server prompt reading doesn't count toward the ceiling (``local_prompt_progress``). + self._prompt_reading = PromptReadingClock(self._started) self._total_ceiling = total_ceiling # Absolute instant the waiting host gives up; checked alongside (not instead of) the # ceiling, and unaffected by pre-construction dispatch/TTFT. @@ -7220,10 +7194,11 @@ class _ChatStreamAccumulator: def _check_deadlines(self) -> None: """Raise TimeoutError past the total ceiling or the host deadline.""" now = time.monotonic() - if self._total_ceiling is not None and (now - self._started) >= self._total_ceiling: + if self._total_ceiling is not None and (now - self._started - self._prompt_reading.seconds) >= self._total_ceiling: raise TimeoutError(f"Auxiliary streamed call timed out after {self._total_ceiling:.0f}s " "total ceiling (stream still open but over budget)") - if self._host_deadline is not None and now >= self._host_deadline: + host_deadline = self._host_deadline() if callable(self._host_deadline) else self._host_deadline + if host_deadline is not None and now >= host_deadline: raise TimeoutError("Auxiliary streamed call timed out at the host compression " f"deadline after {time.monotonic() - self._started:.0f}s " "(the caller already stopped waiting; streaming on would only " @@ -7270,12 +7245,22 @@ class _ChatStreamAccumulator: # Every frame records transport timing (TTFP); only a substantive payload ticks the # forward-progress hook that keeps compression alive (and is reported as True). _notify_aux_timing_response() + progress = prompt_progress(chunk) + # Count the reading before the deadline check it extends. + read_more = progress is not None and self._prompt_reading.tick(progress) self._check_deadlines() self.resp_id = getattr(chunk, "id", None) or self.resp_id self.resp_model = getattr(chunk, "model", None) or self.resp_model chunk_usage = getattr(chunk, "usage", None) if chunk_usage: self.usage = chunk_usage + if progress is not None: + # llama-server is still reading the prompt: new tokens read keep the compression fence + # alive. It is no first token, so the stream watchdog keeps its pre-token patience: a + # busy server reports a batch every minute or more. + if read_more: + _notify_aux_progress() + return False choices = getattr(chunk, "choices", None) or [] if not choices: return False @@ -7327,7 +7312,7 @@ async def _aggregate_chat_stream_async( ) -> Any: """Async mirror of :func:`_aggregate_chat_stream` (AsyncOpenAI streams need ``async for``).""" acc = _ChatStreamAccumulator( - model=model, total_ceiling=total_ceiling, host_deadline=_current_aux_stream_deadline()) + model=model, total_ceiling=total_ceiling, host_deadline=_current_aux_stream_deadline) try: async for chunk in chunks: acc.feed(chunk) @@ -7342,7 +7327,7 @@ async def _aggregate_chat_stream_async( async def _acreate_with_stream(client: Any, kwargs: dict[str, Any], task: Optional[str] = None) -> Any: """Async create() for stream-only providers: ``stream=True`` + aggregate the async chunks.""" - stream_kwargs, model, total_ceiling = _stream_request_plan(kwargs) + stream_kwargs, model, total_ceiling = _stream_request_plan(kwargs, client) chunks = await client.chat.completions.create(**stream_kwargs) if hasattr(chunks, "choices"): # shims may hand back a complete response despite stream=True return chunks @@ -7367,7 +7352,7 @@ async def _acreate_with_progress( if not _async_client_streams_internally(client): _notify_aux_provider_response() return response - stream_kwargs, model, total_ceiling = _stream_request_plan(kwargs) + stream_kwargs, model, total_ceiling = _stream_request_plan(kwargs, client) try: chunks = await client.chat.completions.create(**stream_kwargs) except Exception as exc: diff --git a/agent/auxiliary_managed_local.py b/agent/auxiliary_managed_local.py new file mode 100644 index 0000000000..cb7b6022f5 --- /dev/null +++ b/agent/auxiliary_managed_local.py @@ -0,0 +1,44 @@ +"""Which endpoint is the llama-server this Hermes manages (sibling of ``auxiliary_client``).""" + +from __future__ import annotations + +import json +import time +from typing import Optional +from urllib.parse import urlparse + +_MANAGED_LOCAL_STATE_TTL_S = 15.0 +_managed_local_cache: tuple[float, str] = (0.0, "") + + +def _managed_local_netloc() -> str: + """host:port of the managed local llama-server ("" when none), read with a short TTL from + the supervisor state file provider resolution also uses (exact match).""" + global _managed_local_cache + now = time.monotonic() + ts, cached = _managed_local_cache + if now - ts < _MANAGED_LOCAL_STATE_TTL_S: + return cached + try: + from hermes_cli.local_runtime.supervisor import state_path + + raw = state_path().read_text(encoding="utf-8-sig") + base = str((json.loads(raw) or {}).get("base_url", "")) + netloc = urlparse(base).netloc.lower() + except (OSError, ValueError, AttributeError): # no state file, bad JSON, or not an object + netloc = "" + _managed_local_cache = (now, netloc) + return netloc + + +def _is_managed_local_endpoint(base_url: Optional[str]) -> bool: + """True when *base_url* targets the llama-server this Hermes manages.""" + if not base_url: + return False + managed = _managed_local_netloc() + if not managed: + return False + try: + return urlparse(str(base_url)).netloc.lower() == managed + except ValueError: + return False diff --git a/agent/auxiliary_stream_watchdog.py b/agent/auxiliary_stream_watchdog.py index f3a6c5abfd..6e62bb9129 100644 --- a/agent/auxiliary_stream_watchdog.py +++ b/agent/auxiliary_stream_watchdog.py @@ -12,10 +12,11 @@ from __future__ import annotations import inspect import threading -import time from types import SimpleNamespace from typing import Any, Optional +from agent.sleep_clock import awake_seconds + def _stream_socket(stream: Any) -> Any: """The raw socket under an SDK chunk stream (``stream.response`` is the ``httpx.Response``).""" @@ -40,12 +41,13 @@ def _close_chunk_stream(chunks: Any, *, allow_aclose: bool = False) -> Any: class ChatStreamWatchdog: """Re-armed by substantive chunks only. Silence before the first token is bounded by *first_token_window* (the main loop's stale patience: reasoning models think silently for - minutes); ``None`` leaves it to the request timeout (a local server's silent prefill).""" + minutes); ``None`` leaves it to the request timeout (a local server's silent prefill). + Windows run on the awake clock, so time the computer spends asleep doesn't count.""" def __init__(self, stream: Any, window: float, *, first_token_window: Optional[float] = None): self.window = window self.first_token_window = first_token_window - self._started = time.monotonic() + self._started = awake_seconds() self._lock = threading.Lock() self._deadline: Optional[float] = None self._timer: Optional[threading.Timer] = None @@ -79,7 +81,7 @@ class ChatStreamWatchdog: def _rearm(self, window: float) -> None: # A pending timer re-checks the deadline when it fires; replace it only when the new # deadline is earlier (the first token swaps the long first-token window for the short one). - previous, self._deadline = self._deadline, time.monotonic() + window + previous, self._deadline = self._deadline, awake_seconds() + window if self._timer is None or (previous is not None and self._deadline < previous): if self._timer is not None: self._timer.cancel() @@ -103,7 +105,7 @@ class ChatStreamWatchdog: with self._lock: if self._done or self._deadline is None or generation != self._generation: return - remaining = self._deadline - time.monotonic() + remaining = self._deadline - awake_seconds() if remaining > 0: self._schedule(remaining) return @@ -122,7 +124,7 @@ class ChatStreamWatchdog: def timeout_error(self) -> TimeoutError: """Zero-output stalls say "no-progress timeout" (same-provider retry stays allowed, see ``_should_skip_same_provider_retry``); a mid-stream stall goes straight to fallback.""" - elapsed = time.monotonic() - self._started + elapsed = awake_seconds() - self._started if not self.saw_progress: return TimeoutError( f"Auxiliary chat stream produced no output within {self.first_token_window or 0:.1f}s " diff --git a/agent/chat_completion_helpers.py b/agent/chat_completion_helpers.py index 01c26afe7a..e35a0e90c1 100644 --- a/agent/chat_completion_helpers.py +++ b/agent/chat_completion_helpers.py @@ -31,6 +31,7 @@ from agent.error_classifier import ( _extract_status_code) from agent.sdk_transform_bypass import bypass_chat_sdk_request_transform from agent.errors import EmptyStreamError +from agent.chat_completion_local import _streams_llamacpp, stale_giveup from agent.chat_completion_stream_monitor import StreamingWaitMonitor from agent.transports.chat_completions import is_router_timeout_shim, router_timeout_shim_may_follow from agent.fast_mode import effective_request_overrides @@ -50,6 +51,7 @@ from agent.reasoning_summaries import ( append_streamed_reasoning_detail, separate_glued_reasoning_blocks, streamed_reasoning_detail_text, ) +from agent.local_prompt_progress import prompt_progress, with_prompt_progress from agent.repetition_guard import RunawayStreamWatch, is_repetition_dominated from agent.stream_single_writer import claim_stream_writer, stream_writer_is_current from tools.terminal_tool_lifecycle import is_persistent_env @@ -608,7 +610,7 @@ def _touch_stale_kill_activity(agent, elapsed: float) -> None: def _check_stale_giveup(agent) -> None: """Raise immediately when the consecutive-stale streak is past the give-up threshold — no network attempt, no stale-timeout wait.""" - _giveup = env_int("HERMES_STREAM_STALE_GIVEUP", 5) + _giveup = stale_giveup() _streak = _stale_streak(agent) if _giveup > 0 and _streak >= _giveup: raise RuntimeError( @@ -831,39 +833,6 @@ def should_use_direct_api_call(agent) -> bool: _DIRECT_API_ACTIVITY_HEARTBEAT_SECONDS = 15.0 -def _managed_local_load_notice(agent, api_kwargs: dict) -> Optional[str]: - """Live phase notice ("⏳ loading into memory — N%" / "⚙ processing - prompt — P%") while the managed local server works before the first token; - None when neither applies. Otherwise a cold load reads as a generic stall.""" - try: - base = str(getattr(agent, "base_url", "") or "") - if not base: - return None - from urllib.parse import urlparse - from hermes_cli.local_runtime.load_progress import get_loading_progress, get_prefill_progress - from hermes_cli.local_runtime.supervisor import state_path - state = json.loads(state_path().read_text(encoding="utf-8-sig")) - managed = urlparse(str(state.get("base_url", ""))).netloc.lower() - if not managed or urlparse(base).netloc.lower() != managed: - return None - model = str(api_kwargs.get("model", "")) - progress = get_loading_progress().get(model) - if progress is not None: - return (f"⏳ loading {model} into memory — {progress['percent']}% " - "(responses start once the model is loaded)") - prefill = get_prefill_progress(model) - if prefill is None: - return None - processed = int(prefill["processed"]) - total = estimate_request_context_tokens(api_kwargs) - if total and total >= processed: - return f"⚙ processing prompt — {max(0, min(100, round(processed / total * 100)))}%" - # Counter past the estimate (estimator undercounted): no honest denominator, label-only. - return "⚙ processing prompt" - except Exception: - return None - - def _resolve_direct_stale_timeout(agent, api_kwargs: dict) -> float: """Stale budget for the inline call via ``agent._compute_non_stream_stale_timeout``, plus the same openai-codex floor/hard cap the worker path applies (inline cron Codex, @@ -2751,6 +2720,7 @@ class _StreamingCall(StreamingWaitMonitor): self.provider_tool_in_flight = {"yes": False} # Last REAL chunk; the monitor detects SSE-ping-only connections with it. self.last_chunk_time = {"t": time.time()} + self.prompt_progress = {"progress": None, "t": 0.0} # latest llama-server prompt progress # Shared by the socket read timeout (``_stream_timeouts``) and the stale # detector (``_resolve_stale_timeout``); None until resolved. self._stream_stale_timeout = None @@ -2808,9 +2778,9 @@ class _StreamingCall(StreamingWaitMonitor): with self.stream_attempt_lock: self.stream_attempt_state["current"] += 1 attempt_id = int(self.stream_attempt_state["current"]) - self.provider_tool_in_flight["yes"] = False - # Attempt-local like provider_tool_in_flight: a tool name from a stream that died - # before any text must not label a later attempt's partial stub or its retry decision. + # Attempt-local: a dead attempt's prompt progress must not mask the next load notice, and + # its tool name (died before any text) must not label a later partial stub or retry. + self.provider_tool_in_flight["yes"], self.prompt_progress["progress"] = False, None self.result["partial_tool_names"] = [] return attempt_id @@ -2896,6 +2866,29 @@ class _StreamingCall(StreamingWaitMonitor): # Delta-length estimate: ~3x cheaper than repr() per chunk. diag["bytes"] = int(diag.get("bytes", 0)) + _estimate_chunk_bytes(chunk) + def _take_chunk(self, diag, chunk, llamacpp: bool) -> bool: + """Count a real chunk; a llama-server prompt-progress chunk (True) is liveness, no first token.""" + progress = prompt_progress(chunk) if llamacpp else None + if progress is None: + self._count_chunk(diag, chunk) + return False + self.last_chunk_time["t"] = now = time.time() + self.prompt_progress["progress"], self.prompt_progress["t"] = progress, now + self.agent._touch_activity("model reading the prompt") + return True + + @staticmethod + def _chunk_identity(diag, chunk, model_name, response_id, upstream_provider) -> tuple: + """Model, response id and serving provider, as the stream reports them.""" + if hasattr(chunk, "model") and chunk.model: + model_name = chunk.model + if response_id is None and isinstance(getattr(chunk, "id", None), str) and chunk.id: + response_id = chunk.id + if upstream_provider is None and isinstance(getattr(chunk, "provider", None), str) and chunk.provider: + upstream_provider = chunk.provider # OpenRouter stamps who served + diag["serving_provider"] = upstream_provider.strip()[:64] # attribute a mid-stream drop (#90216) + return model_name, response_id, upstream_provider + @staticmethod def _mark_finish_seen(diag, finish_reason) -> None: """Record that this attempt saw a terminal finish/stop reason (#102766).""" @@ -3064,8 +3057,12 @@ class _StreamingCall(StreamingWaitMonitor): detail_watch = RunawayStreamWatch() runaway = None + llamacpp = _streams_llamacpp(self.agent) + def _open_stream(next_api_kwargs: dict[str, Any]): timeout = _httpx.Timeout(connect=conn_cap, read=read_timeout, write=base_timeout, pool=conn_cap) + if llamacpp: + next_api_kwargs = with_prompt_progress(next_api_kwargs) return self._open_chat_stream({**next_api_kwargs, "stream": True, "timeout": timeout}) def _flush_pending_stream_text(): @@ -3096,20 +3093,17 @@ class _StreamingCall(StreamingWaitMonitor): self.agent._abort_request_openai_client(self._attempt_request_client, reason=reason) for chunk in _iter_provider_stream_chunks(stream, response=lambda: self._attempt_stream_response): - self._count_chunk(_diag, chunk) + is_progress = self._take_chunk(_diag, chunk, llamacpp) if self.agent._interrupt_requested: _close_half_read_stream("interrupt_stream_close_failed") break if not self._stream_attempt_is_active(stream_attempt_id): self._discard_stale_stream_chunk(stream_attempt_id, chunk) continue - if hasattr(chunk, "model") and chunk.model: - model_name = chunk.model - if response_id is None and isinstance(getattr(chunk, "id", None), str) and chunk.id: - response_id = chunk.id - if upstream_provider is None and isinstance(getattr(chunk, "provider", None), str) and chunk.provider: - upstream_provider = chunk.provider # OpenRouter stamps who served - _diag["serving_provider"] = upstream_provider.strip()[:64] # attribute a mid-stream drop (#90216) + if is_progress: + continue + model_name, response_id, upstream_provider = self._chunk_identity( + _diag, chunk, model_name, response_id, upstream_provider) if not chunk.choices: usage, finish_reason = self._choiceless_chunk(chunk, finish_reason) usage_obj = usage or usage_obj @@ -3527,7 +3521,10 @@ class _StreamingCall(StreamingWaitMonitor): AnthropicBedrock IAM lacking InvokeModelWithResponseStream, a custom anthropic_messages provider emitting SSE events out of order (#72833), or a gateway answering with contentless SSE keepalive frames (a degraded gateway answers every - streaming request that way, so the retry must change channel to make progress).""" + streaming request that way, so the retry must change channel to make progress). + llama-server stays on streaming: the router can't cancel a non-streamed request.""" + if _streams_llamacpp(self.agent): + return if _is_provider_stream_empty_frame_error(e): self.agent._disable_streaming = True logger.warning( @@ -3683,6 +3680,9 @@ class _StreamingCall(StreamingWaitMonitor): """ if getattr(self.agent, "_interrupt_requested", False): return False + if _streams_llamacpp(self.agent): + # Same error either way, and the router can't cancel this probe once its watchdog gives up. + return False status = _extract_status_code(e) if status is None or status < 500 or self.deltas_were_sent["yes"]: return False diff --git a/agent/chat_completion_local.py b/agent/chat_completion_local.py new file mode 100644 index 0000000000..c1de9628b2 --- /dev/null +++ b/agent/chat_completion_local.py @@ -0,0 +1,102 @@ +"""Chat completions against llama-server (sibling of ``chat_completion_helpers``). + +Which requests go to llama-server, the turn-start health check that lets a recovered server try +again past the stale breaker, and the live load / prompt-reading notice shown before the first +token. +""" + +from __future__ import annotations + +import http.client +import json +import logging +import urllib.request +from typing import Optional +from urllib.parse import urlparse, urlsplit + +from utils import env_int + +logger = logging.getLogger(__name__) + + +def _streams_llamacpp(agent) -> bool: + """True when this agent's chat requests go to llama-server: the managed server, or a + ``llamacpp`` provider endpoint. These requests always stream and ask for prompt progress + (unless the user turned streaming off in config). A non-streamed request can't be cancelled + through the router, so it would keep the server busy after Hermes gives up on it.""" + if getattr(agent, "api_mode", "") not in ("", "chat_completions"): + return False + from hermes_cli.local_runtime.capabilities import is_managed_provider + + return is_managed_provider(str(getattr(agent, "provider", "") or ""), str(getattr(agent, "base_url", "") or "")) + + +def stale_giveup() -> int: + """Consecutive stale attempts after which calls fail at once; 0 turns the breaker off.""" + return env_int("HERMES_STREAM_STALE_GIVEUP", 5) + + +def _local_server_answers(agent) -> bool: + """True when the llama-server behind *agent* answers ``/health`` within a few seconds.""" + parts = urlsplit(str(getattr(agent, "base_url", "") or "")) + if not parts.scheme or not parts.hostname: + return False + # "localhost" can resolve to ::1 first and wait out a timeout on a server bound to IPv4. + netloc = parts.netloc.replace("localhost", "127.0.0.1", 1) if parts.hostname == "localhost" else parts.netloc + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + try: + with opener.open(f"{parts.scheme}://{netloc}/health", timeout=3) as response: + return response.status == 200 + except (OSError, ValueError, http.client.HTTPException): + return False + + +def retry_recovered_local_server(agent) -> bool: + """Turn start: when the stale breaker is open, the agent talks to llama-server, and the + server answers its health check, let one attempt through. Within a turn the breaker stops + endless retries; across turns, a local server that crashed or fell behind and has since + recovered gets the user's next message instead of an instant error. The router answers + ``/health`` even while its model is stuck, so one more stale attempt shuts the breaker again.""" + from agent.chat_completion_helpers import _stale_streak + + giveup, streak = stale_giveup(), _stale_streak(agent) + if giveup <= 0 or streak < giveup or not _streams_llamacpp(agent) or not _local_server_answers(agent): + return False + logger.info("Local model server answers its health check after %d stale attempts; trying once more.", streak) + agent._consecutive_stale_streams = giveup - 1 + return True + + +def _managed_local_load_notice(agent, api_kwargs: dict) -> Optional[str]: + """Live phase notice ("⏳ loading into memory — N%" / "⚙ processing + prompt — P%") while the managed local server works before the first token; + None when neither applies. Otherwise a cold load reads as a generic stall.""" + try: + base = str(getattr(agent, "base_url", "") or "") + if not base: + return None + from agent.chat_completion_helpers import estimate_request_context_tokens + from hermes_cli.local_runtime.load_progress import get_loading_progress, get_prefill_progress + from hermes_cli.local_runtime.supervisor import state_path + state = json.loads(state_path().read_text(encoding="utf-8-sig")) + managed = urlparse(str(state.get("base_url", ""))).netloc.lower() + if not managed or urlparse(base).netloc.lower() != managed: + return None + model = str(api_kwargs.get("model", "")) + progress = get_loading_progress().get(model) + if progress is not None: + return (f"⏳ loading {model} into memory — {progress['percent']}% " + "(responses start once the model is loaded)") + prefill = get_prefill_progress(model) + if prefill is None: + return None + processed = int(prefill["processed"]) + total = estimate_request_context_tokens(api_kwargs) + if total and total >= processed: + return f"⚙ processing prompt — {max(0, min(100, round(processed / total * 100)))}%" + # Counter past the estimate (estimator undercounted): no honest denominator, label-only. + return "⚙ processing prompt" + except Exception: + # A status line must never break the stream it describes. + logger.debug("local load notice unavailable", exc_info=True) + return None diff --git a/agent/chat_completion_stream_monitor.py b/agent/chat_completion_stream_monitor.py index fa129b5e97..d0c1965682 100644 --- a/agent/chat_completion_stream_monitor.py +++ b/agent/chat_completion_stream_monitor.py @@ -1,10 +1,16 @@ """Display and heartbeat phase of the request-local streaming monitor.""" +import logging import time from types import SimpleNamespace +from urllib.parse import urlsplit from agent import chat_completion_wait_notice as wn from agent.model_metadata import is_local_endpoint +from agent.proxy_bypass import is_loopback_host +from agent.sleep_clock import SleepWatch + +logger = logging.getLogger(__name__) class StreamingWaitMonitor: @@ -14,9 +20,11 @@ class StreamingWaitMonitor: REAL chunk arrived for 2s+ (never during healthy token flow); in-memory, no network. True while loading = heartbeat liveness, skip the rest of this iteration (the stale detector's local floor dwarfs any load).""" - from agent.chat_completion_helpers import _managed_local_load_notice + from agent.chat_completion_local import _managed_local_load_notice m = self._mon + if self._show_prompt_progress(now): + return True if now - self.last_chunk_time["t"] < 2.0 or now - m.last_load_poll < 1.0: return False m.last_load_poll = now @@ -36,6 +44,46 @@ class StreamingWaitMonitor: self.agent._emit_wait_notice("") return False + def _show_prompt_progress(self, now: float) -> bool: + """Prompt progress straight from the stream's own progress chunks (llama-server with + ``return_progress``), at most once a second until the first token. Clears the line + when the first token arrives. True when it showed progress.""" + from agent.local_prompt_progress import prompt_progress_percent + + m = self._mon + state = getattr(self, "prompt_progress", None) + diag = getattr(getattr(self, "clients", None), "diag", None) + first_token = isinstance(diag, dict) and bool(diag.get("first_chunk_at")) + if not state or state["progress"] is None: + return False + if first_token: + if m.load_notice_shown: + m.load_notice_shown, m.load_notice_misses = False, 0 + self.agent._emit_wait_notice("") + return False + if now - state["t"] > 60.0 or now - m.last_load_poll < 1.0: + return False + m.last_load_poll = now + percent = prompt_progress_percent(state["progress"]) + m.wait_notice_started_ts = None # The prompt-progress line owns the display. + m.wait_notice.reset() + self.agent._emit_wait_notice( + f"⚙ processing prompt — {percent}%" if percent is not None else "⚙ processing prompt") + m.load_notice_shown, m.load_notice_misses, m.last_heartbeat = True, 0, now + return True + + def _skip_sleep(self, slept: float) -> None: + """The computer slept and froze this request with a server on the same machine: move the + stall clock past the sleep, so the request gets its full patience again after wake. Never + past now: a chunk can land between the sleep reading and this write.""" + if slept <= 0: + return + now = time.time() + self.last_chunk_time["t"] = min(self.last_chunk_time["t"] + slept, now) + self._mon.last_heartbeat = min(self._mon.last_heartbeat + slept, now) + logger.info("Computer slept for %.0fs during a model request; the stall timer skips that time.", slept) + self.agent._touch_activity(f"resumed after the computer slept for {int(slept)}s") + def _heartbeat(self, waiting_secs: int) -> None: """Gateway inactivity heartbeat: the start-to-first-chunk gap (thinking, local prefill) can exceed the gateway timeout.""" @@ -64,8 +112,13 @@ class StreamingWaitMonitor: wait_notice=wn.WaitNoticeState(), ) _is_local_base = bool(self.agent.base_url) and is_local_endpoint(self.agent.base_url) + # Only a server on this machine sleeps with it; a remote one kept working (or died). + sleep_watch = (SleepWatch(interval=0.3) if self.agent.base_url + and is_loopback_host(urlsplit(self.agent.base_url).hostname) else None) while not self._call_done.is_set(): self._call_done.wait(timeout=0.3) + if sleep_watch is not None: + self._skip_sleep(sleep_watch.slept()) _hb_now = time.time() if _is_local_base and self._poll_local_load_notice(_hb_now): continue diff --git a/agent/conversation_compression.py b/agent/conversation_compression.py index fd0873cf0d..d8e69542e3 100644 --- a/agent/conversation_compression.py +++ b/agent/conversation_compression.py @@ -32,6 +32,7 @@ from agent.auxiliary_client import AuxiliaryExplicitCancellation from agent.context_compressor_summary import _accepts_keyword_argument from agent.context_engine import automatic_compaction_status_message, sanitize_memory_context from agent.conversation_compression_codex import _compress_context_via_codex_app_server +from agent.conversation_compression_deadline import resolve_context_compression_timeouts from agent.conversation_compression_telemetry import ( _emit_aborted_attempt_telemetry, _emit_blocked_attempt_telemetry, _emit_bypassed_attempt_telemetry, _emit_compression_attempt_telemetry, _record_committed_attempt_effect, _snapshot_compression_effect_input, @@ -508,6 +509,20 @@ class CompressionCommitFence: raise ValueError("compression deadline must be positive") self._deadline = deadline + def extend_deadline(self, seconds: float) -> bool: + """Push the armed deadline back by *seconds* (time llama-server spent reading the prompt). A cancelled + attempt or a passed deadline stays final, so a late progress chunk can't revive it; while a commit or a + cancellation holds the lock there is nothing to extend.""" + if seconds <= 0 or not self._lock.acquire(blocking=False): + return False + try: + alive = self._deadline is not None and self._deadline > time.monotonic() and not self.is_cancelled + if alive: + self._deadline += seconds + return alive + finally: + self._lock.release() + def touch_progress(self) -> None: """Record forward progress (a streamed token); a bare float store is atomic, so no lock.""" self._last_progress = time.monotonic() @@ -674,10 +689,6 @@ class CompressionCommitFence: release() -# Defaults for the in-agent progress-aware wrap; mirror hermes_cli.config.DEFAULT_CONFIG["compression"] keys. -DEFAULT_CONTEXT_TIMEOUT_SECONDS = 120.0 -DEFAULT_CONTEXT_TOTAL_CEILING_SECONDS = 600.0 - # Unlike explicit_interrupt, a /stop after the stall window arms the durable backoff (no automatic re-entry). # Distinct from ``explicit_interrupt``: a /stop that arrived after the summary stream had already crossed # the no-progress stall window (#96775). Ordinary early /stop stays cooldown-neutral; this class arms the @@ -772,42 +783,6 @@ def _get_compress_timeout_executor(): return _compress_timeout_executor -def resolve_context_compression_timeouts(compression_cfg: Optional[dict] = None) -> tuple[float, float]: - """Return ``(idle_timeout_seconds, total_ceiling_seconds)``. - ``idle_timeout_seconds <= 0`` disables the progress-aware wrapper. The ceiling is clamped to at least one - idle window when the idle budget is positive.""" - idle = DEFAULT_CONTEXT_TIMEOUT_SECONDS - ceiling = DEFAULT_CONTEXT_TOTAL_CEILING_SECONDS - cfg = compression_cfg - if cfg is None: - cfg = {} - with contextlib.suppress(Exception): - from hermes_cli.config import load_config - raw = load_config() - maybe = raw.get("compression", {}) if isinstance(raw, dict) else {} - cfg = maybe if isinstance(maybe, dict) else {} - if isinstance(cfg, dict): - # Explicit 0/negative idle disables; a non-positive ceiling is ignored. - with contextlib.suppress(TypeError, ValueError): - if cfg.get("context_timeout_seconds") is not None: - idle = float(cfg["context_timeout_seconds"]) - with contextlib.suppress(TypeError, ValueError): - if cfg.get("context_total_ceiling_seconds") is not None and float(cfg["context_total_ceiling_seconds"]) > 0: - ceiling = float(cfg["context_total_ceiling_seconds"]) - if idle > 0: - ceiling = max(ceiling, idle) - # #114594: the aux summary attempt legitimately needs up to its own budget (floor 300s), while the - # host idle watchdog defaults to 120s. Clamp the idle window up to at least the effective aux - # compression timeout so prep/chunking work with no streamed tokens yet is not cut off. Only raises: - # an explicit cfg value above the aux budget is kept, and idle never exceeds the ceiling. - from agent.auxiliary_client import _effective_aux_timeout - _aux_budget = float(_effective_aux_timeout("compression", None)) - if _aux_budget > 0: - ceiling = max(ceiling, _aux_budget) - idle = max(idle, min(_aux_budget, ceiling)) - return idle, ceiling - - def compression_attempt_stalled( *, commit_fence: Optional[CompressionCommitFence], started_at: float, idle_timeout_seconds: Optional[float] = None ) -> bool: @@ -1089,8 +1064,8 @@ def _await_worker_within_budget( ) -> tuple[bool, Any]: """Poll ``future`` under the idle budget + ceiling; ``(True, result)`` when it settled.""" while True: - waited = time.monotonic() - wait_started - remaining_ceiling = ceiling - waited + # The fence deadline moves: prompt reading on llama-server pushes it back. + remaining_ceiling = (fence.deadline_monotonic or wait_started + ceiling) - time.monotonic() if remaining_ceiling <= 0: return False, None # Charge idle budget from LAST PROGRESS, not slice start, or silence could approach 2x the budget. @@ -1109,7 +1084,7 @@ def _await_worker_within_budget( return False, None waited = time.monotonic() - wait_started since_progress = fence.seconds_since_progress() - if not fence.deadline_exceeded and since_progress < idle and waited < ceiling: + if not fence.deadline_exceeded and since_progress < idle: logger.info( "Context compression still streaming after %.0fs (last progress %.1fs ago) — extending wait (ceiling %.0fs)", waited, since_progress, ceiling, @@ -1353,7 +1328,7 @@ def run_compress_context_with_progress_timeout( # F6: a not-yet-started future must not linger as a stale queued job. # cancel() is a no-op for a running worker (fence handles that path). future.cancel() - total_exhausted = time.monotonic() - wait_started >= ceiling or fence.deadline_exceeded + total_exhausted = fence.deadline_exceeded # #97488 teardown (total-ceiling path only): give the cancelled worker a bounded grace to actually # exit before this host moves on. The worker checks the poison fence between provider phases, so a # cooperative worker exits quickly; an uninterruptible provider call is orphaned behind the fence @@ -1371,8 +1346,8 @@ def run_compress_context_with_progress_timeout( on_timeout_cause(total_exhausted, fence.progress_observed) if not _cancel_or_join_worker(fence): result = _await_in_flight_commit( - future, ceiling=ceiling, wait_started=wait_started, on_commit_overrun=on_commit_overrun - ) + future, ceiling=max(ceiling, fence.deadline_monotonic - wait_started), wait_started=wait_started, + on_commit_overrun=on_commit_overrun) handled_exit = True # The cancelled worker can race the host into its commit section while unwinding a stalled # summary. When that commit is only the unchanged snapshot, returning it here skips the @@ -2976,11 +2951,11 @@ def _run_summary_dispatch( # Publish progress to the commit fence so hosts extend deadlines while tokens # flow. Any active hook (even no-op) selects the streamed path: the timeout is # inactivity-based and a byte-trickling provider hits the stream total ceiling. - from agent.auxiliary_client import aux_interrupt_protection, aux_progress_hook, aux_stream_deadline + from agent.auxiliary_client import aux_interrupt_protection, aux_progress_hook + from agent.conversation_compression_deadline import compression_deadline_scope _progress_hook = commit_fence.touch_progress if commit_fence is not None else (lambda: None) - # Return leg: cancel frees the owner but the provider daemon streams on to its - # own larger ceiling; share the host deadline so orphan streams stop with it. - _host_stream_deadline = commit_fence.deadline_monotonic if commit_fence is not None else None + # Return leg: cancel frees the owner but the provider daemon streams on to its own larger + # ceiling; share the host deadline (live: prompt reading pushes it back) so orphans stop with it. # A LATE successful summary must not undo the host's timeout cooldown: the # compressor checks cancellation before clearing; removed in finally (no leak). if commit_fence is not None: @@ -3009,7 +2984,7 @@ def _run_summary_dispatch( compressed = messages else: with ( - aux_progress_hook(_progress_hook), aux_stream_deadline(_host_stream_deadline), + aux_progress_hook(_progress_hook), compression_deadline_scope(commit_fence), aux_interrupt_protection(cancel_check=_compression_cancel_requested), ): # This attempt is now doing real summary work: publish it as the working attempt so later diff --git a/agent/conversation_compression_deadline.py b/agent/conversation_compression_deadline.py new file mode 100644 index 0000000000..0eaa4f799d --- /dev/null +++ b/agent/conversation_compression_deadline.py @@ -0,0 +1,79 @@ +"""Compression time limits: the configured idle and total limits, and the fence deadline shared live +with the summary stream (sibling of ``conversation_compression``). + +Compression waits a total time limit for its summary. That limit stops a summary that trickles +out a token at a time; on a laptop GPU, reading a 150K-token conversation alone can take longer +than the whole limit. Time llama-server spends reading the prompt therefore pushes the fence +deadline back (``CompressionCommitFence.extend_deadline``), and hosts read the deadline from the +fence instead of adding the configured limit to the time they started waiting. +""" + +from __future__ import annotations + +import contextlib +from typing import TYPE_CHECKING, Optional + +if TYPE_CHECKING: + from agent.conversation_compression import CompressionCommitFence + +# Defaults for the in-agent progress-aware wrap; mirror hermes_cli.config.DEFAULT_CONFIG["compression"] keys. +DEFAULT_CONTEXT_TIMEOUT_SECONDS = 120.0 +DEFAULT_CONTEXT_TOTAL_CEILING_SECONDS = 600.0 + + +def resolve_context_compression_timeouts(compression_cfg: Optional[dict] = None) -> tuple[float, float]: + """Return ``(idle_timeout_seconds, total_ceiling_seconds)``. + ``idle_timeout_seconds <= 0`` disables the progress-aware wrapper. The ceiling is clamped to at least one + idle window when the idle budget is positive.""" + idle = DEFAULT_CONTEXT_TIMEOUT_SECONDS + ceiling = DEFAULT_CONTEXT_TOTAL_CEILING_SECONDS + cfg = compression_cfg + if cfg is None: + cfg = {} + with contextlib.suppress(Exception): + from hermes_cli.config import load_config + raw = load_config() + maybe = raw.get("compression", {}) if isinstance(raw, dict) else {} + cfg = maybe if isinstance(maybe, dict) else {} + if isinstance(cfg, dict): + # Explicit 0/negative idle disables; a non-positive ceiling is ignored. + with contextlib.suppress(TypeError, ValueError): + if cfg.get("context_timeout_seconds") is not None: + idle = float(cfg["context_timeout_seconds"]) + with contextlib.suppress(TypeError, ValueError): + if cfg.get("context_total_ceiling_seconds") is not None and float(cfg["context_total_ceiling_seconds"]) > 0: + ceiling = float(cfg["context_total_ceiling_seconds"]) + if idle > 0: + ceiling = max(ceiling, idle) + # #114594: the aux summary attempt legitimately needs up to its own budget (floor 300s), while the + # host idle watchdog defaults to 120s. Clamp the idle window up to at least the effective aux + # compression timeout so prep/chunking work with no streamed tokens yet is not cut off. Only raises: + # an explicit cfg value above the aux budget is kept, and idle never exceeds the ceiling. + from agent.auxiliary_client import _effective_aux_timeout + _aux_budget = float(_effective_aux_timeout("compression", None)) + if _aux_budget > 0: + ceiling = max(ceiling, _aux_budget) + idle = max(idle, min(_aux_budget, ceiling)) + return idle, ceiling + + +def effective_ceiling(fence: CompressionCommitFence, ceiling: float, wait_started: float) -> float: + """The total limit for a host that started waiting at *wait_started*: the configured + *ceiling*, or longer once prompt reading pushed the fence deadline back.""" + deadline = fence.deadline_monotonic + return ceiling if deadline is None else max(ceiling, deadline - wait_started) + + +@contextlib.contextmanager +def compression_deadline_scope(fence: Optional[CompressionCommitFence]): + """Share *fence*'s deadline with the summary stream, live, and push it back while the + server reads the prompt. A stream checks the deadline as it reads, so a later summary + call in the same attempt sees the extended deadline too.""" + from agent.auxiliary_client import aux_stream_deadline + from agent.local_prompt_progress import prompt_reading_hook + + if fence is None: + yield + return + with aux_stream_deadline(lambda: fence.deadline_monotonic), prompt_reading_hook(fence.extend_deadline): + yield diff --git a/agent/local_prompt_progress.py b/agent/local_prompt_progress.py new file mode 100644 index 0000000000..cba0cfa0ac --- /dev/null +++ b/agent/local_prompt_progress.py @@ -0,0 +1,123 @@ +"""Prompt-reading progress from llama-server. + +llama-server reads a long prompt for minutes before it writes the first token. Asked with +``return_progress``, it streams a progress chunk after each batch it reads. Hermes uses those +chunks as signs of life: the stall timers stay quiet while the server works, and the router +notices a client hang-up within seconds (it only checks the client socket when it writes). + +Compression also leaves the time the server spends reading the prompt out of its total time +limit (``conversation_compression_deadline``). +""" + +from __future__ import annotations + +import contextlib +import threading +import time +from typing import Any, Callable, Optional + +_reading = threading.local() + +_CONTENT_FIELDS = ("content", "tool_calls", "reasoning_content", "reasoning", "refusal") + + +def with_prompt_progress(kwargs: dict[str, Any]) -> dict[str, Any]: + """Copy of chat-completions *kwargs* that asks llama-server for prompt progress chunks.""" + extra_body = kwargs.get("extra_body") + merged = dict(extra_body) if isinstance(extra_body, dict) else {} + merged.setdefault("return_progress", True) + return {**kwargs, "extra_body": merged} + + +def prompt_progress(chunk: Any) -> Optional[dict]: + """The ``prompt_progress`` payload of a progress-only chunk, else None. + + A progress chunk carries an empty assistant delta. A chunk with text, tool calls, reasoning or a + finish reason is a real chunk even if a server attaches progress to it. + """ + extra = getattr(chunk, "model_extra", None) + progress = extra.get("prompt_progress") if isinstance(extra, dict) else None + if progress is None and isinstance(chunk, dict): + progress = chunk.get("prompt_progress") + if not isinstance(progress, dict): + return None + choices = getattr(chunk, "choices", None) + if choices is None and isinstance(chunk, dict): + choices = chunk.get("choices") + for choice in choices or []: + finish = choice.get("finish_reason") if isinstance(choice, dict) else getattr(choice, "finish_reason", None) + delta = choice.get("delta") if isinstance(choice, dict) else getattr(choice, "delta", None) + if finish: + return None + for field in _CONTENT_FIELDS: + value = delta.get(field) if isinstance(delta, dict) else getattr(delta, field, None) + if value: + return None + return progress + + +def _count(progress: dict, key: str) -> Optional[int]: + try: + return int(progress[key]) + except (KeyError, TypeError, ValueError): + return None + + +def prompt_progress_percent(progress: dict) -> Optional[int]: + """Share of the prompt the server has read, 0-100, or None without a usable total.""" + try: + total = int(progress.get("total") or 0) + processed = int(progress.get("processed") or 0) + except (TypeError, ValueError): + return None + if total <= 0: + return None + return max(0, min(100, round(processed / total * 100))) + + +@contextlib.contextmanager +def prompt_reading_hook(hook: Optional[Callable[[float], None]]): + """Install ``hook(seconds)`` for this thread; it hears each stretch of prompt reading.""" + previous = getattr(_reading, "hook", None) + _reading.hook = hook + try: + yield + finally: + _reading.hook = previous + + +def current_prompt_reading_hook() -> Optional[Callable[[float], None]]: + """This thread's prompt-reading hook (carried into provider daemon threads).""" + return getattr(_reading, "hook", None) + + +class PromptReadingClock: + """Time llama-server spends reading the prompt. + + Counted per progress chunk, and only when the server reports more tokens read than before: a + server that keeps reporting without reading anything gets no extra time. Each stretch is the + smaller of the wall time since the previous chunk and the server's own reading time, so time + spent queued or loading the model before the reading starts doesn't count. + """ + + def __init__(self, started: float): + self._mark = started + self._processed = -1 + self._server_ms = 0 + self.seconds = 0.0 + + def tick(self, progress: dict) -> bool: + """Count the reading up to this progress chunk; False when it reports nothing new read.""" + now = time.monotonic() + wall, self._mark = now - self._mark, now + processed, server_ms = _count(progress, "processed"), _count(progress, "time_ms") + if processed is None or processed <= self._processed: + return False + server = wall if server_ms is None else (server_ms - self._server_ms) / 1000 + self._processed, self._server_ms = processed, self._server_ms if server_ms is None else server_ms + spent = max(0.0, min(wall, server)) + self.seconds += spent + hook = current_prompt_reading_hook() + if hook is not None: + hook(spent) + return True diff --git a/agent/model_metadata.py b/agent/model_metadata.py index 62fd6a3719..5126590c7d 100644 --- a/agent/model_metadata.py +++ b/agent/model_metadata.py @@ -19,6 +19,7 @@ import hermes_yaml as yaml from agent import model_metadata_http from agent.model_metadata_cache import _write_context_cache, locked_context_cache +from agent.model_metadata_llamacpp import llamacpp_context from utils import atomic_json_write, base_url_host_matches, base_url_hostname @@ -998,7 +999,7 @@ def _apply_llamacpp_props(cache: dict[str, dict[str, Any]], request_candidate: s child_id = child.get("id") if isinstance(child, dict) else None if not child_id or child_id not in cache or (child.get("status") or {}).get("value") not in ("loaded", "ready"): continue - pr = _props({"model": child_id}) + pr = _props({"model": child_id, "autoload": "false"}) # it may have unloaded since /models child_ctx = _n_ctx(pr.json()) if not pr.is_error else None if child_ctx: cache[child_id]["context_length"] = child_ctx @@ -1566,23 +1567,6 @@ def _lmstudio_context(client, lmstudio_url: str, model: str) -> Optional[int]: return None -def _llamacpp_context(client, server_url: str, model: str) -> Optional[int]: - """llama.cpp /props: the RUNTIME n_ctx, answered by the router even for a not-yet-loaded model - (while /v1/models has meta=null), so a lazily-loaded model doesn't fall to a family catch-all.""" - import httpx - for props_path in (f"/props?model={model}", "/props"): - try: - resp = client.get(f"{server_url}{props_path}") - except httpx.HTTPError: - return None - if resp.status_code != 200: - continue - n_ctx = _positive_int((resp.json().get("default_generation_settings") or {}).get("n_ctx")) - if n_ctx is not None: - return n_ctx - return None - - def _openai_models_list_context(client, server_url: str, model: str) -> Optional[int]: """/v1/models list: match by id, else the sole model on single-model servers (llama.cpp reports a GGUF path as id).""" resp = client.get(f"{server_url}/v1/models") @@ -1633,7 +1617,7 @@ def _query_local_context_length_uncached(model: str, base_url: str, api_key: str typed = { "ollama": _ollama_ctx, "lm-studio": lambda client: _lmstudio_context(client, lmstudio_url, model), - "llamacpp": lambda client: _llamacpp_context(client, server_url, model), + "llamacpp": lambda client: llamacpp_context(client, server_url, model), }.get(server_type) probes = ([typed] if typed else []) + [_model_detail_ctx, lambda client: _openai_models_list_context(client, server_url, model)] try: diff --git a/agent/model_metadata_llamacpp.py b/agent/model_metadata_llamacpp.py new file mode 100644 index 0000000000..ce80f15c3d --- /dev/null +++ b/agent/model_metadata_llamacpp.py @@ -0,0 +1,64 @@ +"""llama.cpp context-window probes (sibling of ``model_metadata``).""" + +from __future__ import annotations + +from itertools import pairwise +from typing import Optional +from urllib.parse import quote + + +def _json_object(resp) -> dict: + """The response body when it is a JSON object, else an empty dict.""" + try: + body = resp.json() + except ValueError: + return {} + return body if isinstance(body, dict) else {} + + +def llamacpp_context(client, server_url: str, model: str) -> Optional[int]: + """llama.cpp's RUNTIME n_ctx: /props for a loaded model, else the ``--ctx-size`` the router will + launch it with (its /models entry), so a lazily-loaded model doesn't fall to a family catch-all. + The /props read passes autoload=false: the router loads any model a request names, and a context + probe must not undo an Eject or an idle unload.""" + import httpx + + from agent.model_metadata import _positive_int + + for props_path in (f"/props?model={quote(model)}&autoload=false", "/props"): + try: + resp = client.get(f"{server_url}{props_path}") + except httpx.HTTPError: + return None + if resp.status_code != 200: + continue + settings = _json_object(resp).get("default_generation_settings") + n_ctx = _positive_int(settings.get("n_ctx")) if isinstance(settings, dict) else None + if n_ctx is not None: + return n_ctx + return router_launch_ctx(client, server_url, model) + + +def router_launch_ctx(client, server_url: str, model: str) -> Optional[int]: + """``--ctx-size`` from the router's launch args for *model* (GET /models), or None.""" + import httpx + + from agent.model_metadata import _model_id_matches + + try: + resp = client.get(f"{server_url}/models") + except httpx.HTTPError: + return None + if resp.status_code != 200: + return None + entries = _json_object(resp).get("data") + for entry in entries if isinstance(entries, list) else []: + if not isinstance(entry, dict) or not _model_id_matches(str(entry.get("id", "")), model): + continue + status = entry.get("status") + raw_args = status.get("args") if isinstance(status, dict) else None + args = [str(a) for a in raw_args] if isinstance(raw_args, list) else [] + for flag, value in pairwise(args): + if flag in ("--ctx-size", "-c") and value.isdigit() and int(value) > 0: + return int(value) + return None diff --git a/agent/review_idle_queue.py b/agent/review_idle_queue.py index 902b2760e9..b1ceb9a7a4 100644 --- a/agent/review_idle_queue.py +++ b/agent/review_idle_queue.py @@ -47,7 +47,7 @@ def review_targets_managed_local(agent: Any, task_cfg: Optional[dict[str, Any]]) supervisor state file; any failure reads False (immediate spawn is the safe default). The cheap TTL-cached netloc probe runs FIRST so cloud-only installs skip runtime resolution on the turn's tail.""" try: - from agent.auxiliary_client import _is_managed_local_endpoint, _managed_local_netloc + from agent.auxiliary_managed_local import _is_managed_local_endpoint, _managed_local_netloc if not _managed_local_netloc(): return False @@ -201,7 +201,9 @@ def _managed_server_idle() -> bool: loaded = [m["id"] for m in _get("/models").get("data", []) if (m.get("status") or {}).get("value") in ("loaded", "ready")] return not any( - s.get("is_processing") for mid in loaded for s in _get(f"/slots?model={quote(mid)}") if isinstance(s, dict) + # autoload=false: a model unloaded since the /models read must not load again for a status read. + s.get("is_processing") for mid in loaded + for s in _get(f"/slots?model={quote(mid)}&autoload=false") if isinstance(s, dict) ) except Exception: return True diff --git a/agent/sleep_clock.py b/agent/sleep_clock.py new file mode 100644 index 0000000000..c1276dd476 --- /dev/null +++ b/agent/sleep_clock.py @@ -0,0 +1,57 @@ +"""Clocks that stop while the computer sleeps. + +A laptop that sleeps in the middle of a request freezes Hermes and the local model server +together. Timers measured on the wall clock or on Python's ``time.monotonic`` on Windows keep +counting through the sleep, so every stall watchdog fires the moment the machine wakes and kills +work that was about to continue. Watchdogs that judge "no progress" use these clocks instead. +""" + +from __future__ import annotations + +import sys +import time +from collections.abc import Callable + +_SLEEP_THRESHOLD_S = 5.0 + + +def _windows_unbiased_clock() -> Callable[[], float] | None: + """The unbiased interrupt time in seconds, or None where kernel32 lacks it.""" + try: + import ctypes + + query = ctypes.windll.kernel32.QueryUnbiasedInterruptTime + except (AttributeError, OSError): + return None + + def clock() -> float: + value = ctypes.c_ulonglong(0) + query(ctypes.byref(value)) + return value.value / 10_000_000 + + return clock + + +#: Seconds the machine has spent awake, from an arbitrary start. Windows: the unbiased interrupt +#: time, which excludes sleep and hibernation. Linux and macOS: ``time.monotonic``, which already +#: stops during suspend there. +awake_seconds: Callable[[], float] = ( + (_windows_unbiased_clock() if sys.platform == "win32" else None) or time.monotonic) + + +class SleepWatch: + """Reports how long this process was stopped since the last check, for a caller that checks + every ``interval`` seconds: the wall time past one interval. That covers a machine that slept + and a process that Modern Standby froze, without relying on how any clock treats either. + Gaps under five seconds (clock sync, scheduling noise) don't count. + """ + + def __init__(self, interval: float) -> None: + self._interval = interval + self._wall = time.time() + + def slept(self) -> float: + """Seconds stopped since the previous call, or 0.0.""" + wall = time.time() + stopped, self._wall = wall - self._wall - self._interval, wall + return stopped if stopped >= _SLEEP_THRESHOLD_S else 0.0 diff --git a/agent/turn_context.py b/agent/turn_context.py index ffad01705c..fa61b311c9 100644 --- a/agent/turn_context.py +++ b/agent/turn_context.py @@ -642,6 +642,10 @@ def _reset_per_turn_agent_state(agent: Any) -> None: "issue — cleaned up automatically. Proceeding with fresh " "connection." ) + # A stale streak against a local server that has since recovered must not fail this turn on arrival. + with suppress(Exception): + from agent.chat_completion_local import retry_recovered_local_server + retry_recovered_local_server(agent) # Replay compression warning through status_callback for gateway platforms. if agent._compression_warning: agent._replay_compression_warning() diff --git a/apps/desktop/electron/keep-awake-controller.test.ts b/apps/desktop/electron/keep-awake-controller.test.ts new file mode 100644 index 0000000000..768f2b2d36 --- /dev/null +++ b/apps/desktop/electron/keep-awake-controller.test.ts @@ -0,0 +1,112 @@ +import fs from 'node:fs' +import os from 'node:os' +import path from 'node:path' + +import { describe, expect, it, vi } from 'vitest' + +import { createKeepAwakeController, IDLE_SLEEP_MIN_SECONDS } from './keep-awake-controller' +import type { PowerSaveBlockerLike } from './power-save' + +function fakeBlocker() { + let next = 1 + const started = new Set() + + const blocker: PowerSaveBlockerLike = { + isStarted: id => started.has(id), + start: vi.fn(() => { + const id = next++ + started.add(id) + + return id + }), + stop: vi.fn(id => void started.delete(id)) + } + + return { blocker, started } +} + +describe('createKeepAwakeController: the offer after a sleep mid-turn', () => { + function controller(saved?: object, working = true) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'keep-awake-')) + const configPath = path.join(dir, 'keep-awake.json') + + if (saved) { + fs.writeFileSync(configPath, JSON.stringify(saved), 'utf8') + } + + const offer = vi.fn(() => true) + + const ctl = createKeepAwakeController({ + blocker: fakeBlocker().blocker, + configPath, + isWorking: () => working, + log: () => {}, + offer + }) + + ctl.restore() + + return { configPath, ctl, offer } + } + + const timedSleep = IDLE_SLEEP_MIN_SECONDS + 240 + + it('offers once, on the wake after a timed sleep interrupted a turn with keep-awake off', () => { + const { configPath, ctl, offer } = controller({ mode: 'off' }) + + ctl.noteSuspend(timedSleep) + ctl.noteResume() + expect(offer).toHaveBeenCalledTimes(1) + + ctl.noteSuspend(timedSleep) + ctl.noteResume() + expect(offer).toHaveBeenCalledTimes(1) + expect(JSON.parse(fs.readFileSync(configPath, 'utf8'))).toEqual({ mode: 'off', offered: true }) + }) + + it('does not offer again after a relaunch', () => { + const { ctl, offer } = controller({ mode: 'off', offered: true }) + + ctl.noteSuspend(timedSleep) + ctl.noteResume() + expect(offer).not.toHaveBeenCalled() + }) + + it('stays quiet for a closed lid, an idle app, or keep-awake already on', () => { + const lid = controller({ mode: 'off' }) + lid.ctl.noteSuspend(2) + lid.ctl.noteResume() + expect(lid.offer).not.toHaveBeenCalled() + + const idle = controller({ mode: 'off' }, false) + idle.ctl.noteSuspend(timedSleep) + idle.ctl.noteResume() + expect(idle.offer).not.toHaveBeenCalled() + + const on = controller({ mode: 'while-working' }) + on.ctl.noteSuspend(timedSleep) + on.ctl.noteResume() + expect(on.offer).not.toHaveBeenCalled() + }) + + it('waits for a chat window instead of spending the offer', () => { + const { configPath, ctl, offer } = controller({ mode: 'off' }) + + offer.mockReturnValueOnce(false) + ctl.noteSuspend(timedSleep) + ctl.noteResume() + expect(JSON.parse(fs.readFileSync(configPath, 'utf8')).offered).toBeUndefined() + + ctl.noteSuspend(timedSleep) + ctl.noteResume() + expect(offer).toHaveBeenCalledTimes(2) + expect(JSON.parse(fs.readFileSync(configPath, 'utf8')).offered).toBe(true) + }) + + it('keeps the offer record when the mode changes', () => { + const { configPath, ctl } = controller({ mode: 'off', offered: true }) + + ctl.setMode('while-working') + expect(JSON.parse(fs.readFileSync(configPath, 'utf8'))).toEqual({ mode: 'while-working', offered: true }) + }) +}) diff --git a/apps/desktop/electron/keep-awake-controller.ts b/apps/desktop/electron/keep-awake-controller.ts new file mode 100644 index 0000000000..18dd7dc24a --- /dev/null +++ b/apps/desktop/electron/keep-awake-controller.ts @@ -0,0 +1,110 @@ +/** + * Main's half of keep-awake: the one power-save blocker, its persisted mode, + * and the one-time offer after a timed sleep interrupted a turn. Kept apart + * from power-save.ts so the renderer bundle never pulls in Node's fs. + */ + +import fs from 'node:fs' +import path from 'node:path' + +import { + createKeepAwake, + type KeepAwakeMode, + keepAwakeWanted, + parseKeepAwakeMode, + type PowerSaveBlockerLike, + readKeepAwakeMode +} from './power-save' + +/** + * A sleep that starts after at least this long without input is a timed sleep, the kind + * keep-awake prevents. Closing the lid or choosing Sleep is not, and keep-awake can't stop it. + * Windows' shortest sleep timeout is one minute; the idle reading is in whole seconds and can + * land just under it. + */ +export const IDLE_SLEEP_MIN_SECONDS = 55 + +export interface KeepAwakeControllerDeps { + blocker: PowerSaveBlockerLike + /** `keep-awake.json` in the app's userData: the mode, and whether the offer was made. */ + configPath: string + isWorking: () => boolean + log: (line: string) => void + /** Show the one-time "keep awake while working" offer; false when no chat window can show it. */ + offer: () => boolean +} + +export interface KeepAwakeController { + /** Reconcile the blocker with the mode and the live turn picture. */ + apply(working?: boolean): void + /** App ready: restore the saved mode, or the default when nothing usable is saved. */ + restore(): void + /** A mode from the renderer, or the boolean the pre-mode toggle sent. */ + setMode(value: unknown): void + noteSuspend(idleSeconds: number): void + noteResume(): void +} + +/** + * Main's half of keep-awake: the one blocker, its persisted mode, and the offer. When a timed + * sleep interrupts a turn while keep-awake is off, the next wake offers "While working", once + * per computer. + */ +export function createKeepAwakeController(deps: KeepAwakeControllerDeps): KeepAwakeController { + const keepAwake = createKeepAwake(deps.blocker) + let mode: KeepAwakeMode = 'off' + let offered = false + let sleptMidTurn = false + + const persist = () => { + try { + fs.mkdirSync(path.dirname(deps.configPath), { recursive: true }) + fs.writeFileSync(deps.configPath, JSON.stringify({ mode, offered }, null, 2), 'utf8') + } catch (error) { + deps.log(`[keep-awake] write failed: ${(error as Error).message}`) + } + } + + const apply = (working = deps.isWorking()) => { + keepAwake.set(keepAwakeWanted(mode, working)) + } + + return { + apply, + restore() { + let saved: unknown = null + + try { + saved = JSON.parse(fs.readFileSync(deps.configPath, 'utf8')) + } catch { + // First launch, or an unreadable file: the defaults stand. + } + + mode = readKeepAwakeMode(saved) + offered = typeof saved === 'object' && saved !== null && (saved as Record).offered === true + apply() + }, + setMode(value) { + const next = parseKeepAwakeMode(value) + + if (next === null) { + return + } + + mode = next + apply() + persist() + }, + noteSuspend(idleSeconds) { + sleptMidTurn = mode === 'off' && deps.isWorking() && idleSeconds >= IDLE_SLEEP_MIN_SECONDS + }, + noteResume() { + if (sleptMidTurn && !offered && deps.offer()) { + offered = true + persist() + } + + sleptMidTurn = false + } + } +} diff --git a/apps/desktop/electron/main.ts b/apps/desktop/electron/main.ts index 7736c8f476..a36665bf57 100644 --- a/apps/desktop/electron/main.ts +++ b/apps/desktop/electron/main.ts @@ -325,6 +325,7 @@ import { buildHudWindowUrl } from './hud-url' import { linuxOzoneBackend, resolveHudWindowing } from './hud-windowing' import { INSTALL_STAMP, installShape } from './install-stamp' import type { InstallStamp } from './install-stamp' +import { createKeepAwakeController } from './keep-awake-controller' import { applyLaunchProfileOverride } from './launch-profile' import { fetchLinkTitle, resolveFaviconCached } from './link-metadata' import { CHROMIUM_LOG_FILENAME, enableLinuxCrashDiagnostics, linuxCrashDiagnostics } from './linux-crash-diagnostics' @@ -423,13 +424,6 @@ import { undialedSshRouteSeeds } from './plugin-profile-routes' import { createPortalSession, resolvePortalBaseUrl } from './portal-session' -import { - createKeepAwake, - type KeepAwakeMode, - keepAwakeWanted, - parseKeepAwakeMode, - readKeepAwakeMode -} from './power-save' import { readPreUpdateBackupEnabled } from './pre-update-backup-config' import { registerPreparedSubmissions } from './prepared-submissions' import { capturePreviewContents } from './preview-capture' @@ -6401,6 +6395,8 @@ function registerPowerResumeListeners() { // 'resume' covers sleep/wake; 'unlock-screen' covers lock/unlock without a // full suspend. Either can drop an idle socket. powerMonitor.on('resume', sendPowerResume) + powerMonitor.on('suspend', () => keepAwake.noteSuspend(powerMonitor.getSystemIdleTime())) + powerMonitor.on('resume', () => keepAwake.noteResume()) powerMonitor.on('unlock-screen', sendPowerResume) powerMonitor.on('on-battery', () => broadcastBatteryState(true)) powerMonitor.on('on-ac', () => broadcastBatteryState(false)) @@ -17139,44 +17135,30 @@ ipcMain.on('hermes:translucency', (_event, payload) => { // merged active-work picture that already drives stream throttling, so it is // held exactly while a turn is in flight and released at idle. See // store/keep-awake + power-save.ts. -const KEEP_AWAKE_CONFIG_PATH = path.join(app.getPath('userData'), 'keep-awake.json') -const keepAwake = createKeepAwake(powerSaveBlocker) -let keepAwakeMode: KeepAwakeMode = 'off' +const keepAwake = createKeepAwakeController({ + blocker: powerSaveBlocker, + configPath: path.join(app.getPath('userData'), 'keep-awake.json'), + isWorking: () => isAnyTurnInFlight(), + log: rememberLog, + // The main chat window only: Quick Entry and the HUD have no toast surface. + offer: () => { + if (!mainWindow || mainWindow.isDestroyed()) { + return false + } -function readPersistedKeepAwakeMode(): KeepAwakeMode { - try { - return readKeepAwakeMode(JSON.parse(fs.readFileSync(KEEP_AWAKE_CONFIG_PATH, 'utf8'))) - } catch { - return 'off' - } -} + mainWindow.webContents.send('hermes:keep-awake-offer') -// Reconcile the blocker with the mode and the live turn picture. Called from -// every mode change and every active-work report — both happen after app -// ready, when `keepAwake` and `keepAwakeMode` above are initialised. -function applyKeepAwake(working = isAnyTurnInFlight()) { - keepAwake.set(keepAwakeWanted(keepAwakeMode, working)) -} - -ipcMain.on('hermes:keep-awake', (_event, value) => { - // Accepts the mode string, or the boolean the pre-mode toggle sent. - const mode = parseKeepAwakeMode(value) - - if (mode === null) { - return - } - - keepAwakeMode = mode - applyKeepAwake() - - try { - fs.mkdirSync(path.dirname(KEEP_AWAKE_CONFIG_PATH), { recursive: true }) - fs.writeFileSync(KEEP_AWAKE_CONFIG_PATH, JSON.stringify({ mode }, null, 2), 'utf8') - } catch (error) { - rememberLog(`[keep-awake] write failed: ${error.message}`) + return true } }) +// Called from every mode change and every active-work report, after app ready. +function applyKeepAwake(working = isAnyTurnInFlight()) { + keepAwake.apply(working) +} + +ipcMain.on('hermes:keep-awake', (_event, value) => keepAwake.setMode(value)) + // Quick Entry: the renderer reads the live registration state on settings mount // and writes the preference back. Main is authoritative — it owns the OS // accelerator — so both handlers return the state that ACTUALLY resulted, @@ -18247,8 +18229,7 @@ app.whenReady().then(() => { ensureWslWindowsFonts() configureSpellChecker() registerPowerResumeListeners() - keepAwakeMode = readPersistedKeepAwakeMode() - applyKeepAwake() + keepAwake.restore() void minimizeToTray.start() mainProcessLagWatchdog.start() f12Blocked = readPersistedDisableF12() diff --git a/apps/desktop/electron/power-save.test.ts b/apps/desktop/electron/power-save.test.ts index 58d1a31bc3..43a4cd0c4c 100644 --- a/apps/desktop/electron/power-save.test.ts +++ b/apps/desktop/electron/power-save.test.ts @@ -77,10 +77,10 @@ describe('keep-awake mode', () => { expect(readKeepAwakeMode({ on: false })).toBe('off') // An explicit mode wins over a stale legacy flag beside it. expect(readKeepAwakeMode({ mode: 'off', on: true })).toBe('off') - // Unreadable or absent -> off, the pre-existing default. - expect(readKeepAwakeMode(null)).toBe('off') - expect(readKeepAwakeMode('garbage')).toBe('off') - expect(readKeepAwakeMode({ mode: 'sometimes' })).toBe('off') + // Unreadable or absent -> the default: awake while a turn runs. + expect(readKeepAwakeMode(null)).toBe('while-working') + expect(readKeepAwakeMode('garbage')).toBe('while-working') + expect(readKeepAwakeMode({ mode: 'sometimes' })).toBe('while-working') }) it('holds the blocker only when the mode and the live turn picture agree', () => { diff --git a/apps/desktop/electron/power-save.ts b/apps/desktop/electron/power-save.ts index 350a41cadf..00f164bded 100644 --- a/apps/desktop/electron/power-save.ts +++ b/apps/desktop/electron/power-save.ts @@ -6,6 +6,9 @@ * owns the preference (persisted in localStorage) and mirrors it here over * IPC; the main process owns the one native blocker, same authority split as * translucency/zoom. Electron auto-releases the blocker on quit. + * + * No Node imports here: the renderer's store imports the modes from this file. + * Main's controller, which persists to disk, is in keep-awake-controller.ts. */ export type KeepAwakeType = 'prevent-app-suspension' | 'prevent-display-sleep' @@ -59,6 +62,14 @@ export type KeepAwakeMode = 'always' | 'off' | 'while-working' const KEEP_AWAKE_MODES: readonly KeepAwakeMode[] = ['off', 'while-working', 'always'] +/** + * The mode on a computer where nobody has picked one. A laptop that sleeps + * mid-turn freezes the turn and, with a local model, the model server too; + * stall timers then cut the request on wake. Holding the blocker only while a + * turn runs costs nothing at idle. + */ +export const DEFAULT_KEEP_AWAKE_MODE: KeepAwakeMode = 'while-working' + /** * Coerce a wire or persisted value to a mode. The pre-mode toggle sent and * stored a boolean; `true` meant what 'always' means now, `false` meant 'off'. @@ -80,12 +91,12 @@ export function parseKeepAwakeMode(value: unknown): KeepAwakeMode | null { /** The main process's persisted copy: the `{ mode }` shape, or the legacy `{ on }`. */ export function readKeepAwakeMode(persisted: unknown): KeepAwakeMode { if (typeof persisted !== 'object' || persisted === null) { - return 'off' + return DEFAULT_KEEP_AWAKE_MODE } const record = persisted as Record - return parseKeepAwakeMode(record.mode) ?? parseKeepAwakeMode(record.on) ?? 'off' + return parseKeepAwakeMode(record.mode) ?? parseKeepAwakeMode(record.on) ?? DEFAULT_KEEP_AWAKE_MODE } /** Whether the blocker should be held right now, given the mode and the live turn picture. */ diff --git a/apps/desktop/electron/preload.ts b/apps/desktop/electron/preload.ts index 53997ec732..0c9bc7ef1b 100644 --- a/apps/desktop/electron/preload.ts +++ b/apps/desktop/electron/preload.ts @@ -383,6 +383,13 @@ contextBridge.exposeInMainWorld('hermesDesktop', { setNativeTheme: mode => ipcRenderer.send('hermes:native-theme', mode), setTranslucency: payload => ipcRenderer.send('hermes:translucency', payload), setKeepAwake: mode => ipcRenderer.send('hermes:keep-awake', mode), + // Once per computer, after a timed sleep interrupted a turn with keep-awake off. + onKeepAwakeOffer: callback => { + const listener = () => callback() + ipcRenderer.on('hermes:keep-awake-offer', listener) + + return () => ipcRenderer.removeListener('hermes:keep-awake-offer', listener) + }, minimizeToTray: { get: () => ipcRenderer.invoke('hermes:minimize-to-tray:get'), set: on => ipcRenderer.invoke('hermes:minimize-to-tray:set', on), diff --git a/apps/desktop/src/global.d.ts b/apps/desktop/src/global.d.ts index 33da11bd05..740881860b 100644 --- a/apps/desktop/src/global.d.ts +++ b/apps/desktop/src/global.d.ts @@ -372,6 +372,7 @@ declare global { localSkin?: { profile: string; skin: HermesSkin } | null setTranslucency?: (payload: TranslucencyState) => void setKeepAwake?: (mode: KeepAwakeMode) => void + onKeepAwakeOffer?: (callback: () => void) => () => void minimizeToTray?: { get: () => Promise<{ enabled: boolean; available: boolean }> set: (on: boolean) => Promise<{ enabled: boolean; available: boolean }> diff --git a/apps/desktop/src/i18n/ar.ts b/apps/desktop/src/i18n/ar.ts index 1948a6b5ad..56e8fee79e 100644 --- a/apps/desktop/src/i18n/ar.ts +++ b/apps/desktop/src/i18n/ar.ts @@ -36,6 +36,7 @@ export const arOverrides = { remoteDisplayBanner: arBoot.remoteDisplayBanner, butterbar: arBoot.butterbar, promptNotices: arBoot.promptNotices, + keepAwakeOffer: arBoot.keepAwakeOffer, titlebar: arChrome.titlebar, keybinds: arChrome.keybinds, language: arSettings.language, diff --git a/apps/desktop/src/i18n/ar_boot.ts b/apps/desktop/src/i18n/ar_boot.ts index 7cf0b4e9f4..54924b032f 100644 --- a/apps/desktop/src/i18n/ar_boot.ts +++ b/apps/desktop/src/i18n/ar_boot.ts @@ -55,6 +55,12 @@ export const arBoot = { identityProvider: 'مزود الهوية' } }, + keepAwakeOffer: { + title: 'دخل جهاز الكمبيوتر في وضع السكون أثناء عمل Hermes', + message: + 'يمكن لـ Hermes إبقاء جهازك مستيقظًا أثناء تشغيل مهمة. ستنطفئ الشاشة كالمعتاد، وقد يدخل الجهاز في وضع السكون عند العمل بالبطارية.', + action: 'إبقاء الجهاز مستيقظًا أثناء العمل' + }, remoteDisplayBanner: { message: reason => `العرض البرمجي نشط — تم اكتشاف شاشة بعيدة (${reason}). تم تعطيل تسريع GPU لمنع الوميض.` }, @@ -339,5 +345,5 @@ export const arBoot = { } } satisfies Pick< TranslationOverrides, - 'boot' | 'remoteDisplayBanner' | 'butterbar' | 'promptNotices' | 'updates' | 'install' | 'onboarding' + 'boot' | 'remoteDisplayBanner' | 'butterbar' | 'promptNotices' | 'keepAwakeOffer' | 'updates' | 'install' | 'onboarding' > diff --git a/apps/desktop/src/i18n/de_notices.ts b/apps/desktop/src/i18n/de_notices.ts index 237d56a2c6..f33a765ccb 100644 --- a/apps/desktop/src/i18n/de_notices.ts +++ b/apps/desktop/src/i18n/de_notices.ts @@ -1,7 +1,7 @@ import { deSessionCommands } from './de_session_commands' import type { TranslationOverrides } from './define-locale' -// Shell notices (remote-display toast, butterbar), spread into de.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into de.ts. export const deNotices = { remoteDisplayBanner: { message: reason => @@ -21,5 +21,11 @@ export const deNotices = { legacySendUnconfirmed: 'Dieser Server konnte das frühere Senden dieser Nachricht nicht bestätigen; sie wurde möglicherweise bereits ausgeführt. Prüfe die Unterhaltung, bevor du sie erneut sendest.', sessionCommands: deSessionCommands + }, + keepAwakeOffer: { + title: 'Ihr Computer ist eingeschlafen, während Hermes gearbeitet hat', + message: + 'Hermes kann den Computer wach halten, solange eine Aufgabe läuft. Der Bildschirm schaltet sich weiterhin aus, und im Akkubetrieb kann der Computer trotzdem einschlafen.', + action: 'Während der Arbeit wach halten' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/i18n/en_notices.ts b/apps/desktop/src/i18n/en_notices.ts index efb02c0de8..1938b20da4 100644 --- a/apps/desktop/src/i18n/en_notices.ts +++ b/apps/desktop/src/i18n/en_notices.ts @@ -1,7 +1,7 @@ import { enSessionCommands } from './en_session_commands' import type { Translations } from './types' -// Shell notices (remote-display toast, butterbar), spread into en.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into en.ts. export const enNotices = { remoteDisplayBanner: { message: reason => @@ -21,5 +21,11 @@ export const enNotices = { legacySendUnconfirmed: 'This server could not confirm the earlier send of this message, so it may already have run. Check the conversation before sending it again.', sessionCommands: enSessionCommands + }, + keepAwakeOffer: { + title: 'Your computer slept while Hermes was working', + message: + 'Hermes can keep your computer awake while a task runs. The screen still turns off, and on battery your computer may still sleep.', + action: 'Keep awake while working' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/i18n/es_notices.ts b/apps/desktop/src/i18n/es_notices.ts index bd6a1610e8..abf5112dcf 100644 --- a/apps/desktop/src/i18n/es_notices.ts +++ b/apps/desktop/src/i18n/es_notices.ts @@ -1,7 +1,7 @@ import type { TranslationOverrides } from './define-locale' import { esSessionCommands } from './es_session_commands' -// Shell notices (remote-display toast, butterbar), spread into es.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into es.ts. export const esNotices = { remoteDisplayBanner: { message: reason => @@ -21,5 +21,11 @@ export const esNotices = { legacySendUnconfirmed: 'Este servidor no pudo confirmar el envío anterior de este mensaje, así que es posible que ya se haya ejecutado. Revisa la conversación antes de volver a enviarlo.', sessionCommands: esSessionCommands + }, + keepAwakeOffer: { + title: 'Tu equipo entró en suspensión mientras Hermes trabajaba', + message: + 'Hermes puede mantener tu equipo activo mientras se ejecuta una tarea. La pantalla se sigue apagando, y con batería el equipo aún puede entrar en suspensión.', + action: 'Mantener activo mientras trabaja' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/i18n/fr_notices.ts b/apps/desktop/src/i18n/fr_notices.ts index 17069ca5bd..b13120cf1d 100644 --- a/apps/desktop/src/i18n/fr_notices.ts +++ b/apps/desktop/src/i18n/fr_notices.ts @@ -1,7 +1,7 @@ import type { TranslationOverrides } from './define-locale' import { frSessionCommands } from './fr_session_commands' -// Shell notices (remote-display toast, butterbar), spread into fr.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into fr.ts. export const frNotices = { remoteDisplayBanner: { message: reason => @@ -21,5 +21,11 @@ export const frNotices = { legacySendUnconfirmed: "Ce serveur n'a pas pu confirmer l'envoi précédent de ce message ; il a peut-être déjà été exécuté. Vérifiez la conversation avant de le renvoyer.", sessionCommands: frSessionCommands + }, + keepAwakeOffer: { + title: 'Votre ordinateur s’est mis en veille pendant que Hermes travaillait', + message: + 'Hermes peut empêcher la mise en veille pendant qu’une tâche s’exécute. L’écran s’éteint toujours, et sur batterie l’ordinateur peut quand même se mettre en veille.', + action: 'Empêcher la veille pendant le travail' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/i18n/ja_notices.ts b/apps/desktop/src/i18n/ja_notices.ts index cad027ccfa..3e832557ed 100644 --- a/apps/desktop/src/i18n/ja_notices.ts +++ b/apps/desktop/src/i18n/ja_notices.ts @@ -1,7 +1,7 @@ import type { TranslationOverrides } from './define-locale' import { jaSessionCommands } from './ja_session_commands' -// Shell notices (remote-display toast, butterbar), spread into ja.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into ja.ts. export const jaNotices = { remoteDisplayBanner: { message: reason => @@ -21,5 +21,11 @@ export const jaNotices = { legacySendUnconfirmed: 'このサーバーはこのメッセージの前回の送信を確認できなかったため、すでに実行された可能性があります。再送信する前に会話を確認してください。', sessionCommands: jaSessionCommands + }, + keepAwakeOffer: { + title: 'Hermes の作業中にコンピューターがスリープしました', + message: + 'タスクの実行中は、Hermes がコンピューターのスリープを防げます。画面はいつもどおりオフになり、バッテリー駆動中はスリープすることがあります。', + action: '実行中はスリープしない' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/i18n/ru_notices.ts b/apps/desktop/src/i18n/ru_notices.ts index deb1b0df6c..e154fb9b65 100644 --- a/apps/desktop/src/i18n/ru_notices.ts +++ b/apps/desktop/src/i18n/ru_notices.ts @@ -1,7 +1,7 @@ import type { TranslationOverrides } from './define-locale' import { ruSessionCommands } from './ru_session_commands' -// Shell notices (remote-display toast, butterbar), spread into ru.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into ru.ts. export const ruNotices = { remoteDisplayBanner: { message: reason => @@ -21,5 +21,11 @@ export const ruNotices = { legacySendUnconfirmed: 'Этот сервер не смог подтвердить предыдущую отправку этого сообщения, поэтому оно, возможно, уже выполнено. Проверьте беседу, прежде чем отправлять его снова.', sessionCommands: ruSessionCommands + }, + keepAwakeOffer: { + title: 'Компьютер ушёл в сон, пока Hermes работал', + message: + 'Hermes может не давать компьютеру засыпать, пока выполняется задача. Экран по-прежнему выключается, а от батареи компьютер всё равно может уснуть.', + action: 'Не засыпать во время работы' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/i18n/types_notices.ts b/apps/desktop/src/i18n/types_notices.ts index d0c078c6c6..121e85cc19 100644 --- a/apps/desktop/src/i18n/types_notices.ts +++ b/apps/desktop/src/i18n/types_notices.ts @@ -1,5 +1,5 @@ -// Shell notice strings: the remote-display toast, the butterbar and send-outcome toasts; -// `Translations` spreads this in. +// Shell notice strings: the remote-display toast, the butterbar, send-outcome toasts and the +// keep-awake offer; `Translations` spreads this in. export interface NoticeTranslations { remoteDisplayBanner: { message: (reason: string) => string @@ -16,6 +16,9 @@ export interface NoticeTranslations { /** Typed /undo /retry /steer /queue lines (the Ink TUI's wording on the same gateway). */ sessionCommands: SessionCommandNotices } + + /** Once per computer, after a timed sleep interrupted a turn with keep-awake off. */ + keepAwakeOffer: { title: string; message: string; action: string } } export interface SessionCommandNotices { diff --git a/apps/desktop/src/i18n/zh-hant.ts b/apps/desktop/src/i18n/zh-hant.ts index 27ea112b84..56b480e458 100644 --- a/apps/desktop/src/i18n/zh-hant.ts +++ b/apps/desktop/src/i18n/zh-hant.ts @@ -83,6 +83,7 @@ export const zhHantOverrides = { remoteDisplayBanner: zhHantBoot.remoteDisplayBanner, butterbar: zhHantBoot.butterbar, promptNotices: zhHantBoot.promptNotices, + keepAwakeOffer: zhHantBoot.keepAwakeOffer, billingBlock: zhHantCommon.billingBlock, sendDiagnostics: zhHantDiagnostics.sendDiagnostics, titlebar: zhHantChrome.titlebar, diff --git a/apps/desktop/src/i18n/zh-hant_boot.ts b/apps/desktop/src/i18n/zh-hant_boot.ts index 3eaf5ebf91..94308fe225 100644 --- a/apps/desktop/src/i18n/zh-hant_boot.ts +++ b/apps/desktop/src/i18n/zh-hant_boot.ts @@ -59,6 +59,11 @@ export const zhHantBoot = { } }, + keepAwakeOffer: { + title: '電腦在 Hermes 工作時進入了睡眠', + message: 'Hermes 可以在工作執行時保持電腦喚醒。螢幕仍會照常關閉，使用電池時電腦仍可能進入睡眠。', + action: '執行期間保持喚醒' + }, remoteDisplayBanner: { message: reason => `軟體繪圖已啟用 — 偵測到遠端顯示（${reason}）。為防止畫面閃爍，已停用 GPU 加速。` }, @@ -343,5 +348,5 @@ export const zhHantBoot = { } } satisfies Pick< TranslationOverrides, - 'boot' | 'remoteDisplayBanner' | 'butterbar' | 'promptNotices' | 'updates' | 'install' | 'onboarding' + 'boot' | 'remoteDisplayBanner' | 'butterbar' | 'promptNotices' | 'keepAwakeOffer' | 'updates' | 'install' | 'onboarding' > diff --git a/apps/desktop/src/i18n/zh_notices.ts b/apps/desktop/src/i18n/zh_notices.ts index 828dcc4a4b..360c716553 100644 --- a/apps/desktop/src/i18n/zh_notices.ts +++ b/apps/desktop/src/i18n/zh_notices.ts @@ -1,7 +1,7 @@ import type { TranslationOverrides } from './define-locale' import { zhSessionCommands } from './zh_session_commands' -// Shell notices (remote-display toast, butterbar), spread into zh.ts. +// Shell notices (remote-display toast, butterbar, keep-awake offer), spread into zh.ts. export const zhNotices = { remoteDisplayBanner: { message: reason => `软件渲染已启用 — 检测到远程显示（${reason}）。为防止画面闪烁，已禁用 GPU 加速。` @@ -19,5 +19,10 @@ export const zhNotices = { promptNotices: { legacySendUnconfirmed: '此服务器无法确认这条消息之前的发送，它可能已经运行过。再次发送前请先查看对话。', sessionCommands: zhSessionCommands + }, + keepAwakeOffer: { + title: '电脑在 Hermes 工作时进入了睡眠', + message: 'Hermes 可以在任务运行时保持电脑唤醒。屏幕仍会照常关闭，使用电池时电脑仍可能睡眠。', + action: '运行期间保持唤醒' } -} satisfies Pick +} satisfies Pick diff --git a/apps/desktop/src/main.tsx b/apps/desktop/src/main.tsx index ba7996e704..4ef94636b9 100644 --- a/apps/desktop/src/main.tsx +++ b/apps/desktop/src/main.tsx @@ -9,6 +9,9 @@ import './store/translucency' import './store/user-bubble-transparency' // Side-effect: restores chat typography before the first conversation paints. import './store/chat-text-scale' +// Side-effect: mirrors the keep-awake mode to main and listens for main's +// one-time keep-awake offer. Settings loads lazily, too late for the offer. +import './store/keep-awake' // Dev-only render/state churn counters. MUST precede the `react-dom` import // below: react-dom captures the devtools hook at module init, so bippy has to // install during THIS import's evaluation or every commit goes unseen diff --git a/apps/desktop/src/store/keep-awake.test.ts b/apps/desktop/src/store/keep-awake.test.ts index a4b5ffc836..72e1025c8b 100644 --- a/apps/desktop/src/store/keep-awake.test.ts +++ b/apps/desktop/src/store/keep-awake.test.ts @@ -40,10 +40,10 @@ describe('keep-awake store', () => { expect(setKeepAwakeBridge).toHaveBeenLastCalledWith('off') }) - it('defaults to off on a machine with no preference', async () => { + it('defaults to while-working on a machine with no preference', async () => { const { $keepAwakeMode } = await loadStore() - expect($keepAwakeMode.get()).toBe('off') + expect($keepAwakeMode.get()).toBe('while-working') }) it('migrates the pre-mode toggle: on meant always, off meant off', async () => { @@ -65,9 +65,34 @@ describe('keep-awake store', () => { expect((await loadStore()).$keepAwakeMode.get()).toBe('while-working') }) - it('ignores an unrecognised saved mode instead of trusting it', async () => { + it('ignores an unrecognised saved mode and falls back to the default', async () => { writeKey(KEY, 'sometimes') - expect((await loadStore()).$keepAwakeMode.get()).toBe('off') + expect((await loadStore()).$keepAwakeMode.get()).toBe('while-working') + }) + it("shows main's offer, and its button turns on while-working", async () => { + let deliverOffer: (() => void) | undefined + + desktopWindow.hermesDesktop = { + onKeepAwakeOffer: (callback: () => void) => { + deliverOffer = callback + + return () => {} + }, + setKeepAwake: setKeepAwakeBridge + } as unknown as Window['hermesDesktop'] + writeKey(KEY, 'off') + + const { $keepAwakeMode } = await loadStore() + const { $notifications, clearNotifications } = await import('./notifications') + + deliverOffer?.() + clearNotifications() // sending the next prompt clears turn feedback, not the offer + const offer = $notifications.get().find(item => item.id === 'keep-awake-offer') + + expect(offer?.action?.label).toBe('Keep awake while working') + offer?.action?.onClick() + expect($keepAwakeMode.get()).toBe('while-working') + expect(setKeepAwakeBridge).toHaveBeenLastCalledWith('while-working') }) }) diff --git a/apps/desktop/src/store/keep-awake.ts b/apps/desktop/src/store/keep-awake.ts index 68e60e7bd6..b8231e9a09 100644 --- a/apps/desktop/src/store/keep-awake.ts +++ b/apps/desktop/src/store/keep-awake.ts @@ -1,7 +1,7 @@ /** * Keep-awake — stop the machine sleeping during long, unattended runs. * - * A device-local preference (each computer keeps its own), off by default. This + * A device-local preference (each computer keeps its own), 'while-working' by default. This * atom backs the Settings → Advanced picker and mirrors changes to the main * process, which owns the real power-save blocker AND its own persisted copy — * so a cold launch restores the blocker without the renderer visiting Settings @@ -16,9 +16,11 @@ import { atom } from 'nanostores' +import { translateNow } from '@/i18n' import { persistString, storedBoolean, storedString } from '@/lib/storage' +import { notify } from '@/store/notifications' -import { type KeepAwakeMode, parseKeepAwakeMode } from '../../electron/power-save' +import { DEFAULT_KEEP_AWAKE_MODE, type KeepAwakeMode, parseKeepAwakeMode } from '../../electron/power-save' export type { KeepAwakeMode } @@ -34,18 +36,38 @@ function initialKeepAwakeMode(): KeepAwakeMode { } // No mode saved yet: honour what the old toggle said. On meant "always". - return storedBoolean(LEGACY_KEY, false) ? 'always' : 'off' + if (storedString(LEGACY_KEY) !== null) { + return storedBoolean(LEGACY_KEY, false) ? 'always' : 'off' + } + + return DEFAULT_KEEP_AWAKE_MODE } -export const $keepAwakeMode = atom(typeof window === 'undefined' ? 'off' : initialKeepAwakeMode()) +export const $keepAwakeMode = atom( + typeof window === 'undefined' ? DEFAULT_KEEP_AWAKE_MODE : initialKeepAwakeMode() +) export function setKeepAwakeMode(mode: KeepAwakeMode): void { $keepAwakeMode.set(mode) } +/** The one-time offer main sends after a timed sleep interrupted a turn with keep-awake off. */ +export function offerKeepAwake(): void { + notify({ + id: 'keep-awake-offer', + kind: 'info', + title: translateNow('keepAwakeOffer.title'), + message: translateNow('keepAwakeOffer.message'), + action: { label: translateNow('keepAwakeOffer.action'), onClick: () => setKeepAwakeMode('while-working') }, + durationMs: 0, + persistent: true + }) +} + if (typeof window !== 'undefined') { $keepAwakeMode.subscribe(mode => { persistString(KEY, mode) window.hermesDesktop?.setKeepAwake?.(mode) }) + window.hermesDesktop?.onKeepAwakeOffer?.(offerKeepAwake) } diff --git a/apps/desktop/src/store/notifications.ts b/apps/desktop/src/store/notifications.ts index d205531eb2..7548a6465a 100644 --- a/apps/desktop/src/store/notifications.ts +++ b/apps/desktop/src/store/notifications.ts @@ -33,6 +33,8 @@ export interface AppNotification { onDismiss?: () => void createdAt: number placement?: NotificationPlacement + /** Survives clearNotifications: a standing offer, not feedback on the last turn. */ + persistent?: boolean } export interface NotificationInput { @@ -49,6 +51,7 @@ export interface NotificationInput { onDismiss?: () => void durationMs?: number placement?: NotificationPlacement + persistent?: boolean } let notificationCounter = 0 @@ -271,7 +274,8 @@ function showNotification(input: NotificationInput, errorCategory: ErrorToastCat secondaryAction: input.secondaryAction, onDismiss: input.onDismiss, createdAt: Date.now(), - placement: input.placement ?? defaultPlacement(kind, input.action) + placement: input.placement ?? defaultPlacement(kind, input.action), + persistent: input.persistent } window.clearTimeout(timers.get(id)) @@ -339,15 +343,13 @@ export function dismissNotification(id: string) { } export function clearNotifications() { - for (const timer of timers.values()) { - window.clearTimeout(timer) - } - - timers.clear() const all = $notifications.get() - $notifications.set([]) + const cleared = all.filter(item => !item.persistent) + $notifications.set(all.filter(item => item.persistent)) - for (const item of all) { + for (const item of cleared) { + window.clearTimeout(timers.get(item.id)) + timers.delete(item.id) item.onDismiss?.() } } diff --git a/gateway/run_turn_hygiene.py b/gateway/run_turn_hygiene.py index 2688b6ca6f..2d379a973c 100644 --- a/gateway/run_turn_hygiene.py +++ b/gateway/run_turn_hygiene.py @@ -230,9 +230,9 @@ class GatewayTurnHygieneMixin: async def _hmwa_hygiene_wait_for_summary(self, attempt, hs, session_entry): """Progress-aware inline wait for the detached hygiene compressor. Returns the compressed transcript; raises ``HygieneTurnHoldExceeded`` (turn-hold budget) or - ``asyncio.TimeoutError`` (idle/ceiling/fence cancel) for the caller's handlers. - - Idle timeout (fence ticks per streamed token) + hard ceiling + turn-hold cap.""" + ``asyncio.TimeoutError`` (idle/ceiling/fence cancel) for the caller's handlers. Idle timeout + (fence ticks per streamed token) + hard ceiling (moved back by prompt reading) + turn-hold cap.""" + from agent.conversation_compression_deadline import effective_ceiling from gateway.run import HygieneTurnHoldExceeded, hygiene_wait_should_extend fence = attempt.commit_fence while True: @@ -240,10 +240,9 @@ class GatewayTurnHygieneMixin: raise TimeoutError # Charge the idle budget from the LAST PROGRESS event, else silence can approach 2x timeout. _hyg_waited = time.monotonic() - attempt.wait_started - _slice = min( - max(hs.timeout_seconds - fence.seconds_since_progress(), 0.005), - max(hs.total_ceiling_seconds - _hyg_waited, 0.005), - ) + _ceiling = effective_ceiling(fence, hs.total_ceiling_seconds, attempt.wait_started) + _slice = min(max(hs.timeout_seconds - fence.seconds_since_progress(), 0.005), + max(_ceiling - _hyg_waited, 0.005)) # Cap the slice at the remaining turn-hold budget so a continuously-streaming worker can't # hold the turn until the ceiling. Budget exhausted → immediate timeout → abandonment. _turn_hold_remaining = hs.max_turn_hold_seconds - (time.monotonic() - attempt.wait_started) @@ -258,6 +257,7 @@ class GatewayTurnHygieneMixin: if fence.is_cancelled: raise _hyg_waited = time.monotonic() - attempt.wait_started + _ceiling = effective_ceiling(fence, hs.total_ceiling_seconds, attempt.wait_started) _idle = fence.seconds_since_progress() # Never hold the TURN past the budget even while the summary streams: proceed on the # uncompressed transcript so the wire never trips a transport idle-timeout. @@ -274,13 +274,13 @@ class GatewayTurnHygieneMixin: ) if hygiene_wait_should_extend( idle=_idle, timeout=hs.timeout_seconds, waited=_hyg_waited, - ceiling=hs.total_ceiling_seconds, fence_cancelled=fence.is_cancelled, + ceiling=_ceiling, fence_cancelled=fence.is_cancelled, ): if _slice >= _idle_left - 1e-9: logger.info( "Session hygiene compression for session %s still streaming after " "%.0fs (last progress %.1fs ago) — extending wait (ceiling %.0fs)", - session_entry.session_id, _hyg_waited, _idle, hs.total_ceiling_seconds, + session_entry.session_id, _hyg_waited, _idle, _ceiling, ) continue raise @@ -424,8 +424,7 @@ class GatewayTurnHygieneMixin: worker crossed the commit boundary first.""" from gateway.run import _hygiene_compression_timeout_message fence = attempt.commit_fence - _hyg_waited = time.monotonic() - attempt.wait_started - _hyg_total_exhausted = _hyg_waited >= hs.total_ceiling_seconds or fence.deadline_exceeded + _hyg_total_exhausted = fence.deadline_exceeded # armed with the ceiling before the wait began if _hyg_total_exhausted: # The worker checks this deadline between digest calls; keep its lease until it exits so # an unchanged session cannot overlap a retry (the release below is then a no-op). diff --git a/hermes_cli/local_runtime/bootstrap.py b/hermes_cli/local_runtime/bootstrap.py index e553fee980..749e0e65f2 100644 --- a/hermes_cli/local_runtime/bootstrap.py +++ b/hermes_cli/local_runtime/bootstrap.py @@ -214,13 +214,15 @@ def _admitted_models_max(mdir: Path, configured: int) -> int: A cap of "four" on a card that holds one model is how a second child ends up paged (WDDM) and silently slow — llama.cpp evicts its LRU before an incoming load only when the cap says the card is full. A probe miss or an unpriceable model keeps the configured count: this must never - block a boot. + block a boot. On Windows unified memory, free commit bounds the count too. """ try: from hermes_cli.local_runtime.hardware import probe_budget from hermes_cli.local_runtime.presets import admitted_residency_count - cap = admitted_residency_count(mdir, probe_budget(planning=True), configured) + capacity = probe_budget(planning=True) + live = _launch_budget(capacity) if capacity.uma else None + cap = admitted_residency_count(mdir, capacity, configured, live=live) except Exception as exc: logger.warning("residency cap probe failed (%s); using models_max=%s", exc, configured) return configured @@ -230,7 +232,7 @@ def _admitted_models_max(mdir: Path, configured: int) -> int: return cap -def _generate_presets(mdir: Path, preset_path: Path) -> Path | None: +def _generate_presets(mdir: Path, preset_path: Path, residents: int = 1) -> Path | None: """Write the launch-policy INI for every staged model; returns the path to hand the router. Priced against CAPACITY, not live free VRAM: this runs while the outgoing server instance may @@ -243,13 +245,16 @@ def _generate_presets(mdir: Path, preset_path: Path) -> Path | None: Degradation ladder on failure: a STALE policy still beats no policy — stock fit (f16 KV at max context, no placement) is the silent-busy-wait failure on Windows. Keep serving with the previous INI when one exists; only a first boot with no INI at all falls to stock fit. + ``residents`` is the router's model count, which splits the prompt-cache room on Windows + unified memory. """ from hermes_cli.local_runtime.hardware import probe_budget from hermes_cli.local_runtime.presets import generate_presets try: capacity = probe_budget(planning=True) - for entry in generate_presets(mdir, capacity, preset_path, live=_launch_budget(capacity)): + for entry in generate_presets(mdir, capacity, preset_path, live=_launch_budget(capacity), + residents=residents): if entry.refusal: logger.warning("model refused by physics check: %s", entry.refusal) return preset_path @@ -303,7 +308,7 @@ def refit_idle_presets(sup) -> bool: last = sup._refit_usable if last is not None and abs(live.usable_vram_bytes - last) < _REFIT_STEP_BYTES: return False - text = render_presets(plan_presets(models_dir(), capacity, live=live)) + text = render_presets(plan_presets(models_dir(), capacity, live=live, residents=sup.models_max)) # Held across the write and the reload so a restart can't spawn between them; everything # slow ran above. with sup._lifecycle_lock: @@ -453,11 +458,11 @@ def ensure_local_runtime(config: dict, force: bool = False) -> object | None: mdir = models_dir() mdir.mkdir(parents=True, exist_ok=True) - preset_path = _generate_presets(mdir, runtimes_root() / "presets.ini") + models_max = _admitted_models_max(mdir, int(section.get("models_max", 4))) + preset_path = _generate_presets(mdir, runtimes_root() / "presets.ini", models_max) sup = LlamaServerSupervisor(engine.binary, mdir, preset_path=preset_path, - models_max=_admitted_models_max( - mdir, int(section.get("models_max", 4))), + models_max=models_max, port=int(section.get("port", 0)) or None) try: sup.start() diff --git a/hermes_cli/local_runtime/capabilities.py b/hermes_cli/local_runtime/capabilities.py index 3c69a2ff1e..02414695f9 100644 --- a/hermes_cli/local_runtime/capabilities.py +++ b/hermes_cli/local_runtime/capabilities.py @@ -41,12 +41,12 @@ def _props_modalities(model_id: str) -> bool | None: """Ask the running server whether this loaded child sees images. None when the server is down, the model isn't loaded, or the build doesn't report modalities.""" with suppress(Exception): - from hermes_cli.local_runtime.endpoint import managed_get_json, managed_root + from hermes_cli.local_runtime.endpoint import child_query, managed_get_json, managed_root ep = managed_root() if ep is None: return None - modalities = managed_get_json(*ep, f"/props?model={model_id}", timeout_s=3).get("modalities") + modalities = managed_get_json(*ep, f"/props?{child_query(model_id)}", timeout_s=3).get("modalities") if isinstance(modalities, dict) and "vision" in modalities: return bool(modalities["vision"]) return None diff --git a/hermes_cli/local_runtime/endpoint.py b/hermes_cli/local_runtime/endpoint.py index 4e5d3542fb..2abb34cfc7 100644 --- a/hermes_cli/local_runtime/endpoint.py +++ b/hermes_cli/local_runtime/endpoint.py @@ -16,6 +16,18 @@ import urllib.request LLAMACPP_ALIASES = frozenset({"llamacpp", "llama.cpp", "llama-cpp"}) + +def child_query(model_id: str) -> str: + """Query string for a status read of one router child (``/props``, ``/slots``, ``/metrics``). + + The router loads an unloaded model for any request that names it (``--models-autoload``), so a + status read right after an Eject or an idle unload would load the model straight back. + ``autoload=false`` makes the router answer 400 "model is not loaded" instead. + """ + from urllib.parse import quote + + return f"model={quote(model_id)}&autoload=false" + logger = logging.getLogger(__name__) diff --git a/hermes_cli/local_runtime/estimator.py b/hermes_cli/local_runtime/estimator.py index 13f88e39c4..6f35632e17 100644 --- a/hermes_cli/local_runtime/estimator.py +++ b/hermes_cli/local_runtime/estimator.py @@ -117,6 +117,10 @@ class HardwareBudget: # except integrated GPUs (b11370 #28160); Hermes passes --lazy-mode on to NVIDIA's, so only # AMD/Intel integrated GPUs load them up front. lazy_reads: bool = True + # Windows unified memory, live budgets only: commit the server may take for the models and + # their prompt caches. Windows charges every GPU allocation to commit (RAM it can see plus + # the page file), so this is the real ceiling there. None everywhere else. + commit_room_bytes: int | None = None def as_loaded(profile: ModelProfile, budget: HardwareBudget) -> ModelProfile: diff --git a/hermes_cli/local_runtime/hardware.py b/hermes_cli/local_runtime/hardware.py index b2976b1f57..1565cf12fc 100644 --- a/hermes_cli/local_runtime/hardware.py +++ b/hermes_cli/local_runtime/hardware.py @@ -40,6 +40,11 @@ _LAUNCH_HEADROOM = 1 << 30 # UMA headroom: on unified-memory machines the model shares physical memory with the OS and every # app, so budget from RAM minus this fraction. _UMA_HEADROOM_FRACTION = 0.20 +# llama-server's prompt cache: its own default size, and the smallest worth keeping. The cache holds +# conversations the server switched away from (a background job, an earlier chat); without it each +# switch re-reads the whole conversation. +_PROMPT_CACHE_MAX = 8 << 30 +_PROMPT_CACHE_MIN = 512 << 20 # Engine-fallback gates for the unified-pool quirk — BOTH must hold, and no discrete card can # meet either: (1) the allocator's pool exceeds the smi report by well past rounding/ECC slack @@ -118,23 +123,31 @@ def _linux_ram_from_meminfo(text: str) -> tuple[int, int] | None: return None -def _ram_bytes() -> tuple[int, int]: - """(total, available) physical memory, cross-platform stdlib.""" +def _windows_memory_status(): + """GlobalMemoryStatusEx, or None off Windows.""" + import ctypes + + class MEMORYSTATUSEX(ctypes.Structure): + _fields_ = ([("dwLength", ctypes.c_ulong), ("dwMemoryLoad", ctypes.c_ulong)] + + [(name, ctypes.c_ulonglong) for name in ( + "ullTotalPhys", "ullAvailPhys", "ullTotalPageFile", "ullAvailPageFile", + "ullTotalVirtual", "ullAvailVirtual", "ullAvailExtendedVirtual")]) + try: - import ctypes - - class MEMORYSTATUSEX(ctypes.Structure): - _fields_ = ([("dwLength", ctypes.c_ulong), ("dwMemoryLoad", ctypes.c_ulong)] - + [(name, ctypes.c_ulonglong) for name in ( - "ullTotalPhys", "ullAvailPhys", "ullTotalPageFile", "ullAvailPageFile", - "ullTotalVirtual", "ullAvailVirtual", "ullAvailExtendedVirtual")]) - stat = MEMORYSTATUSEX() stat.dwLength = ctypes.sizeof(MEMORYSTATUSEX) - ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)) - return stat.ullTotalPhys, stat.ullAvailPhys + if not ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)): + return None + return stat except (AttributeError, OSError): - pass + return None + + +def _ram_bytes() -> tuple[int, int]: + """(total, available) physical memory, cross-platform stdlib.""" + stat = _windows_memory_status() + if stat is not None: + return stat.ullTotalPhys, stat.ullAvailPhys try: if sys.platform == "darwin": # macOS getconf has no _PHYS_PAGES/_AVPHYS_PAGES (exit 64) — the POSIX branch would @@ -405,6 +418,44 @@ def _uma_budget(base: int, total: int, *, gpu_name: str = "", gpu_pci_id=gpu_pci_id, lazy_reads=lazy_reads) +def _windows_commit_room(own_bytes: int = 0) -> int | None: + """Commit the managed server may take on Windows right now, or None off Windows. + + Windows charges every GPU allocation to commit: the RAM Windows can see plus the page file. + Memory carved out for the GPU is invisible to Windows, so a bigger carve-out leaves less + commit for the model it was meant to hold. ``own_bytes`` is what the managed server holds now + and frees before the new instance loads. A desktop margin stays free. + """ + stat = _windows_memory_status() + if stat is None or not stat.ullTotalPageFile: + return None + margin = max(_MARGIN_FLOOR, int(stat.ullTotalPhys * _MARGIN_FRACTION)) + return max(0, int(stat.ullAvailPageFile) + max(0, own_bytes) - margin) + + +def model_commit_bytes(model_bytes: int) -> int: + """What Windows charges to commit for a model estimated at ``model_bytes`` of device memory: + the estimate runs ~20% under the charge, which is the UMA headroom.""" + return int(model_bytes / (1 - _UMA_HEADROOM_FRACTION)) + + +def prompt_cache_bytes(live: HardwareBudget | None, model_bytes: int, residents: int = 1) -> int | None: + """Size for llama-server's prompt cache beside a model of ``model_bytes`` (the estimate the + window was planned with), or None when the engine default applies. + + Windows unified memory only: each loaded model's cache comes out of the same commit as the + models, so each of the ``residents`` models the router may hold at once (its ``--models-max``) + gets an even share of what they leave. + """ + room = live.commit_room_bytes if live is not None else None + if room is None or model_bytes <= 0: + return None + residents = max(1, residents) + model_commit = model_commit_bytes(model_bytes) + cache = min(_PROMPT_CACHE_MAX, max(0, room - residents * model_commit) // residents) + return cache if cache >= _PROMPT_CACHE_MIN else 0 + + def _gpu_engine_runs_the_model() -> bool: """Whether a Vulkan/HIP engine runs the model: the serving or installed engine, else the one setup would install for ``local_runtime.backend``. False when the engine can't be resolved.""" @@ -451,9 +502,9 @@ def probe_budget(*, planning: bool = False) -> HardwareBudget: # Unified-memory NVIDIA: the CUDA allocator pool is the real capacity. Classification comes # from the driver API/engine and must not require nvidia-smi (stripped-PATH sessions lose smi - # but nvcuda loads via the system loader). Crossing the carve-out costs nothing — it is an OS - # accounting knob, not a GPU limit. Deliberately NOT clamped to OS RAM: carved-out memory is - # invisible to GlobalMemoryStatusEx, so a RAM clamp would throw away exactly that capacity. + # but nvcuda loads via the system loader). Deliberately NOT clamped to OS RAM: carved-out + # memory is invisible to GlobalMemoryStatusEx, so a RAM clamp would throw away exactly that + # capacity. On Windows, commit narrows the launch window instead (``launch_budget``). unified = _unified_pool_bytes(vram[0] if vram else 0, ram_total) if unified is not None: logger.info( @@ -506,14 +557,19 @@ def launch_budget(capacity: HardwareBudget, *, own_bytes: int = 0) -> HardwareBu an error: a 32 GiB card with 4.5-6.3 GiB held by other apps decoded at ~24 tok/s at the capacity window, and at ~90 tok/s at a window sized from free memory. - ``own_bytes`` is what the managed server holds now and frees before the new instance loads. - A stopped server's memory reads as free by the time its process has exited (measured on - Windows: free memory was fully back at the first reading after exit). None for unified memory - (its live budget is already free memory) and when the device query fails, so callers keep the - capacity plan. + ``own_bytes`` is the device-memory estimate of what the managed server holds now and frees + before the new instance loads. A stopped server's memory reads as free by the time its process + has exited (measured on Windows: free memory was fully back at the first reading after exit). + Unified memory: free commit on Windows (the model and its prompt cache must fit there), with + ``own_bytes`` converted to the commit Windows charged for it, else None (its live budget is + already free memory). None too when the device query fails, so callers keep the capacity plan. """ if capacity.uma: - return None + room = _windows_commit_room(model_commit_bytes(own_bytes) if own_bytes > 0 else 0) + if room is None: + return None + usable = min(capacity.usable_vram_bytes, int(room * (1 - _UMA_HEADROOM_FRACTION))) + return replace(capacity, usable_vram_bytes=usable, commit_room_bytes=room) vram = _nvidia_vram() if vram is not None: total, free = vram[0], vram[1] diff --git a/hermes_cli/local_runtime/load_progress.py b/hermes_cli/local_runtime/load_progress.py index 96bb1ca0b2..8c6a4a1429 100644 --- a/hermes_cli/local_runtime/load_progress.py +++ b/hermes_cli/local_runtime/load_progress.py @@ -143,11 +143,9 @@ def get_prefill_progress(model: str) -> dict | None: if ep is None: return None try: - from urllib.parse import quote + from hermes_cli.local_runtime.endpoint import child_query, managed_get_json - from hermes_cli.local_runtime.endpoint import managed_get_json - - slots = managed_get_json(*ep, f"/slots?model={quote(model)}", timeout_s=2) + slots = managed_get_json(*ep, f"/slots?{child_query(model)}", timeout_s=2) except Exception: return None best = 0 diff --git a/hermes_cli/local_runtime/presets.py b/hermes_cli/local_runtime/presets.py index 7a57a7bd5a..c0af3d860f 100644 --- a/hermes_cli/local_runtime/presets.py +++ b/hermes_cli/local_runtime/presets.py @@ -100,11 +100,12 @@ def _draft_fits(path: Path, profile, budget: HardwareBudget, window: int, overhe def preset_for_model(gguf: Path, budget: HardwareBudget, mtp_capable: set[str], *, requested_window: int | None = None, - live: HardwareBudget | None = None) -> PresetEntry | None: + live: HardwareBudget | None = None, residents: int = 1) -> PresetEntry | None: """The launch decision for one staged model, or None when its header is unreadable. ``budget`` is the card's capacity; ``live`` (from ``hardware.launch_budget``) narrows the - window to what fits beside other programs now. + window to what fits beside other programs now. ``residents`` is how many models the router may + hold at once; on Windows unified memory their prompt caches share the commit they leave. """ from hermes_cli.local_runtime.catalog import entry_for_model from hermes_cli.local_runtime.growth import load_window_overrides @@ -136,6 +137,11 @@ def preset_for_model(gguf: Path, budget: HardwareBudget, profile, decision, mtp_capable=is_mtp, uma=budget.uma, mtp_prefill=plan.mtp_prefill, mtp_draft_depth=entry.mtp_draft_depth if entry is not None else 3)) keys["model"] = str(gguf) + from hermes_cli.local_runtime.hardware import prompt_cache_bytes + cache = prompt_cache_bytes(live, footprint_bytes(plan.profile, decision.window, + overhead_bytes=plan.overhead_bytes), residents) + if cache is not None: + keys["cache-ram"] = str(cache >> 20) # MiB; 0 turns the cache off if profile.lazy_bytes: # The price above leaves these tensors on disk; llama.cpp's own default loads them on # integrated GPUs. @@ -215,7 +221,8 @@ def _launch_footprint(gguf: Path, budget: HardwareBudget) -> int | None: return footprint_bytes(plan.profile, plan.decision.window, overhead_bytes=plan.overhead_bytes) -def admitted_residency_count(models_dir: Path, budget: HardwareBudget, configured: int) -> int: +def admitted_residency_count(models_dir: Path, budget: HardwareBudget, configured: int, + live: HardwareBudget | None = None) -> int: """How many models the card may hold resident at once: priced against the budget, not a count. Residency used to be bounded by a count alone, so a second model was admitted against an @@ -228,7 +235,8 @@ def admitted_residency_count(models_dir: Path, budget: HardwareBudget, configure The cap rises above one only while the LARGEST staged model still fits TWICE — any pair of staged models then fits by construction. ``configured`` stays a ceiling (a user's smaller number is honoured), and an unpriceable input (no usable device memory, no readable model) - keeps today's behaviour. + keeps today's behaviour. On Windows unified memory ``live`` (free commit) bounds it too: the + models the router holds together must fit in commit, not just in the pool. """ from hermes_cli.local_runtime.bootstrap import staged_in @@ -241,25 +249,28 @@ def admitted_residency_count(models_dir: Path, budget: HardwareBudget, configure largest = max(largest, need) if largest <= 0: return configured - return max(1, min(configured, budget.usable_vram_bytes // largest)) + usable = budget.usable_vram_bytes + if live is not None and live.commit_room_bytes is not None: + usable = min(usable, live.usable_vram_bytes) + return max(1, min(configured, usable // largest)) def plan_presets(models_dir: Path, budget: HardwareBudget, mtp_capable: set[str] | None = None, - *, live: HardwareBudget | None = None) -> list[PresetEntry]: + *, live: HardwareBudget | None = None, residents: int = 1) -> list[PresetEntry]: """The launch decision for every staged model; unreadable headers are skipped.""" from hermes_cli.local_runtime.bootstrap import staged_in - entries = (preset_for_model(gguf, budget, mtp_capable or set(), live=live) + entries = (preset_for_model(gguf, budget, mtp_capable or set(), live=live, residents=residents) for gguf in staged_in(models_dir)) return [entry for entry in entries if entry is not None] def generate_presets(models_dir: Path, budget: HardwareBudget, preset_path: Path, mtp_capable: set[str] | None = None, *, - live: HardwareBudget | None = None) -> list[PresetEntry]: + live: HardwareBudget | None = None, residents: int = 1) -> list[PresetEntry]: """Plan every staged model and write one INI. Refused models get no section (the picker surfaces the refusal from the returned entries).""" - entries = plan_presets(models_dir, budget, mtp_capable, live=live) + entries = plan_presets(models_dir, budget, mtp_capable, live=live, residents=residents) write_presets(entries, preset_path) return entries diff --git a/hermes_cli/local_runtime/supervisor.py b/hermes_cli/local_runtime/supervisor.py index 741f402584..d2ef1b50d7 100644 --- a/hermes_cli/local_runtime/supervisor.py +++ b/hermes_cli/local_runtime/supervisor.py @@ -25,6 +25,7 @@ import urllib.request from pathlib import Path from hermes_cli.local_runtime.binaries import runtimes_root +from hermes_cli.local_runtime.endpoint import child_query from hermes_cli.local_runtime.processes import server_child_env, spawn_server logger = logging.getLogger(__name__) @@ -460,15 +461,21 @@ class LlamaServerSupervisor: def _probe_idle(self, model_id: str | None = None) -> bool | None: """Tri-state idle probe for the sweeper: True = confirmed idle, False = confirmed busy, None = the probe itself failed. The sweeper must never mistake a dead probe - for activity — that resets the idle clock and pins the model's VRAM.""" + for activity — that resets the idle clock and pins the model's VRAM. A model that + isn't loaded is idle: the router answers 400 for it (the reads don't autoload).""" try: loaded = ([model_id] if model_id is not None else [m for m, status in self.models().items() if status in _RESIDENT]) for mid in loaded: - slots = self._request(f"/slots?model={mid}") + try: + slots = self._request(f"/slots?{child_query(mid)}") + except urllib.error.HTTPError as exc: + if exc.code == 400: + continue + raise if any(s.get("is_processing") for s in slots): return False - with self._open(f"/metrics?model={mid}", timeout_s=10, json_type=False) as r: + with self._open(f"/metrics?{child_query(mid)}", timeout_s=10, json_type=False) as r: text = r.read().decode() for line in text.splitlines(): if (line.startswith("llamacpp:requests_processing") diff --git a/hermes_cli/web_routers/local_models.py b/hermes_cli/web_routers/local_models.py index 44ad06134a..80a2063a7c 100644 --- a/hermes_cli/web_routers/local_models.py +++ b/hermes_cli/web_routers/local_models.py @@ -38,7 +38,7 @@ from hermes_cli.local_runtime import ( from agent.memory_provider import spawn_context_thread from pm.downloader import Download, DownloadPaused, Source -from hermes_cli.local_runtime.endpoint import _state_endpoint +from hermes_cli.local_runtime.endpoint import _state_endpoint, child_query from hermes_cli.local_runtime.gguf import split_parts logger = logging.getLogger(__name__) @@ -461,7 +461,7 @@ def _loaded_models(running: dict[str, Any]) -> tuple[dict[str, str], dict[str, A if plan is not None: facts.update(window=plan.window, window_label=_k_label(plan.window), spilled=plan.spilled) n_ctx = state in ("loaded", "ready") and _quiet( - lambda: _router_request(running, f"/props?model={model_id}", timeout=3) + lambda: _router_request(running, f"/props?{child_query(model_id)}", timeout=3) .get("default_generation_settings", {}).get("n_ctx"), None) if n_ctx: facts.update(granted_window=int(n_ctx), granted_window_label=_k_label(int(n_ctx))) diff --git a/locales/_keys.desktop.json b/locales/_keys.desktop.json index 4c4047e49e..18dadd7faa 100644 --- a/locales/_keys.desktop.json +++ b/locales/_keys.desktop.json @@ -1696,6 +1696,9 @@ "interfaceMode.simple.label", "interfaceMode.title", "intro.custom", + "keepAwakeOffer.action", + "keepAwakeOffer.message", + "keepAwakeOffer.title", "keybinds.actions.appearance.toggleMode", "keybinds.actions.composer.cancel", "keybinds.actions.composer.dictate", diff --git a/tests/agent/test_compression_prompt_reading.py b/tests/agent/test_compression_prompt_reading.py new file mode 100644 index 0000000000..2c82d757e4 --- /dev/null +++ b/tests/agent/test_compression_prompt_reading.py @@ -0,0 +1,154 @@ +"""Compression leaves llama-server's prompt reading out of its total time limit. + +The limit (600 s by default) stops a summary that trickles out a token at a time. On a laptop +GPU, reading a 158K-token conversation took a press reviewer's Spark about nine minutes before the +first summary token, so every compaction of a long session hit the limit and failed. llama-server +reports its reading as progress chunks; the reading they report pushes the deadline back, and the +limit applies to generation only. A server that reports without reading gets no extra time, and a +fence the host has closed stays closed. +""" + +import concurrent.futures +import threading +import time +from types import SimpleNamespace +from unittest.mock import patch + +import pytest + +from agent import auxiliary_client, conversation_compression, local_prompt_progress +from agent.auxiliary_client import _ChatStreamAccumulator, _current_aux_stream_deadline +from agent.conversation_compression import CompressionCommitFence, _await_worker_within_budget +from agent.conversation_compression_deadline import compression_deadline_scope, effective_ceiling +from agent.local_prompt_progress import PromptReadingClock, prompt_reading_hook + + +@pytest.fixture +def clock(monkeypatch): + """One fake monotonic clock for the stream, the reading clock and the fence.""" + now = [1000.0] + fake = SimpleNamespace(monotonic=lambda: now[0], time=time.time) + for module in (auxiliary_client, local_prompt_progress, conversation_compression): + monkeypatch.setattr(module, "time", fake) + return now + + +def _progress(processed, time_ms=None): + payload = {"total": 150_000, "cache": 0, "processed": processed} + if time_ms is not None: + payload["time_ms"] = time_ms + return SimpleNamespace(model_extra={"prompt_progress": payload}, + choices=[SimpleNamespace(delta=SimpleNamespace(content=None), finish_reason=None)], + usage=None) + + +def _token(text): + return SimpleNamespace(model_extra={}, choices=[SimpleNamespace( + delta=SimpleNamespace(content=text, reasoning=None, reasoning_content=None, tool_calls=None), + finish_reason=None)], usage=None) + + +def test_reading_time_pushes_back_the_shared_deadline(clock): + fence = CompressionCommitFence(total_ceiling_seconds=60) + with compression_deadline_scope(fence): + reading = PromptReadingClock(clock[0]) + assert reading.tick({"processed": 0}) is True + clock[0] += 45 + assert reading.tick({"processed": 40_000}) is True + assert fence.deadline_monotonic == pytest.approx(1105.0) + assert _current_aux_stream_deadline() == pytest.approx(1105.0) + assert reading.seconds == pytest.approx(45.0) + + +def test_server_reading_time_bounds_each_stretch(clock): + reading = PromptReadingClock(clock[0]) + clock[0] += 30 # queued behind another request, then the model loads + reading.tick({"processed": 0, "time_ms": 0}) + clock[0] += 50 + reading.tick({"processed": 20_000, "time_ms": 40_000}) + assert reading.seconds == pytest.approx(40.0) + + +def test_progress_without_new_tokens_gets_no_time(clock): + fence = CompressionCommitFence(total_ceiling_seconds=60) + with compression_deadline_scope(fence): + acc = _ChatStreamAccumulator(total_ceiling=60, host_deadline=_current_aux_stream_deadline) + with patch.object(auxiliary_client, "_notify_aux_progress") as alive: + clock[0] += 10 + acc.feed(_progress(5_000)) + for _ in range(2): # a server stuck at the same count + clock[0] += 20 + acc.feed(_progress(5_000)) + clock[0] += 20 + with pytest.raises(TimeoutError, match="timed out"): # the limit still runs + acc.feed(_progress(5_000)) + assert alive.call_count == 1 + assert fence.deadline_monotonic == pytest.approx(1070.0) # only the first 10 s of reading + + +def test_a_closed_fence_stays_closed(clock): + cancelled = CompressionCommitFence(total_ceiling_seconds=60) + assert cancelled.try_cancel_before_commit() + assert cancelled.extend_deadline(30) is False + assert cancelled.is_cancelled + + expired = CompressionCommitFence(total_ceiling_seconds=60) + clock[0] += 61 + assert expired.is_cancelled + assert expired.extend_deadline(120) is False # a chunk landing after the deadline + assert expired.is_cancelled + + committing = CompressionCommitFence(total_ceiling_seconds=60) + assert committing.begin_commit() + assert committing.extend_deadline(30) is False # the commit owns the fence; nothing to extend + committing.finish_commit() + assert committing.deadline_monotonic == pytest.approx(clock[0] + 60) + + +def test_stream_outlives_a_ceiling_shorter_than_its_prompt_reading(clock): + fence = CompressionCommitFence(total_ceiling_seconds=60) + with compression_deadline_scope(fence): + acc = _ChatStreamAccumulator(total_ceiling=60, host_deadline=_current_aux_stream_deadline) + for processed in (0, 50_000, 100_000, 150_000): # 50 s between batches, 200 s of reading + clock[0] += 50 + acc.feed(_progress(processed)) + clock[0] += 20 + assert acc.feed(_token("summary")) is True + assert not fence.deadline_exceeded + clock[0] += 45 # 65 s of generation: over the ceiling + with pytest.raises(TimeoutError, match="timed out"): + acc.feed(_token(" more")) + + +def test_reading_hook_is_scoped_to_its_block(clock): + heard = [] + with prompt_reading_hook(heard.append): + reading = PromptReadingClock(clock[0]) + reading.tick({"processed": 0}) + clock[0] += 3 + reading.tick({"processed": 100}) + outside = PromptReadingClock(clock[0]) + outside.tick({"processed": 0}) + clock[0] += 3 + outside.tick({"processed": 100}) + assert heard == [0.0, pytest.approx(3.0)] + + +def test_hosts_apply_the_moved_deadline(clock): + fence = CompressionCommitFence(total_ceiling_seconds=60) + wait_started = clock[0] + assert effective_ceiling(fence, 60, wait_started) == pytest.approx(60) + fence.extend_deadline(90) + assert effective_ceiling(fence, 60, wait_started) == pytest.approx(150) + + +def test_host_keeps_waiting_while_the_deadline_moves(): + fence = CompressionCommitFence(total_ceiling_seconds=2.0) + wait_started = time.monotonic() + future = concurrent.futures.Future() + # Reading already pushed the deadline to 6 s; the summary lands at 3 s, past the 2 s ceiling. + fence.extend_deadline(4.0) + fence.touch_progress() + threading.Timer(3.0, lambda: future.set_result(("summary", ""))).start() + settled, result = _await_worker_within_budget(future, fence, idle=10.0, ceiling=2.0, wait_started=wait_started) + assert settled and result == ("summary", "") diff --git a/tests/agent/test_local_prompt_progress.py b/tests/agent/test_local_prompt_progress.py new file mode 100644 index 0000000000..cbe6705fb7 --- /dev/null +++ b/tests/agent/test_local_prompt_progress.py @@ -0,0 +1,269 @@ +"""llama-server prompt progress, sleep-aware stall timers, and the local stale-breaker reset. + +A laptop running a local model reads long prompts for minutes and sometimes sleeps mid-request. +These tests pin the behaviour that keeps such requests alive: progress chunks count as signs of +life, local requests never fall back to a non-streamed (uncancellable) request, time asleep +doesn't count as a stall, and a recovered local server gets the next message. +""" + +import time +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +import pytest + +from agent.local_prompt_progress import prompt_progress, prompt_progress_percent, with_prompt_progress + + +def _delta(**fields): + base = dict(role="assistant", content=None, tool_calls=None, reasoning_content=None, reasoning=None, refusal=None) + base.update(fields) + return SimpleNamespace(**base) + + +def _progress_chunk(processed, total=1000, cache=0): + choice = SimpleNamespace(index=0, delta=_delta(), finish_reason=None) + return SimpleNamespace(choices=[choice], model=None, usage=None, id="c1", + model_extra={"prompt_progress": {"total": total, "cache": cache, + "processed": processed, "time_ms": 1}}) + + +def _text_chunk(content, finish_reason=None): + choice = SimpleNamespace(index=0, delta=_delta(content=content), finish_reason=finish_reason) + return SimpleNamespace(choices=[choice], model="qwen", usage=None, id="c1", model_extra={}) + + +class TestPromptProgressHelpers: + def test_progress_chunk_is_recognised(self): + assert prompt_progress(_progress_chunk(250)) == {"total": 1000, "cache": 0, "processed": 250, "time_ms": 1} + + def test_wire_dict_chunk_is_recognised(self): + chunk = {"choices": [{"delta": {"role": "assistant", "content": None}, "finish_reason": None}], + "prompt_progress": {"total": 10, "processed": 5}} + assert prompt_progress(chunk) == {"total": 10, "processed": 5} + + def test_chunk_with_text_is_a_real_chunk(self): + chunk = _text_chunk("hi") + chunk.model_extra = {"prompt_progress": {"total": 10, "processed": 10}} + assert prompt_progress(chunk) is None + + def test_chunk_with_finish_reason_is_a_real_chunk(self): + chunk = _progress_chunk(10, total=10) + chunk.choices[0].finish_reason = "stop" + assert prompt_progress(chunk) is None + + def test_plain_chunk_has_no_progress(self): + assert prompt_progress(_text_chunk("hi")) is None + + def test_with_prompt_progress_merges_extra_body(self): + kwargs = {"model": "m", "extra_body": {"cache_prompt": True}} + out = with_prompt_progress(kwargs) + assert out["extra_body"] == {"cache_prompt": True, "return_progress": True} + assert kwargs["extra_body"] == {"cache_prompt": True} # caller's dict untouched + + def test_percent(self): + assert prompt_progress_percent({"total": 200, "processed": 50}) == 25 + assert prompt_progress_percent({"total": 0, "processed": 50}) is None + assert prompt_progress_percent({"total": "x"}) is None + + +def _local_agent(provider="llamacpp", base_url="http://127.0.0.1:18434/v1"): + from run_agent import AIAgent + + agent = AIAgent(api_key="k", base_url=base_url, provider=provider, model="qwen", + quiet_mode=True, skip_context_files=True, skip_memory=True) + agent.api_mode = "chat_completions" + agent._interrupt_requested = False + return agent + + +class TestMainStreamProgress: + @patch("run_agent.AIAgent._create_request_openai_client") + @patch("run_agent.AIAgent._close_request_openai_client") + def test_llamacpp_stream_asks_for_progress_and_skips_progress_chunks(self, _close, mock_create): + chunks = [_progress_chunk(0), _progress_chunk(500), _progress_chunk(1000), + _text_chunk("Hello"), _text_chunk(" there", finish_reason="stop")] + client = MagicMock() + client.chat.completions.create.return_value = iter(chunks) + mock_create.return_value = client + agent = _local_agent() + deltas = [] + agent.stream_delta_callback = deltas.append + + response = agent._interruptible_streaming_api_call({"model": "qwen", "messages": [{"role": "user", "content": "hi"}]}) + + sent = client.chat.completions.create.call_args.kwargs + assert sent["extra_body"]["return_progress"] is True + assert response.choices[0].message.content == "Hello there" + assert "".join(deltas) == "Hello there" + + @patch("run_agent.AIAgent._create_request_openai_client") + @patch("run_agent.AIAgent._close_request_openai_client") + def test_cloud_stream_does_not_ask_for_progress(self, _close, mock_create): + client = MagicMock() + client.chat.completions.create.return_value = iter([_text_chunk("ok", finish_reason="stop")]) + mock_create.return_value = client + agent = _local_agent(provider="openrouter", base_url="https://openrouter.ai/api/v1") + + agent._interruptible_streaming_api_call({"model": "qwen", "messages": [{"role": "user", "content": "hi"}]}) + + sent = client.chat.completions.create.call_args.kwargs + assert "return_progress" not in (sent.get("extra_body") or {}) + + +class TestNoNonStreamingFallbackForLlamacpp: + def _call(self, agent): + from agent.chat_completion_helpers import _StreamingCall + + return SimpleNamespace(agent=agent, deltas_were_sent={"yes": False}, api_kwargs={"model": "qwen"}, + _stream_stale_timeout=900.0, result={"error": None, "response": None}) + + def test_server_error_probe_skipped(self): + from agent.chat_completion_helpers import _StreamingCall + + agent = SimpleNamespace(_interrupt_requested=False, provider="llamacpp", base_url="http://127.0.0.1:18434/v1", + api_mode="chat_completions", _stream_5xx_probe_ts=None) + err = RuntimeError("boom") + err.status_code = 500 + with patch("agent.chat_completion_helpers.interruptible_api_call") as probe: + assert _StreamingCall._unmask_server_error_with_nonstreaming(self._call(agent), err) is False + probe.assert_not_called() + + def test_streaming_never_disabled(self): + from agent.chat_completion_helpers import _StreamingCall + + agent = SimpleNamespace(provider="llamacpp", base_url="http://127.0.0.1:18434/v1", + api_mode="chat_completions", _disable_streaming=False) + with patch("agent.chat_completion_helpers._is_provider_stream_empty_frame_error", return_value=True): + _StreamingCall._maybe_disable_streaming(self._call(agent), RuntimeError("empty frame")) + assert agent._disable_streaming is False + + +class TestAuxProgress: + def test_managed_local_stream_asks_for_progress(self): + from agent import auxiliary_client as ac + + client = SimpleNamespace(base_url="http://127.0.0.1:18434/v1") + with patch.object(ac, "_is_managed_local_endpoint", return_value=True): + stream_kwargs, _model, _ceiling = ac._stream_request_plan({"model": "qwen", "messages": []}, client) + assert stream_kwargs["extra_body"]["return_progress"] is True + with patch.object(ac, "_is_managed_local_endpoint", return_value=False): + stream_kwargs, _model, _ceiling = ac._stream_request_plan({"model": "qwen", "messages": []}, client) + assert "extra_body" not in stream_kwargs + + def test_progress_chunk_counts_as_forward_progress(self): + from agent import auxiliary_client as ac + + acc = ac._ChatStreamAccumulator(model="qwen") + with patch.object(ac, "_notify_aux_progress") as tick: + # Keeps the compression fence alive without starting the watchdog's inter-token window. + assert acc.feed(_progress_chunk(100)) is False + tick.assert_called_once() + acc.feed(_text_chunk("summary", finish_reason="stop")) + assert acc.finish().choices[0].message.content == "summary" + + +class TestSleepClock: + def test_sleep_watch_reports_only_real_sleeps(self): + from agent import sleep_clock + + wall = [1000.0] + with patch.object(sleep_clock.time, "time", lambda: wall[0]): + watch = sleep_clock.SleepWatch(interval=0.3) + wall[0] += 0.3 + assert watch.slept() == 0.0 + wall[0] += 1.3 # clock sync nudge + assert watch.slept() == 0.0 + wall[0] += 1200.3 # 20 minutes asleep, or frozen by Modern Standby + assert watch.slept() == pytest.approx(1200.0) + wall[0] += 0.3 + assert watch.slept() == 0.0 + + def test_awake_seconds_moves_forward(self): + from agent.sleep_clock import awake_seconds + + first = awake_seconds() + assert awake_seconds() >= first + + def test_monitor_moves_stall_clock_past_sleep(self): + from agent.chat_completion_stream_monitor import StreamingWaitMonitor + + mon = StreamingWaitMonitor() + mon.agent = MagicMock() + mon.last_chunk_time = {"t": 100.0} + mon._mon = SimpleNamespace(last_heartbeat=100.0) + mon._skip_sleep(900.0) + assert mon.last_chunk_time["t"] == 1000.0 + assert mon._mon.last_heartbeat == 1000.0 + mon._skip_sleep(0.0) + assert mon.last_chunk_time["t"] == 1000.0 + # A chunk that landed during the reading is not pushed into the future. + now = time.time() + mon.last_chunk_time["t"] = now - 1.0 + mon._skip_sleep(900.0) + assert mon.last_chunk_time["t"] <= time.time() + + +class TestPromptProgressNotice: + def _monitor(self, first_chunk_at=None): + from agent import chat_completion_wait_notice as wn + from agent.chat_completion_stream_monitor import StreamingWaitMonitor + + mon = StreamingWaitMonitor() + mon.agent = MagicMock() + mon.clients = SimpleNamespace(diag={"first_chunk_at": first_chunk_at}) + mon.prompt_progress = {"progress": {"total": 1000, "processed": 400}, "t": 50.0} + mon._mon = SimpleNamespace(last_load_poll=0.0, load_notice_shown=False, load_notice_misses=0, + wait_notice_started_ts=None, wait_notice=wn.WaitNoticeState(), last_heartbeat=0.0) + return mon + + def test_shows_percent_before_first_token(self): + mon = self._monitor() + assert mon._show_prompt_progress(51.0) is True + mon.agent._emit_wait_notice.assert_called_once_with("⚙ processing prompt — 40%") + assert mon._show_prompt_progress(51.5) is False # at most once a second + + def test_clears_line_at_first_token(self): + mon = self._monitor() + mon._show_prompt_progress(51.0) + mon.clients.diag["first_chunk_at"] = 52.0 + assert mon._show_prompt_progress(53.0) is False + mon.agent._emit_wait_notice.assert_called_with("") + + +class TestLocalStaleBreakerReset: + def test_healthy_local_server_gets_one_attempt(self): + from agent import chat_completion_helpers as helpers + from agent import chat_completion_local as h + + agent = SimpleNamespace(_consecutive_stale_streams=5) + with patch.object(h, "_streams_llamacpp", return_value=True), \ + patch.object(h, "_local_server_answers", return_value=True): + assert h.retry_recovered_local_server(agent) is True + helpers._check_stale_giveup(agent) # the next call goes through + helpers._bump_stale_streak(agent) # ...and one more stale attempt shuts the breaker + with pytest.raises(RuntimeError, match="unresponsive"): + helpers._check_stale_giveup(agent) + + def test_closed_breaker_skips_the_health_check(self): + from agent import chat_completion_local as h + + agent = SimpleNamespace(_consecutive_stale_streams=2) + with patch.object(h, "_streams_llamacpp", return_value=True), \ + patch.object(h, "_local_server_answers", return_value=True) as health: + assert h.retry_recovered_local_server(agent) is False + health.assert_not_called() + assert agent._consecutive_stale_streams == 2 + + def test_unhealthy_or_cloud_keeps_streak(self): + from agent import chat_completion_local as h + + agent = SimpleNamespace(_consecutive_stale_streams=5) + with patch.object(h, "_streams_llamacpp", return_value=True), \ + patch.object(h, "_local_server_answers", return_value=False): + assert h.retry_recovered_local_server(agent) is False + with patch.object(h, "_streams_llamacpp", return_value=False), \ + patch.object(h, "_local_server_answers", return_value=True) as health: + assert h.retry_recovered_local_server(agent) is False + health.assert_not_called() + assert agent._consecutive_stale_streams == 5 diff --git a/tests/agent/test_review_idle_queue.py b/tests/agent/test_review_idle_queue.py index ed32b9279a..ef475f4c07 100644 --- a/tests/agent/test_review_idle_queue.py +++ b/tests/agent/test_review_idle_queue.py @@ -253,7 +253,7 @@ def test_wrapper_cloud_fast_path_skips_runtime_resolution(monkeypatch): raise AssertionError("runtime resolution must not run") monkeypatch.setattr( - "agent.auxiliary_client._managed_local_netloc", lambda: "") + "agent.auxiliary_managed_local._managed_local_netloc", lambda: "") monkeypatch.setattr( "agent.background_review._resolve_review_runtime", _explode) assert riq.review_targets_managed_local(object(), {}) is False diff --git a/tests/agent/test_stream_wait_notice.py b/tests/agent/test_stream_wait_notice.py index 7d73936432..73622eef24 100644 --- a/tests/agent/test_stream_wait_notice.py +++ b/tests/agent/test_stream_wait_notice.py @@ -4,6 +4,7 @@ from types import SimpleNamespace import pytest from agent import chat_completion_helpers as h +from agent import chat_completion_local @pytest.mark.parametrize("local_loading,heartbeat_race", [(False, False), (True, False), (False, True)]) @@ -21,7 +22,7 @@ def test_resumed_chunks_clear_wait_without_erasing_local_load(monkeypatch, local call.last_chunk_time = {"t": now[0]} call._stream_stale_timeout = 180.0 loading = "Loading local model weights" - monkeypatch.setattr(h, "_managed_local_load_notice", + monkeypatch.setattr(chat_completion_local, "_managed_local_load_notice", lambda *args: loading if now[0] >= 1060.6 else None) monkeypatch.setattr(h.time, "time", lambda: now[0]) diff --git a/tests/gateway/test_hygiene_wait_moved_deadline.py b/tests/gateway/test_hygiene_wait_moved_deadline.py new file mode 100644 index 0000000000..b2f98c0ee0 --- /dev/null +++ b/tests/gateway/test_hygiene_wait_moved_deadline.py @@ -0,0 +1,46 @@ +"""The gateway's hygiene wait follows the compression fence when prompt reading moves its deadline. + +A local llama-server reading a long conversation reports progress, and each report of newly read +tokens pushes the fence deadline back. The gateway waits in short slices and must judge each slice +against the deadline as it stands after the slice, not as it stood before: an extension that lands +in the last slice before the old deadline would otherwise cancel a healthy summary. +""" + +import asyncio +import time +from types import SimpleNamespace + +from gateway.run import GatewayRunner + + +class _ReadingFence: + """A fence whose deadline moves as the old one passes, the way a late progress chunk moves it.""" + + def __init__(self, ceiling: float): + self.wait_started = time.monotonic() + self.deadline_monotonic = self.wait_started + ceiling + self._moved = False + + @property + def is_cancelled(self) -> bool: + if not self._moved and time.monotonic() >= self.deadline_monotonic: + self.deadline_monotonic += 5.0 + self._moved = True + return time.monotonic() >= self.deadline_monotonic + + def seconds_since_progress(self) -> float: + return 0.0 + + +def test_hygiene_wait_reads_the_moved_deadline_after_each_slice(): + async def drive(): + fence = _ReadingFence(ceiling=0.6) + future = asyncio.get_running_loop().create_future() + asyncio.get_running_loop().call_later(1.2, future.set_result, ("compressed", None)) + attempt = GatewayRunner._HygieneAttempt( + agent=None, meta=None, commit_fence=fence, future=future, wait_started=fence.wait_started) + hs = SimpleNamespace(timeout_seconds=30.0, total_ceiling_seconds=0.6, max_turn_hold_seconds=30.0) + return await GatewayRunner._hmwa_hygiene_wait_for_summary( + None, attempt, hs, SimpleNamespace(session_id="sess-1")) + + assert asyncio.run(drive()) == "compressed" diff --git a/tests/hermes_cli/test_load_progress.py b/tests/hermes_cli/test_load_progress.py index c7362dd100..ffd6a16f45 100644 --- a/tests/hermes_cli/test_load_progress.py +++ b/tests/hermes_cli/test_load_progress.py @@ -96,7 +96,7 @@ def test_stale_entries_expire(): def test_load_notice_for_managed_model(tmp_path, monkeypatch): - from agent.chat_completion_helpers import _managed_local_load_notice + from agent.chat_completion_local import _managed_local_load_notice state = tmp_path / "server.json" state.write_text(json.dumps({"base_url": "http://127.0.0.1:18434/v1", @@ -128,7 +128,7 @@ def test_load_notice_for_managed_model(tmp_path, monkeypatch): def test_prefill_notice_for_managed_model(tmp_path, monkeypatch): - from agent.chat_completion_helpers import _managed_local_load_notice + from agent.chat_completion_local import _managed_local_load_notice state = tmp_path / "server.json" state.write_text(json.dumps({"base_url": "http://127.0.0.1:18434/v1", @@ -161,7 +161,7 @@ def test_load_notice_outranks_prefill(tmp_path, monkeypatch): """While a load entry exists the load notice wins — prefill can't start before the model is resident, so a simultaneous claim means the load snapshot is authoritative.""" - from agent.chat_completion_helpers import _managed_local_load_notice + from agent.chat_completion_local import _managed_local_load_notice state = tmp_path / "server.json" state.write_text(json.dumps({"base_url": "http://127.0.0.1:18434/v1", diff --git a/tests/hermes_cli/test_local_abandoned_requests.py b/tests/hermes_cli/test_local_abandoned_requests.py index 02f194074a..3cb41042b1 100644 --- a/tests/hermes_cli/test_local_abandoned_requests.py +++ b/tests/hermes_cli/test_local_abandoned_requests.py @@ -25,6 +25,7 @@ import types import pytest import agent.auxiliary_client as aux +import agent.auxiliary_managed_local as managed_local from hermes_cli.local_runtime.supervisor import LlamaServerSupervisor @@ -39,7 +40,7 @@ def managed_state(tmp_path, monkeypatch): "pid": 4242}), encoding="utf-8") monkeypatch.setattr("hermes_cli.local_runtime.supervisor.state_path", lambda: state) - monkeypatch.setattr(aux, "_managed_local_cache", (0.0, "")) + monkeypatch.setattr(managed_local, "_managed_local_cache", (0.0, "")) return state @@ -58,7 +59,7 @@ def test_managed_detection_matches_netloc_not_substring(managed_state): def test_no_state_file_means_no_managed_endpoint(tmp_path, monkeypatch): monkeypatch.setattr("hermes_cli.local_runtime.supervisor.state_path", lambda: tmp_path / "absent.json") - monkeypatch.setattr(aux, "_managed_local_cache", (0.0, "")) + monkeypatch.setattr(managed_local, "_managed_local_cache", (0.0, "")) assert aux._is_managed_local_endpoint(MANAGED_URL) is False @@ -84,7 +85,7 @@ def test_no_default_cap_policy_unchanged_for_remote(monkeypatch): # A generic remote provider still drops the cap (the forwarding gate # is an allow-list). openrouter no longer qualifies as the example # here: main forwards its caps deliberately (#41035, 402 affordability). - monkeypatch.setattr(aux, "_managed_local_cache", (0.0, "")) + monkeypatch.setattr(managed_local, "_managed_local_cache", (0.0, "")) kwargs = aux._build_call_kwargs( "openai", "some/model", [{"role": "user", "content": "hi"}], max_tokens=64, timeout=30.0) diff --git a/tests/hermes_cli/test_local_launch_budget.py b/tests/hermes_cli/test_local_launch_budget.py index 037cec796c..355d2efbbe 100644 --- a/tests/hermes_cli/test_local_launch_budget.py +++ b/tests/hermes_cli/test_local_launch_budget.py @@ -99,6 +99,7 @@ def test_launch_budget_counts_own_server_as_free(monkeypatch): def test_launch_budget_skips_unified_memory_and_failed_probes(monkeypatch): uma = replace(CARD_CAPACITY, uma=True) monkeypatch.setattr(hardware, "_nvidia_vram", lambda: pytest.fail("UMA must not probe")) + monkeypatch.setattr(hardware, "_windows_memory_status", lambda: None) # Windows narrows by commit assert hardware.launch_budget(uma) is None monkeypatch.setattr(hardware, "_nvidia_vram", lambda: None) assert hardware.launch_budget(CARD_CAPACITY) is None @@ -224,6 +225,7 @@ class _Router: self.preset_path = preset_path self.statuses = statuses self.reloads = 0 + self.models_max = 1 self.proc = SimpleNamespace(poll=lambda: None) self._refit_usable = None import threading diff --git a/tests/hermes_cli/test_local_models_routes.py b/tests/hermes_cli/test_local_models_routes.py index 1ea400e57f..78d6ceefe5 100644 --- a/tests/hermes_cli/test_local_models_routes.py +++ b/tests/hermes_cli/test_local_models_routes.py @@ -124,7 +124,7 @@ def test_status_tracks_preset_spill_and_restored_window(client, tmp_path, monkey def router_response(running, route, **kwargs): if route == "/models": return {"data": [{"id": model_id, "status": {"value": "loaded"}}]} - assert route == f"/props?model={model_id}" + assert route == f"/props?model={model_id}&autoload=false" return {"default_generation_settings": {"n_ctx": server_window}} monkeypatch.setattr(local_models, "_router_request", router_response) diff --git a/tests/hermes_cli/test_local_residency_admission.py b/tests/hermes_cli/test_local_residency_admission.py index dcf5ffc68c..b98208071e 100644 --- a/tests/hermes_cli/test_local_residency_admission.py +++ b/tests/hermes_cli/test_local_residency_admission.py @@ -83,7 +83,7 @@ def test_boot_hands_the_router_the_derived_cap(tmp_path, monkeypatch, configured monkeypatch.setattr(bs, "models_dir", lambda: mdir) monkeypatch.setattr(hardware, "probe_budget", probe) monkeypatch.setattr(bs, "_SUPERVISOR", None) - monkeypatch.setattr(bs, "_generate_presets", lambda mdir, preset_path: None) + monkeypatch.setattr(bs, "_generate_presets", lambda mdir, preset_path, residents=1: None) monkeypatch.setattr(bs, "_detect_gpu_vendor", lambda: None) monkeypatch.setattr(binaries, "installed_engine", lambda backend="auto", **k: binaries.Engine("cpu", "build-1", tmp_path / "install" / "llama-server")) diff --git a/tests/hermes_cli/test_local_router_autoload.py b/tests/hermes_cli/test_local_router_autoload.py new file mode 100644 index 0000000000..aba796852c --- /dev/null +++ b/tests/hermes_cli/test_local_router_autoload.py @@ -0,0 +1,80 @@ +"""Status reads never load a model. + +llama-server's router (b11370, ``--models-autoload``) loads an unloaded model for any request that +names it, /props, /slots and /metrics included. Clicking Eject on the local models page unloaded the +model, and a capability read 76 ms later loaded it straight back. A per-request ``autoload=false`` +makes the router answer 400 "model is not loaded" instead (checked against a router serving a fake +model: zero spawns with it, a spawn per request without it). +""" + +import io +import urllib.error +from types import SimpleNamespace + +import pytest + +from hermes_cli.local_runtime import capabilities, endpoint, load_progress +from hermes_cli.local_runtime.supervisor import LlamaServerSupervisor + + +def test_child_query_opts_out_of_autoload(): + assert endpoint.child_query("org/model q4") == "model=org/model%20q4&autoload=false" + + +def test_capability_read_does_not_load_the_model(monkeypatch): + seen = [] + monkeypatch.setattr(endpoint, "managed_root", lambda: ("http://127.0.0.1:1", "k")) + monkeypatch.setattr(endpoint, "managed_get_json", lambda base, key, path, timeout_s: seen.append(path) or {}) + capabilities._props_modalities("qwen") + assert seen == ["/props?model=qwen&autoload=false"] + + +def test_prefill_read_does_not_load_the_model(monkeypatch): + seen = [] + monkeypatch.setattr(load_progress, "_endpoint", lambda: ("http://127.0.0.1:1", "k")) + monkeypatch.setattr(endpoint, "managed_get_json", lambda base, key, path, timeout_s: seen.append(path) or []) + assert load_progress.get_prefill_progress("qwen") is None + assert seen == ["/slots?model=qwen&autoload=false"] + + +class _Client: + """httpx-shaped client over a router whose model is unloaded.""" + + def __init__(self): + self.paths = [] + + def get(self, url): + path = url.split("127.0.0.1:1", 1)[1] + self.paths.append(path) + if path == "/models": + return SimpleNamespace(status_code=200, json=lambda: {"data": [{"id": "qwen", "status": { + "value": "unloaded", "args": ["llama-server.exe", "--ctx-size", "65536", "--jinja"]}}]}) + return SimpleNamespace(status_code=400, json=lambda: {"error": {"message": "model is not loaded"}}) + + +def test_context_probe_reads_launch_args_for_an_unloaded_model(): + from agent.model_metadata_llamacpp import llamacpp_context + + client = _Client() + assert llamacpp_context(client, "http://127.0.0.1:1", "qwen") == 65536 + assert "/props?model=qwen&autoload=false" in client.paths + assert "/props?model=qwen" not in client.paths + + +@pytest.mark.parametrize("body", [["not", "an", "object"], {"data": "oops"}, {"data": [{"id": "qwen", "status": "loading"}]}]) +def test_context_probe_survives_an_odd_models_body(body): + from agent.model_metadata_llamacpp import router_launch_ctx + + client = SimpleNamespace(get=lambda url: SimpleNamespace(status_code=200, json=lambda: body)) + assert router_launch_ctx(client, "http://127.0.0.1:1", "qwen") is None + + +def test_an_unloaded_model_counts_as_idle(monkeypatch): + """Growth and the idle sweeper ask whether a model is busy; one that isn't loaded is not.""" + sup = LlamaServerSupervisor.__new__(LlamaServerSupervisor) + + def request(route, body=None, timeout_s=30): + raise urllib.error.HTTPError(route, 400, "model is not loaded", {}, io.BytesIO(b"")) + + monkeypatch.setattr(sup, "_request", request, raising=False) + assert sup._probe_idle("qwen") is True diff --git a/tests/hermes_cli/test_local_runtime.py b/tests/hermes_cli/test_local_runtime.py index 4f12e1d6d5..11094e12e5 100644 --- a/tests/hermes_cli/test_local_runtime.py +++ b/tests/hermes_cli/test_local_runtime.py @@ -1005,7 +1005,7 @@ def test_boot_prices_the_engine_it_serves(tmp_path, monkeypatch): lambda: {"local_runtime": {"backend": "cpu"}}) # another profile's config priced: list = [] monkeypatch.setattr(bootstrap, "_generate_presets", - lambda mdir, path: priced.append(hardware._configured_engine()) or path) + lambda mdir, path, residents=1: priced.append(hardware._configured_engine()) or path) monkeypatch.setattr(bootstrap, "_admitted_models_max", lambda mdir, configured: configured) class _NoStart: diff --git a/tests/hermes_cli/test_local_windows_commit_budget.py b/tests/hermes_cli/test_local_windows_commit_budget.py new file mode 100644 index 0000000000..feab6721b9 --- /dev/null +++ b/tests/hermes_cli/test_local_windows_commit_budget.py @@ -0,0 +1,146 @@ +"""Windows unified memory: the model and its prompt cache must fit in commit. + +Windows charges every GPU allocation to commit (RAM Windows can see plus the page file). On an +RTX Spark the GPU carve-out is invisible to Windows, so the allocator pool overstates what Windows +will hand out. Measured on a 64 GB unit at the shipped 32/32 split: 61.2 GiB commit limit, 48.4 GiB +free at idle; the 27B at a 256K window took 37.5 GiB of commit, and llama-server's default 8 GiB +prompt cache brought free commit down to ~1.4 GiB with a busy desktop. + +Free commit is a live reading, so it only narrows the launch window and sizes the cache. Capacity, +the physics check and the catalog's recommendation stay on the pool: a busy desktop at boot must +not refuse a model that fits. +""" + +from types import SimpleNamespace + +import pytest + +from hermes_cli.local_runtime import hardware as hw +from hermes_cli.local_runtime import presets +from hermes_cli.local_runtime.estimator import HardwareBudget, ModelProfile + +G = 1 << 30 + + +def _stat(ram_visible, commit_limit, commit_free): + return SimpleNamespace(ullTotalPhys=int(ram_visible * G), ullAvailPhys=int(ram_visible * G / 2), + ullTotalPageFile=int(commit_limit * G), ullAvailPageFile=int(commit_free * G)) + + +@pytest.fixture +def spark(monkeypatch): + """An RTX Spark at 32/32: Windows sees 30.2 GiB, the CUDA pool is 50.6 GiB.""" + state = {"stat": _stat(30.2, 61.2, 48.4)} + monkeypatch.setattr(hw, "_windows_memory_status", lambda: state["stat"]) + monkeypatch.setattr(hw, "_unified_pool_bytes", lambda smi_total, ram_total: int(50.6 * G)) + monkeypatch.setattr(hw, "_nvidia_vram", lambda: (32 * G, 31 * G, "NVIDIA RTX Spark N1X", None)) + monkeypatch.setattr(hw, "_ram_bytes", lambda: (int(30.2 * G), int(15 * G))) + return state + + +def test_commit_room_leaves_a_desktop_margin(spark): + assert hw._windows_commit_room() == pytest.approx((48.4 - 2.718) * G, rel=1e-3) # 9% of visible RAM + + +def test_the_managed_server_counts_as_free_before_its_relaunch(spark): + before = hw._windows_commit_room() + spark["stat"] = _stat(30.2, 61.2, 48.4 - 37.5) + assert hw._windows_commit_room(own_bytes=int(37.5 * G)) == pytest.approx(before, rel=1e-6) + + +def test_off_windows_has_no_commit_room(monkeypatch): + monkeypatch.setattr(hw, "_windows_memory_status", lambda: None) + assert hw._windows_commit_room() is None + capacity = HardwareBudget(int(16 * G), int(50 * G), 0, uma=True) + assert hw.launch_budget(capacity) is None + + +def test_capacity_stays_on_the_pool_however_busy_the_desktop(spark): + idle = hw.probe_budget(planning=True) + spark["stat"] = _stat(30.2, 61.2, 12.0) + busy = hw.probe_budget(planning=True) + assert busy.usable_vram_bytes == idle.usable_vram_bytes + assert busy.commit_room_bytes is None + + +def test_launch_budget_narrows_to_free_commit(spark): + capacity = hw.probe_budget(planning=True) + spark["stat"] = _stat(22.2, 44.4, 31.6) # 40/24 split: 8 GiB less RAM, a smaller page file + live = hw.launch_budget(capacity) + assert live.commit_room_bytes == hw._windows_commit_room() + assert live.usable_vram_bytes == int(live.commit_room_bytes * 0.8) < capacity.usable_vram_bytes + + +def test_pool_binds_when_commit_is_plentiful(spark): + spark["stat"] = _stat(94, 120, 106) # 128 GB unit at 32/96 + capacity = hw.probe_budget(planning=True) + live = hw.launch_budget(capacity) + assert live.usable_vram_bytes == capacity.usable_vram_bytes == int(50.6 * G * 0.8) + + +def test_prompt_cache_takes_what_the_model_leaves(): + live = HardwareBudget(0, 0, 0, uma=True, commit_room_bytes=int(45.7 * G)) + # 20 GiB estimated -> 25 GiB charged: 20.7 GiB left, capped at llama-server's default. + assert hw.prompt_cache_bytes(live, 20 * G) == 8 * G + # 31 GiB estimated -> ~38.75 GiB charged: ~6.9 GiB left for the cache. + assert hw.prompt_cache_bytes(live, 31 * G) == pytest.approx(45.7 * G - 31 * G / 0.8, rel=1e-6) + assert hw.prompt_cache_bytes(live, int(36.5 * G)) == 0 # under 512 MiB: off + assert hw.prompt_cache_bytes(None, 10 * G) is None # engine default without a live reading + + +def test_models_the_router_holds_together_share_the_cache_room(): + live = HardwareBudget(0, 0, 0, uma=True, commit_room_bytes=int(45.7 * G)) + # Two 14 GiB models: each charges 17.5 GiB, and 10.7 GiB remain for two caches. + each = hw.prompt_cache_bytes(live, 14 * G, residents=2) + assert each == pytest.approx((45.7 - 2 * 17.5) * G / 2, rel=1e-6) + assert 2 * (each + 17.5 * G) <= 45.7 * G + # A router that holds one model leaves that model the whole room. + assert hw.prompt_cache_bytes(live, 14 * G, residents=1) == 8 * G + + +def test_router_holds_only_the_models_free_commit_fits(tmp_path, monkeypatch): + """Capacity alone would admit two 12 GiB models; 20 GiB of commit holds one.""" + from hermes_cli.local_runtime import bootstrap + + monkeypatch.setattr(bootstrap, "staged_in", lambda mdir: [tmp_path / "a.gguf", tmp_path / "b.gguf"]) + monkeypatch.setattr(presets, "_launch_footprint", lambda gguf, budget: 12 * G) + capacity = HardwareBudget(int(30 * G), int(50 * G), 0, uma=True) + live = HardwareBudget(int(16 * G), int(50 * G), 0, uma=True, commit_room_bytes=int(20 * G)) + assert presets.admitted_residency_count(tmp_path, capacity, 4) == 2 + assert presets.admitted_residency_count(tmp_path, capacity, 4, live=live) == 1 + + +def test_growth_counts_the_outgoing_model_at_its_commit_charge(spark): + """The outgoing model frees what Windows charged for it, about 1/0.8 of its estimate.""" + capacity = hw.probe_budget(planning=True) + spark["stat"] = _stat(30.2, 61.2, 48.4 - 25.0) # a 20 GiB estimate, charged 25 GiB, is loaded + live = hw.launch_budget(capacity, own_bytes=20 * G) + assert live.commit_room_bytes == pytest.approx((48.4 - 2.718) * G, rel=1e-3) + + +def _staged(tmp_path, monkeypatch): + monkeypatch.setenv("HERMES_HOME", str(tmp_path)) + gguf = tmp_path / "model.gguf" + gguf.touch() + monkeypatch.setattr(presets, "read_gguf_header", lambda p: SimpleNamespace(path=p, sampling_defaults={})) + monkeypatch.setattr(presets, "profile_from_gguf", lambda h: ModelProfile( + name="model", weights_bytes=8 * G, embd_table_bytes=0, n_ctx_train=65536, layers=[])) + return gguf + + +def test_preset_writes_the_cache_size_from_live_commit(tmp_path, monkeypatch): + gguf = _staged(tmp_path, monkeypatch) + capacity = HardwareBudget(int(16 * G), int(50 * G), 0, uma=True) + live = HardwareBudget(int(16 * G), int(50 * G), 0, uma=True, commit_room_bytes=int(20 * G)) + entry = presets.preset_for_model(gguf, capacity, set(), live=live) + assert 0 < int(entry.keys["cache-ram"]) <= 8192 + assert "cache-ram" not in presets.preset_for_model(gguf, capacity, set()).keys + + +def test_a_busy_desktop_narrows_but_never_refuses(tmp_path, monkeypatch): + gguf = _staged(tmp_path, monkeypatch) + capacity = HardwareBudget(int(16 * G), int(50 * G), 0, uma=True) + live = HardwareBudget(int(1 * G), int(50 * G), 0, uma=True, commit_room_bytes=int(1.25 * G)) + entry = presets.preset_for_model(gguf, capacity, set(), live=live) + assert entry.refusal is None and entry.window > 0 + assert entry.keys["cache-ram"] == "0" diff --git a/tests/tools/test_smart_approval_reasoning.py b/tests/tools/test_smart_approval_reasoning.py new file mode 100644 index 0000000000..be78cc3bfb --- /dev/null +++ b/tests/tools/test_smart_approval_reasoning.py @@ -0,0 +1,45 @@ +"""The smart-approval guardian asks for its verdict with thinking off, unless the operator chose. + +Its budget is 16 tokens for one word. A reasoning model left to think spends the whole budget on +reasoning and returns nothing: a press reviewer's local Qwen 27B returned 45 empty verdicts out of +45. With thinking off, the managed llama-server answers ``APPROVE`` in 4 tokens. An operator who +set ``auxiliary.approval.reasoning_effort`` keeps that setting. +""" + +from unittest.mock import MagicMock, patch + +import pytest + +from agent import auxiliary_client +from tools.approval_smart import _smart_approve + +LOCAL = "http://127.0.0.1:18434/v1" + + +def _reply(text): + return MagicMock(choices=[MagicMock(message=MagicMock(content=text))]) + + +def _verdict_wire(task_config): + """The request the guardian's verdict call puts on the wire to a local llama-server.""" + with patch("agent.auxiliary_task_config._get_auxiliary_task_config", return_value=task_config), \ + patch.object(auxiliary_client, "_get_auxiliary_task_config", return_value=task_config), \ + patch("agent.auxiliary_client.call_llm", return_value=_reply("APPROVE")) as call_llm: + assert _smart_approve("python -c \"print('hi')\"", "script execution via -c flag") == "approve" + sent = call_llm.call_args.kwargs + return auxiliary_client._build_call_kwargs( + provider="custom", model="qwen", messages=sent["messages"], temperature=sent["temperature"], + max_tokens=sent["max_tokens"], extra_body=auxiliary_client._get_task_extra_body("approval"), + reasoning_config=sent["reasoning_config"], base_url=LOCAL, task="approval") + + +def test_verdict_goes_out_with_thinking_off(): + assert _verdict_wire({})["reasoning_effort"] == "none" + + +@pytest.mark.parametrize("task_config", [ + {"reasoning_effort": "high"}, + {"extra_body": {"reasoning": {"enabled": True, "effort": "high"}}}, +]) +def test_operator_reasoning_setting_wins(task_config): + assert _verdict_wire(task_config)["reasoning_effort"] == "high" diff --git a/tools/approval_smart.py b/tools/approval_smart.py index 61a27fb4a0..6370cd4f0a 100644 --- a/tools/approval_smart.py +++ b/tools/approval_smart.py @@ -71,6 +71,20 @@ def _get_smart_policy() -> str: return policy.strip() if isinstance(policy, str) else "" +def _verdict_reasoning() -> dict | None: + """Thinking off for the verdict, unless the operator set reasoning for the approval task + (``auxiliary.approval.reasoning_effort`` or ``extra_body.reasoning``). A one-word verdict needs + no thinking, and a reasoning model otherwise spends the whole 16-token budget thinking and + returns nothing (a local Qwen 27B: 45 of 45 verdicts came back empty).""" + from agent.auxiliary_task_config import _get_auxiliary_task_config + + config = _get_auxiliary_task_config("approval") + extra_body = config.get("extra_body") + if config.get("reasoning_effort") not in (None, "") or (isinstance(extra_body, dict) and "reasoning" in extra_body): + return None + return {"enabled": False} + + def _smart_approve(command: str, description: str) -> str: """Ask the auxiliary LLM; return 'approve', 'deny', or 'escalate' (uncertain/failed). @@ -109,6 +123,7 @@ def _smart_approve(command: str, description: str) -> str: response = call_llm( task="approval", temperature=0, max_tokens=16, timeout=smart_timeout, messages=[{"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}], + reasoning_config=_verdict_reasoning(), ) logger.debug("Smart approvals: LLM call completed in %.1fs", time.monotonic() - _smart_t0) answer = (response.choices[0].message.content or "").strip().upper() diff --git a/website/docs/user-guide/configuration.md b/website/docs/user-guide/configuration.md index e40a2188ee..8fff6a72cb 100644 --- a/website/docs/user-guide/configuration.md +++ b/website/docs/user-guide/configuration.md @@ -1028,7 +1028,7 @@ The same limit is also a **fail-closed bound on what the model is sent** wheneve `hygiene_timeout_seconds` is the gateway's **inactivity budget** for this pre-agent compression pass — not a total wall-clock cap. The compression summary call streams from the model, and each arriving token counts as forward progress: a slow reasoning model that is still generating keeps extending its own deadline, so slow-but-healthy summary models are never cut off mid-generation. Only when the summary model produces **no output** for this many seconds (backend down, hung connection, silent provider) does the gateway warn the user, continue the incoming message without compression, and record a temporary per-session failure cooldown instead of appearing stuck. -`hygiene_total_ceiling_seconds` (default `600`) bounds the total wait even while tokens are still moving, so a degenerate trickle stream can't hold a turn hostage indefinitely. It is clamped to at least `hygiene_timeout_seconds`. +`hygiene_total_ceiling_seconds` (default `600`) bounds the total wait even while tokens are still moving, so a degenerate trickle stream can't hold a turn hostage indefinitely. It is clamped to at least `hygiene_timeout_seconds`. Time a llama.cpp server spends reading the prompt does not count toward it (see `context_total_ceiling_seconds`). `hygiene_max_turn_hold_seconds` (default `10`) is the gateway's **turn-hold budget** — the maximum wall-clock the incoming message is held waiting on hygiene compression before the gateway stops waiting and proceeds on the uncompressed transcript. It exists because `hygiene_total_ceiling_seconds` alone can leave the wire silent for far longer than a chat transport's idle-timeout: a summary model that keeps streaming tokens keeps resetting the inactivity slice, so without a turn-hold budget the wait can stretch toward the ceiling while zero bytes reach the user — Telegram (and similar transports) then drop the connection and the turn appears frozen. Capping the turn's wait at this budget (well under the typical ~30s transport idle-timeout) guarantees the message is answered promptly. **The compression is not lost when the budget expires**: the worker keeps running detached and — when its commit is watermark-fenced (the normal case with a session DB) — it keeps its commit admission, so the finished summary is adopted at the next safe boundary and turns appended after the wait was abandoned survive verbatim as concurrent tail. This matters especially for **thinking/reasoning summary models** (DeepSeek, QwQ, etc.) whose reasoning phase alone can exceed the budget: their summaries land one turn late instead of never. If the commit cannot be safely fenced, the late result is discarded (`CompressionCommitFence`) and it cannot overwrite newer turns. Raise the budget if you'd rather have compression apply within the same turn and your transport tolerates the wait; lower it for snappier recovery on very slow backends. @@ -1038,7 +1038,7 @@ The value is the **first rung** of an escalating ladder, not a fixed interval: c `context_timeout_seconds` (default `120`) is the same **inactivity budget** for in-agent `compress_context` — the conversation loop, preflight compaction, and manual `/compress` — so a hung summary model cannot stall a session indefinitely. Streamed summary tokens extend the wait; only a silent worker is cut off. The budget is floored at the auxiliary compression request's own timeout (`auxiliary.compression.timeout`, minimum 300s), so the host never gives up on a silent summariser before the request itself would — a reasoning summariser thinking before its first token, or a route that cannot stream, gets the same budget the provider call has. On timeout Hermes retries the summary once against the first entry of `auxiliary.compression.fallback_chain` (using that entry's own `timeout` when it declares one) — a stalled route never raises, so the auxiliary client's own fallback handling cannot see it. If that attempt also fails, or no fallback chain is configured, what happens next depends on whether the request still fits the model's context window: a request that fits is sent uncompressed this turn (the summary-failure cooldown stops the retry from repeating every turn); a request above the window cannot be sent at all, so Hermes commits its deterministic fallback summary (old tool results pruned, a static handoff in place of the summarised middle) instead of ending the turn — ending the turn with the "compression timed out" recovery result (and, on the messaging gateway, the automatic session reset) is the last resort, reached only when even the deterministic pass cannot shrink the transcript. Set to `0` to disable. A preflight pass that reclaims nothing on a request still above the window ends the turn immediately with guidance to start a new session (`/new`) instead of sending a request the model cannot accept. Gateway session hygiene keeps its own `hygiene_timeout_seconds` path and is not double-wrapped. -`context_total_ceiling_seconds` (default `600`) bounds the in-agent **pre-commit** wait (summary / stream phase) even while tokens are still moving. It is clamped to at least `context_timeout_seconds`. For a request already above the model's context window the pre-commit wait is bounded by one `context_timeout_seconds` budget instead of this ceiling: such a request cannot be sent uncompressed anyway, and a summary that keeps streaming while reclaiming nothing would otherwise hold the session (and the Desktop UI) for the full ceiling on every turn — the deterministic fallback summary then carries the compaction. Raise `context_timeout_seconds` if your summariser legitimately needs longer. The exact guarantee: **the summary phase is bounded by this ceiling; the commit phase is logged and surfaced if it exceeds it.** Once the worker has entered the compression commit fence and SessionDB mutation is in flight, the commit is never abandoned mid-flight — that would risk transcript divergence — but the wait is no longer silent: if the commit runs past the ceiling, Hermes logs the overrun (WARNING, escalating to ERROR on repeat), sends a one-shot warning through the user-visible warning channel, and keeps waiting in bounded increments until the commit completes. When the ceiling expires during the summary phase, the summary model's stream is closed at that same instant on every auxiliary wire (chat.completions, Codex Responses, Anthropic Messages) — an abandoned summary is not billed to completion on a connection nobody is waiting for, and its session lease is freed for the next attempt. +`context_total_ceiling_seconds` (default `600`) bounds the in-agent **pre-commit** wait (summary / stream phase) even while tokens are still moving. It is clamped to at least `context_timeout_seconds`. Time a llama.cpp server spends reading the prompt does not count toward it: on a laptop GPU, reading a 150K-token conversation alone can take longer than the ceiling. The server reports its reading as progress, and each report of newly read tokens pushes the deadline back by the reading time; a server that reports without reading anything gets no extra time, and the idle timeout still applies between reports. For a request already above the model's context window the pre-commit wait is bounded by one `context_timeout_seconds` budget instead of this ceiling: such a request cannot be sent uncompressed anyway, and a summary that keeps streaming while reclaiming nothing would otherwise hold the session (and the Desktop UI) for the full ceiling on every turn — the deterministic fallback summary then carries the compaction. Raise `context_timeout_seconds` if your summariser legitimately needs longer. The exact guarantee: **the summary phase is bounded by this ceiling; the commit phase is logged and surfaced if it exceeds it.** Once the worker has entered the compression commit fence and SessionDB mutation is in flight, the commit is never abandoned mid-flight — that would risk transcript divergence — but the wait is no longer silent: if the commit runs past the ceiling, Hermes logs the overrun (WARNING, escalating to ERROR on repeat), sends a one-shot warning through the user-visible warning channel, and keeps waiting in bounded increments until the commit completes. When the ceiling expires during the summary phase, the summary model's stream is closed at that same instant on every auxiliary wire (chat.completions, Codex Responses, Anthropic Messages) — an abandoned summary is not billed to completion on a connection nobody is waiting for, and its session lease is freed for the next attempt. `protect_first_n` controls how many **non-system** head messages are pinned across every compaction. Default `3` — the opening user/assistant exchange survives every summarizer pass so the original goal stays visible. On long-running rolling-compaction sessions where the opening turn is no longer relevant, set `protect_first_n: 0` to pin nothing but the system prompt + summary + tail. The system prompt itself is always preserved regardless of this setting. diff --git a/website/docs/user-guide/desktop.md b/website/docs/user-guide/desktop.md index 962c5ae1c7..4c8ee2e0c5 100644 --- a/website/docs/user-guide/desktop.md +++ b/website/docs/user-guide/desktop.md @@ -255,7 +255,7 @@ Manage providers, models, tools, and credentials from a real UI instead of editi - **Auxiliary-model warning** — if you switch the main model to a new provider while auxiliary tasks (titling, summarization, and similar helpers) are still pinned to another provider, the app warns you so you don't unknowingly split work across two providers. - **Per-task reasoning effort** — each row under **Settings → Model → Auxiliary models** has a reasoning selector next to its provider/model pick: a level, **Off**, or **inherit · main model effort** (the default, which removes the task's override). It is saved as `auxiliary..reasoning_effort` in `config.yaml`, the same key `hermes model` writes, and shows in the row's summary when set. Use it to run frequent helpers such as compression or titling at low or no reasoning while the main agent stays at high. - **VS Code Marketplace themes** — beyond the built-in theme presets, the appearance settings include a live VS Code Marketplace search: pick any color theme and the app downloads, converts, and installs it as a desktop theme. The same importer is available from the command palette (*Install theme*), and imported themes can be removed again from the appearance settings. -- **Keep computer awake** — **Settings → Advanced → Keep computer awake** stops the machine from sleeping so long or overnight agent runs keep going (the display can still dim). **While working** holds the machine awake only while a turn is in flight and lets it sleep normally once the run finishes or fails; **Always** holds it whenever Hermes is open. This is a per-computer setting. +- **Keep computer awake** — **Settings → Advanced → Keep computer awake** stops the machine from sleeping so long or overnight agent runs keep going (the display can still dim). **While working** holds the machine awake only while a turn is in flight and lets it sleep normally once the run finishes or fails; **Always** holds it whenever Hermes is open. **While working** is the default on a computer where nobody has picked a mode. If you turned it **Off** and a timed sleep interrupts a turn, Hermes offers **Keep awake while working** once on wake. Closing the lid or choosing Sleep still sleeps the machine, and on battery Windows may still sleep after its own timeout. This is a per-computer setting. First-run onboarding has been redesigned on a unified overlay design system, and you can pick **Choose provider later** to skip provider setup and get into the app first.