fix(api): poll a paused canonical run as unknown, not terminal interrupted

Mechanism: run_projection mapped an admission in status `unknown` (claimed
before an authority-owner restart, outcome unknown) to `interrupted`, which is
in TERMINAL_STATUSES and documented as terminal. Stop refuses that run with
409 unknown_execution and followers stay queued until resolve-unknown, so a
poller stopped at a paused head and could not tell it needed resolution.

Fix: project `unknown` as `unknown` (non-terminal); the docs name the status
in the run-status paragraph and the resolve-unknown section.

Red on r12-base:
tests/gateway/test_api_unknown_resolution.py::test_http_poll_reports_unknown_head_as_non_terminal_until_resolved
polls GET /v1/runs/{id}, reads admission_id/execution_generation from that
body, resolves, checks the follower runs once, then polls terminal.

Patch by abundantbeing (issue comment 6103536385), applied as posted.

Co-authored-by: abundantbeing <222723615+abundantbeing@users.noreply.github.com>
Review: JoaoMarcos44 B31
Review: abundantbeing C-ab-5
Review: abundantbeing C-ab-6
Review: abundantbeing C-ab-7
(cherry picked from commit 9afa467d765ebc798b827b0f1b876c7e29e6deac)

gateway/platforms/api_server_authority_runs.py

@@ -33,7 +33,9 @@ def run_projection(adapter, run_id):
     if owned is None:
         return None
     authority, row = owned
-    status = {'queued': 'queued', 'started': 'running', 'unknown': 'interrupted', 'terminal': row['outcome']}.get(row['status'])
+    # ``unknown`` is not terminal: later turns wait for POST /v1/runs/{id}/resolve-unknown, and
+    # ``interrupted`` is in TERMINAL_STATUSES, so a poller would stop at a paused head.
+    status = {'queued': 'queued', 'started': 'running', 'unknown': 'unknown', 'terminal': row['outcome']}.get(row['status'])
     saved = admission_result(authority.db, row['admission_id'])
     result = saved.get('result', {}) if saved else {}
     if row['status'] == 'terminal':

tests/gateway/test_api_unknown_resolution.py

@@ -270,3 +270,43 @@ async def test_fresh_adapter_recovers_non_idempotent_owner_but_refuses_other_cre
         assert (await response.json())["outcome"] == "interrupted"
     finally:
         await _close_fixture(client, adapters, store, tasks)
+
+
+@pytest.mark.asyncio
+async def test_http_poll_reports_unknown_head_as_non_terminal_until_resolved(
+    tmp_path, monkeypatch
+):
+    """The documented discovery path: a client polls GET /v1/runs/{id}, learns the head is
+    unknown (not finished), reads admission_id/execution_generation from that body and
+    resolves it. ``interrupted`` is in TERMINAL_STATUSES and documented as terminal, so a
+    poller cannot tell a paused head from a finished run if the projection reuses it."""
+    from gateway.platforms.api_server_run_idempotency import TERMINAL_STATUSES
+
+    client, adapters, store, owner, unknown, _follower, executed, tasks = (
+        await _restarted_api_runs(tmp_path, monkeypatch)
+    )
+    try:
+        path = "/v1/runs/" + unknown["request_id"]
+        polled = await client.get(path, headers=_AUTH_HEADERS)
+        assert polled.status == 200, await polled.text()
+        body = await polled.json()
+        print("POLL_BEFORE_RESOLVE", json.dumps({k: body.get(k) for k in (
+            "status", "admission_id", "execution_generation")}, sort_keys=True))
+        assert body["status"] not in TERMINAL_STATUSES, body
+        assert body["status"] == "unknown", body
+        assert body["admission_id"] == unknown["admission_id"]
+        assert body["execution_generation"] == unknown["generation"]
+
+        resolved = await client.post(path + "/resolve-unknown", headers=_AUTH_HEADERS, json={
+            "admission_id": body["admission_id"],
+            "execution_generation": body["execution_generation"],
+        })
+        assert resolved.status == 200, await resolved.text()
+        await owner.sessions["api-session"].task
+        assert executed == ["FOLLOWER"]
+
+        after = await (await client.get(path, headers=_AUTH_HEADERS)).json()
+        print("POLL_AFTER_RESOLVE", json.dumps({"status": after.get("status")}))
+        assert after["status"] in TERMINAL_STATUSES, after
+    finally:
+        await _close_fixture(client, adapters, store, tasks)

website/docs/user-guide/features/api-server.md

@@ -514,7 +514,7 @@ Poll the current run state. This is useful for dashboards that need status witho
 
 `model` echoes what the request asked for. On a completed run, `runtime` is the provider/model pair that actually served the turn — after a [fallback provider](fallback-providers.md) switch it names the fallback pair, so a cost-attribution poller books the run to the right provider. `usage.cache_read_tokens` / `usage.cache_write_tokens` are the session's prompt-cache reads and writes, so cached input is not priced as full-price input. `runtime` has the same shape as on `/v1/chat/completions` and `/v1/responses`: `route_source` says how the runtime was chosen (`global`, `raw_request`, `model_routes`), and a request that named a `model`/`provider` also gets `requested: {provider, model}` so the asked-for and served pairs can be compared. The `run.completed` event on the events stream carries the same `usage` and `runtime` fields.
 
-Statuses are retained briefly after terminal states (`completed`, `failed`, `cancelled`, or `interrupted`) for polling and UI reconciliation. When the gateway shuts down while a run is active, the run is persisted as `interrupted` (error `Gateway shutdown interrupted the run.`, terminal event `run.interrupted`) before the agent is asked to stop, so a durable run never survives a restart as `running`; a late result from the interrupted turn cannot overwrite it.
+Statuses are retained briefly after terminal states (`completed`, `failed`, `cancelled`, or `interrupted`) for polling and UI reconciliation. When the gateway shuts down while a run is active, the run is persisted as `interrupted` (error `Gateway shutdown interrupted the run.`, terminal event `run.interrupted`) before the agent is asked to stop, so a durable run never survives a restart as `running`; a late result from the interrupted turn cannot overwrite it. A canonical run whose owner restarted mid-turn reports `unknown` instead, which is not terminal: later turns in that session wait until it is acknowledged with `POST /v1/runs/{run_id}/resolve-unknown` (below).
 
 While the gateway is still draining (a `hermes gateway stop`/`restart` or SIGTERM with a turn in flight), every non-terminal run additionally carries `shutdown_requested_at` (Unix seconds) from the moment new turns are refused. `status` stays `running` because the turn is still being served; a poller that sees the field knows the process is on its way out and the run will end `interrupted` at the latest when the drain budget expires. Terminal runs never gain the field.
 
@@ -584,9 +584,10 @@ subscriber continues draining normally.
 ### POST /v1/runs/\{run_id\}/resolve-unknown
 
 After an authority-owner restart, a turn that was already claimed has an
-unknown execution outcome. Hermes pauses later work in that session rather than
-risk replaying the lost head. Read the exact `admission_id` and
-`execution_generation` from `GET /v1/runs/{run_id}`, then acknowledge that the
+unknown execution outcome, and `GET /v1/runs/{run_id}` reports it as
+`"status": "unknown"` (not terminal). Hermes pauses later work in that session
+rather than risk replaying the lost head. Read the exact `admission_id` and
+`execution_generation` from that body, then acknowledge that the
 lost execution will not finish:
 
 ```json