fix(text): refuse NoEligibleWorker in place of a kudos refusal when no worker can serve the request

With no eligible worker the free allowance takes its floor, and a request over it was refused for kudos that do not help until a worker that can serve it comes online.
- A text or chat request the kudos rules would refuse while no online worker passes the dispatch gates is refused 403 NoEligibleWorker (TextAsyncGenerate.refuse_when_no_worker_is_eligible). The waiting prompt is deleted and the decision counter records refused.
- The NoEligibleWorker message states the missing declarations or exceeded limits when they are the cause. The body carries impossible_reason in the status route's shape.
- handle_bad_requests merges an exception's body_fields into the error body. Exceptions without body_fields keep the message and rc body.
- Funded requests and requests within the allowance are created and queued whether or not a worker can serve them.
- The front reads impossible_reason from a native error body onto HordeAPIError. _horde_error maps NoEligibleWorker to no_worker_supports_features, no_worker_supports_limit or no_eligible_worker.
- The front's gap and no-eligible-worker errors take request_id and job_may_exist. A refusal at submit carries no request id and does not state that the request was cancelled.
- _size_limit_gap_chat_offers takes the gap directly, so a submit refused on a size limit gap states the largest declared limit.
- NoEligibleWorker is in KNOWN_RC, README_return_codes.md, ADR 0037, chat_requests.md and chat_front.md.

README_return_codes.md

@@ -98,6 +98,7 @@ The errors returned by the AI horde are always in this json format
 | TimeoutIP | Operation rejected because user IP in timeout |
 | TooManyNewIPs | Too many workers from new IPs currently |
 | KudosUpfront | This request requires upfront kudos to accept |
+| NoEligibleWorker | A text or chat request would be refused for kudos while no online worker can serve it (HTTP 403); the message states the declaration gap when there is one, and the body then carries `impossible_reason`. Nothing was queued or charged |
 | SharedKeyEmpty | Shared Key used in the request does not have any more kudos |
 | InvalidJobID | Job not found when trying to submit. This probably means its request was delected for inactivity |
 | RequestNotFound | Request not found. This probably means it was delected for inactivity |

docs/decisions/0037-size-the-free-output-allowance-by-the-estimated-wait.md

@@ -47,7 +47,9 @@ Chosen option: "Estimated wait on the request's eligible workers".
 The allowance is `HORDE_TEXT_FREE_OUTPUT_CEILING` (4096 tokens) when the request's estimated wait is at or below
 `HORDE_TEXT_FREE_OUTPUT_WAIT_FULL_SECONDS` (60 s). It falls linearly to `HORDE_TEXT_FREE_OUTPUT_FLOOR` (512 tokens)
 at `HORDE_TEXT_FREE_OUTPUT_WAIT_FLOOR_SECONDS` (300 s) and stays at the floor beyond that. A request with no eligible
-worker gets the floor.
+worker gets the floor. A request the floor would refuse for kudos, when no online worker can serve it, is refused as
+`NoEligibleWorker` instead, with the declaration gap when there is one, since kudos do not help it and a kudos message
+would mislead the requester. A funded request with no eligible worker is admitted and queued.
 
 The estimated wait is the one ADR 0040 defines: the queued output ahead of the request that its eligible workers could
 take, divided by the combined measured speed of those workers. It is the same figure the status route reports as

docs/reference/chat_front.md

@@ -242,6 +242,14 @@ when the model list does not hold an entry for any requested model, the sentence
 limit name in its place. Volunteer workers come and go, so the message describes the online workers at the time and
 leaves retrying to the client. Any other gap, including a size limit beside another cause, keeps the 404.
 
+The Horde can also refuse the submit with 403 `NoEligibleWorker`, when the request would be refused for kudos while
+no online worker can serve it (see [Free output allowance](chat_requests.md#free-output-allowance)). The front
+answers it as above: the body's `impossible_reason` gives 404 `no_worker_supports_features` or 400
+`no_worker_supports_limit`, and a body without one gives 503 `no_eligible_worker` with `Retry-After: 30`. The Horde
+did not create a request, so these answers carry no `request_id`, do not state that the request was cancelled and
+do not set `x-should-retry: false`. The refusal comes before any stream header, so a streaming request gets the same
+JSON error.
+
 A request with `horde.availability_mode: "fail_fast"` is also decided at its first successful status read, with
 nothing processing or finished. Zero `eligible_worker_threads` cancels it at once and returns 503
 `queue_unavailable` with `Retry-After: 30`; a `wait_time` over the time left in the budget

docs/reference/chat_requests.md

@@ -898,6 +898,24 @@ kudos available, and they are held at admission. A chat request does not downgra
 the hold gets `403 KudosUpfront`, whose message states the allowance. A text request with `allow_downgrade` is
 lowered to the allowance instead.
 
+A request with no eligible worker gets the floor. When no online worker can serve it and it would be refused for
+kudos (`KudosUpfront` or `SharedKeyInsufficientKudos`), the route deletes its row and returns `403 NoEligibleWorker`
+instead, since kudos do not help it until such a worker comes online. The message reads "No online worker can serve
+this request", followed, when the workers' declarations are the cause, by the gap: "no online worker serving the model
+declares vision" for missing declarations, "the request exceeds the max_length limit of every online worker serving
+the model" for exceeded limits. With a gap, the body also carries `impossible_reason` in the shape the status route
+reports (see [Impossible reason](#impossible-reason)):
+
+```json
+{
+  "message": "No online worker can serve this request: no online worker serving the model declares vision.",
+  "rc": "NoEligibleWorker",
+  "impossible_reason": {"kind": "declaration_gap", "missing_declarations": ["vision"], "exceeded_limits": []}
+}
+```
+
+A request that covers its hold, or fits the floor, is created and queued whether or not a worker can serve it.
+
 The allowance is measured when the request is created, from the request's estimated wait on the workers eligible for
 it at its requested `max_length` (`TextWaitingPrompt.require_upfront_kudos`, `compute_free_output_allowance`):
 
@@ -937,7 +955,7 @@ These instruments, in `horde/metrics.py`, show the allowance's effect. A dry run
 
 | Instrument | Attributes | Records |
 | --- | --- | --- |
-| `horde.text.free_allowance.decisions` (counter) | `horde.requester` (`anonymous`, `registered`, `shared_key`), `horde.outcome` (`free`, `funded`, `refused`, `downgraded`), `horde.requested_bucket`, `horde.allowance_bucket`, `horde.chat` | Each request creation: admitted without a hold, admitted with kudos reserved, refused with `KudosUpfront` (including `SharedKeyInsufficientKudos`), or lowered to the allowance |
+| `horde.text.free_allowance.decisions` (counter) | `horde.requester` (`anonymous`, `registered`, `shared_key`), `horde.outcome` (`free`, `funded`, `refused`, `downgraded`), `horde.requested_bucket`, `horde.allowance_bucket`, `horde.chat` | Each request creation: admitted without a hold, admitted with kudos reserved, refused with `KudosUpfront` (including `SharedKeyInsufficientKudos`) or with `NoEligibleWorker` in their place, or lowered to the allowance |
 | `horde.text.free_allowance.estimated_wait_seconds` (histogram) | `horde.chat`, `horde.requester` | The wait the allowance was sized by; not recorded when no worker is eligible |
 | `horde.text.queue_wait_seconds` (histogram) | `horde.requester`, `horde.chat`, `horde.requested_bucket` | Each assignment's time from request creation to pop; a fake generation is not recorded |
 

horde/apis/v2/base.py

@@ -178,6 +178,7 @@ handle_unsafe_ip = api.errorhandler(e.UnsafeIP)(e.handle_bad_requests)
 handle_timeout_ip = api.errorhandler(e.TimeoutIP)(e.handle_bad_requests)
 handle_too_many_new_ips = api.errorhandler(e.TooManyNewIPs)(e.handle_bad_requests)
 handle_kudos_upfront = api.errorhandler(e.KudosUpfront)(e.handle_bad_requests)
+handle_no_eligible_worker = api.errorhandler(e.NoEligibleWorker)(e.handle_bad_requests)
 handle_invalid_procgen = api.errorhandler(e.InvalidJobID)(e.handle_bad_requests)
 handle_request_not_found = api.errorhandler(e.RequestNotFound)(e.handle_bad_requests)
 handle_worker_not_found = api.errorhandler(e.WorkerNotFound)(e.handle_bad_requests)

horde/apis/v2/kobold.py

@@ -280,6 +280,8 @@ class TextAsyncGenerate(GenerateTemplate):
         self.upload_source_images_or_delete_waiting_prompt()
         # Measured once here, at the requested max_length, and reused by the check after activation.
         self.creation_availability = database.get_worker_availability_for_request(self.wp)
+        # Kept apart from creation_availability, which a downgrade discards, for the kudos refusals below.
+        measured_availability = self.creation_availability
         highest_multiplier = 0
         if len(self.models) == 0:
             required_kudos = 20 * self.wp.n
@@ -309,6 +311,7 @@ class TextAsyncGenerate(GenerateTemplate):
                 self.downgrade_wp_priority = True
             else:
                 record_allowance_decision(metrics.FreeAllowanceOutcome.REFUSED)
+                self.refuse_when_no_worker_is_eligible(measured_availability)
                 self.wp.delete()
                 raise e.KudosUpfront(
                     required_kudos,
@@ -332,6 +335,7 @@ class TextAsyncGenerate(GenerateTemplate):
                     record_allowance_decision(metrics.FreeAllowanceOutcome.DOWNGRADED)
                 else:
                     record_allowance_decision(metrics.FreeAllowanceOutcome.REFUSED)
+                    self.refuse_when_no_worker_is_eligible(measured_availability)
                     # The quote is computed on the row before it is deleted, and only on this path: the refusal
                     # states the quoted cost beside the reservation so the two figures are not mistaken for one.
                     quoted_kudos = self.quote_kudos()
@@ -357,6 +361,24 @@ class TextAsyncGenerate(GenerateTemplate):
                     self.wp.delete()
                     raise e.BadRequest(fail_message)
 
+    def refuse_when_no_worker_is_eligible(self, availability: database.RequestWorkerAvailability) -> None:
+        """Delete the waiting prompt and raise ``NoEligibleWorker`` when no online worker can serve the request.
+
+        Called only where the request would be refused for kudos. With no eligible worker the free allowance takes
+        its floor, and kudos do not help until a worker that can serve the request comes online, so the refusal
+        states the declaration gap, when there is one, in place of the kudos message.
+
+        Args:
+            availability: The eligible workers measured at creation, at the requested max_length.
+
+        Raises:
+            NoEligibleWorker: No online worker passes the dispatch gates for the request.
+        """
+        if availability.worker_count > 0:
+            return
+        self.wp.delete()
+        raise e.NoEligibleWorker(availability.impossible_reason)
+
     def refuse_over_kudos_cap(self) -> None:
         """Refuse the request when its kudos estimate exceeds a cap its client set, deleting its row first.
 

horde/chat_contract.py

@@ -64,6 +64,7 @@ __all__ = [
     "MAX_CHAT_TOOL_NAME_LENGTH",
     "JSON_WORD",
     "KUDOS_CAP_EXCESS_PATTERN",
+    "NO_ELIGIBLE_WORKER_RC",
     "OBJECT_SCHEMA_TYPE",
     "SCHEMA_KEYWORDS_HOLDING_A_SCHEMA",
     "SCHEMA_KEYWORDS_HOLDING_SCHEMA_ARRAYS",
@@ -172,6 +173,10 @@ request without a model; the Horde's generic code for a 400, which the native ro
 CHAT_KUDOS_CAP_EXCEEDED_RC: str = "ChatKudosCapExceeded"
 """Return code for a request whose kudos estimate exceeds the ``horde.max_kudos`` its client set."""
 
+NO_ELIGIBLE_WORKER_RC: str = "NoEligibleWorker"
+"""Return code for a request the kudos rules would refuse while no online worker can serve it; the error body holds
+``impossible_reason`` when a declaration gap is the cause."""
+
 KUDOS_CAP_EXCESS_PATTERN: re.Pattern[str] = re.compile(
     r"kudos estimate at current pricing is (?P<kudos_estimate>\d+), above its horde\.max_kudos of (?P<max_kudos>\d+)",
 )

horde/exceptions.py

@@ -6,7 +6,7 @@ from flask import has_request_context, request
 from werkzeug import exceptions as wze
 
 from horde import metrics
-from horde.chat_contract import CHAT_KUDOS_CAP_EXCEEDED_RC, KudosCapExcess
+from horde.chat_contract import CHAT_KUDOS_CAP_EXCEEDED_RC, NO_ELIGIBLE_WORKER_RC, ChatImpossibleReason, KudosCapExcess
 from horde.logger import logger
 from horde.vars import horde_title
 
@@ -231,6 +231,7 @@ KNOWN_RC = [
     "ChatProgressCapReached",
     "ChatProgressUnavailable",
     "ChatJobAborted",
+    "NoEligibleWorker",
 ]
 
 
@@ -569,6 +570,43 @@ class KudosUpfront(wze.Forbidden):
         self.rc = rc
 
 
+class NoEligibleWorker(wze.Forbidden):
+    """A request the kudos rules would refuse while no online worker can serve it.
+
+    Kudos do not help such a request until a worker that can serve it comes online, so it is refused with the
+    declaration gap, when there is one, in place of ``KudosUpfront``. The status stays 403, as for ``KudosUpfront``.
+    """
+
+    body_fields: dict[str, object]
+    """Fields added to the error body beside ``message`` and ``rc``: ``impossible_reason`` in the shape the status
+    route reports it, when there is a declaration gap."""
+
+    def __init__(self, impossible_reason: ChatImpossibleReason | None, rc: str = NO_ELIGIBLE_WORKER_RC) -> None:
+        """Create the refusal.
+
+        Args:
+            impossible_reason: The missing declarations and exceeded limits that leave no worker eligible; None when
+                the cause is not a declaration gap, such as no online worker serving the model.
+            rc: The return code.
+        """
+        causes: list[str] = []
+        if impossible_reason is not None and impossible_reason.missing_declarations:
+            causes.append(
+                "no online worker serving the model declares " + ", ".join(impossible_reason.missing_declarations),
+            )
+        if impossible_reason is not None and impossible_reason.exceeded_limits:
+            causes.append(
+                f"the request exceeds the {', '.join(impossible_reason.exceeded_limits)} limit of every online worker serving the model",
+            )
+        self.specific = "No online worker can serve this request"
+        self.specific += f": {'; '.join(causes)}." if causes else "."
+        self.log = None
+        self.rc = rc
+        self.body_fields = {}
+        if impossible_reason is not None:
+            self.body_fields["impossible_reason"] = impossible_reason.model_dump(mode="json")
+
+
 class InvalidJobID(wze.NotFound):
     def __init__(self, job_id, rc="InvalidJobID"):
         self.specific = f"Processing Job with ID {job_id} does not exist."
@@ -759,6 +797,7 @@ def handle_bad_requests(error):
         {
             "message": error.specific,
             "rc": error.rc,
+            **getattr(error, "body_fields", {}),
         },
         error.code,
     )

horde/oai/app.py

@@ -60,6 +60,7 @@ from horde.chat_contract import (
     MAX_CHAT_IMAGE_BYTES,
     MAX_CHAT_IMAGE_COUNT,
     MAX_CHAT_IMAGE_TOTAL_BYTES,
+    NO_ELIGIBLE_WORKER_RC,
     ChatAvailabilityMode,
     ChatImpossibleReason,
     ChatRequestRefusal,
@@ -790,15 +791,42 @@ def _request_source(request: Request) -> str:
     return request.client.host if request.client is not None else UNKNOWN_SOURCE
 
 
-def _horde_error(error: HordeAPIError, *, request_id: str | None, job_may_exist: bool) -> ChatFrontError:
+def _horde_error(
+    error: HordeAPIError,
+    *,
+    request_id: str | None,
+    job_may_exist: bool,
+    chat_offers: Sequence[HordeChatModelOffer] = (),
+    listed_model_count: int = 1,
+) -> ChatFrontError:
     """Create the front error for a Horde error, keeping its status, ``rc`` and forwarded headers.
 
     A native payload validation failure, which has no ``rc``, is reported as ``BadRequest`` with its first failing
     field as ``param``. A 429 without an ``rc`` is reported as ``rate_limit_exceeded``, keeping the Horde's message and
-    ``Retry-After``. A refusal over the client's kudos cap is reported as ``max_kudos_exceeded``.
+    ``Retry-After``. A refusal over the client's kudos cap is reported as ``max_kudos_exceeded``. A ``NoEligibleWorker``
+    refusal is reported as the declaration gap or size limit gap its body holds, as for a wait that ends on that gap,
+    and as ``no_eligible_worker`` when the body holds none.
+
+    Args:
+        error: The Horde error.
+        request_id: The Horde request id, or None when the Horde did not create a request.
+        job_may_exist: Whether the Horde holds, or may hold, a request for this call.
+        chat_offers: What the online workers serving each requested model declare, for the largest limit a size
+            limit gap states.
+        listed_model_count: How many models the request accepts, which words the size limit gap message.
     """
     if error.return_code == CHAT_KUDOS_CAP_EXCEEDED_RC:
         return _kudos_cap_error(error, request_id=request_id, job_may_exist=job_may_exist)
+    if error.return_code == NO_ELIGIBLE_WORKER_RC:
+        if error.impossible_reason is None:
+            return _no_eligible_worker_error(request_id=request_id, job_may_exist=job_may_exist)
+        return _gap_error(
+            error.impossible_reason,
+            request_id=request_id,
+            job_may_exist=job_may_exist,
+            chat_offers=chat_offers,
+            listed_model_count=listed_model_count,
+        )
     retry_after_header = next(
         (
             header_value
@@ -1174,6 +1202,7 @@ async def _submit(
     operation_key: uuid.UUID,
     credentials: HordeCredentials,
     deadline: float,
+    models: Sequence[str],
 ) -> HordeAsyncSubmission:
     """Submit a chat request within the time left in its budget and return the Horde's acknowledgement.
 
@@ -1186,6 +1215,7 @@ async def _submit(
         operation_key: The key of this client request.
         credentials: Who the call is made for.
         deadline: The monotonic time the request's total budget ends.
+        models: The models the request accepts, for the largest limit a size limit gap refusal states.
 
     Raises:
         ChatFrontError: The Horde refused the request, no root accepted the connection, or the outcome stayed unknown.
@@ -1200,7 +1230,19 @@ async def _submit(
             "ChatImageStorageUnavailable",
         }
         if error.status_code < HTTPStatus.INTERNAL_SERVER_ERROR or native_refusal:
-            front_error = _horde_error(error, request_id=None, job_may_exist=False)
+            chat_offers = await _size_limit_gap_chat_offers(
+                front_state,
+                error.impossible_reason,
+                models=models,
+                credentials=credentials,
+            )
+            front_error = _horde_error(
+                error,
+                request_id=None,
+                job_may_exist=False,
+                chat_offers=chat_offers,
+                listed_model_count=len(models),
+            )
             if error.status_code == HTTPStatus.SERVICE_UNAVAILABLE and front_error.metadata.retry_after is None:
                 front_error.headers[RETRY_AFTER_HEADER] = str(HORDE_UNAVAILABLE_RETRY_AFTER_SECONDS)
                 front_error.metadata = front_error.metadata.model_copy(update={"retry_after": HORDE_UNAVAILABLE_RETRY_AFTER_SECONDS})
@@ -1731,7 +1773,7 @@ async def _settle_wait_failure(
     requested_models = chat_request.requested_models()
     chat_offers = await _size_limit_gap_chat_offers(
         front_state,
-        error,
+        error.impossible_reason if isinstance(error, DeclarationGapError) else None,
         models=requested_models,
         credentials=open_request.credentials,
     )
@@ -1798,28 +1840,71 @@ def _wait_error(
             job_may_exist=True,
         )
     if isinstance(error, DeclarationGapError):
-        if _is_size_limit_gap(error.impossible_reason):
-            return _size_limit_gap_error(
-                error.impossible_reason,
-                request_id=request_id,
-                chat_offers=chat_offers,
-                listed_model_count=listed_model_count,
-            )
-        return _declaration_gap_error(error.impossible_reason, request_id=request_id)
+        return _gap_error(
+            error.impossible_reason,
+            request_id=request_id,
+            job_may_exist=True,
+            chat_offers=chat_offers,
+            listed_model_count=listed_model_count,
+        )
     if isinstance(error, QueueUnavailableError):
         return _queue_unavailable_error(error, request_id=request_id)
     if isinstance(error, NoEligibleWorkerError):
-        return ChatFrontError(
-            status_code=HTTPStatus.SERVICE_UNAVAILABLE,
-            message="No online worker can serve this request at the moment; it was cancelled. Retry later.",
-            code=NO_ELIGIBLE_WORKER_CODE,
-            metadata=ErrorMetadata(request_id=request_id, retry_after=NO_ELIGIBLE_WORKER_RETRY_AFTER_SECONDS),
-            headers={RETRY_AFTER_HEADER: str(NO_ELIGIBLE_WORKER_RETRY_AFTER_SECONDS)},
-            job_may_exist=True,
-        )
+        return _no_eligible_worker_error(request_id=request_id, job_may_exist=True)
     return _horde_error(error, request_id=request_id, job_may_exist=True)
 
 
+def _no_eligible_worker_error(*, request_id: str | None, job_may_exist: bool) -> ChatFrontError:
+    """Create the 503 for a request no online worker can serve, without a declaration gap to state.
+
+    Args:
+        request_id: The Horde request id, which the front cancelled; None when the Horde refused the request at
+            creation and holds no request.
+        job_may_exist: Whether the Horde holds, or may hold, a request for this call.
+    """
+    message = "No online worker can serve this request at the moment"
+    message += "; it was cancelled. Retry later." if request_id is not None else ". Retry later."
+    return ChatFrontError(
+        status_code=HTTPStatus.SERVICE_UNAVAILABLE,
+        message=message,
+        code=NO_ELIGIBLE_WORKER_CODE,
+        metadata=ErrorMetadata(request_id=request_id, retry_after=NO_ELIGIBLE_WORKER_RETRY_AFTER_SECONDS),
+        headers={RETRY_AFTER_HEADER: str(NO_ELIGIBLE_WORKER_RETRY_AFTER_SECONDS)},
+        job_may_exist=job_may_exist,
+    )
+
+
+def _gap_error(
+    impossible_reason: ChatImpossibleReason,
+    *,
+    request_id: str | None,
+    job_may_exist: bool,
+    chat_offers: Sequence[HordeChatModelOffer],
+    listed_model_count: int,
+) -> ChatFrontError:
+    """Create the front error for a declaration gap: the size limit gap 400 when the gap holds only size limits, the
+    declaration gap 404 otherwise.
+
+    Args:
+        impossible_reason: The gap.
+        request_id: The Horde request id, which the front cancelled; None when the Horde refused the request at
+            creation and holds no request.
+        job_may_exist: Whether the Horde holds, or may hold, a request for this call.
+        chat_offers: What the online workers serving each requested model declare, for the largest limit a size
+            limit gap states.
+        listed_model_count: How many models the request accepts.
+    """
+    if _is_size_limit_gap(impossible_reason):
+        return _size_limit_gap_error(
+            impossible_reason,
+            request_id=request_id,
+            job_may_exist=job_may_exist,
+            chat_offers=chat_offers,
+            listed_model_count=listed_model_count,
+        )
+    return _declaration_gap_error(impossible_reason, request_id=request_id, job_may_exist=job_may_exist)
+
+
 def _queue_unavailable_error(error: QueueUnavailableError, *, request_id: str) -> ChatFrontError:
     """Create the 503 for a ``fail_fast`` request its first status read refused.
 
@@ -1866,11 +1951,22 @@ def _queue_unavailable_error(error: QueueUnavailableError, *, request_id: str) -
     )
 
 
-def _declaration_gap_error(impossible_reason: ChatImpossibleReason, *, request_id: str) -> ChatFrontError:
+def _declaration_gap_error(
+    impossible_reason: ChatImpossibleReason,
+    *,
+    request_id: str | None,
+    job_may_exist: bool,
+) -> ChatFrontError:
     """Create the 404 for a request the online workers serving its model cannot take because of what they declare.
 
     Retrying the same request does not help until a worker declaring the missing values comes online, so the error
     carries no ``Retry-After``.
+
+    Args:
+        impossible_reason: The gap.
+        request_id: The Horde request id, which the front cancelled; None when the Horde refused the request at
+            creation and holds no request.
+        job_may_exist: Whether the Horde holds, or may hold, a request for this call.
     """
     causes: list[str] = []
     if impossible_reason.missing_declarations:
@@ -1884,13 +1980,14 @@ def _declaration_gap_error(impossible_reason: ChatImpossibleReason, *, request_i
     message = f"No worker can serve this request: {'; '.join(causes) or 'the workers lack a declaration it needs'}."
     if GUARANTEE_NAMES.intersection(impossible_reason.missing_declarations):
         message += " Guarantees are not required under horde.conformance 'best_effort'."
-    message += " The request was cancelled."
+    if request_id is not None:
+        message += " The request was cancelled."
     return ChatFrontError(
         status_code=HTTPStatus.NOT_FOUND,
         message=message,
         code=NO_WORKER_SUPPORTS_FEATURES_CODE,
         metadata=ErrorMetadata(request_id=request_id),
-        job_may_exist=True,
+        job_may_exist=job_may_exist,
     )
 
 
@@ -1910,7 +2007,8 @@ def _is_size_limit_gap(impossible_reason: ChatImpossibleReason) -> bool:
 def _size_limit_gap_error(
     impossible_reason: ChatImpossibleReason,
     *,
-    request_id: str,
+    request_id: str | None,
+    job_may_exist: bool,
     chat_offers: Sequence[HordeChatModelOffer],
     listed_model_count: int,
 ) -> ChatFrontError:
@@ -1923,7 +2021,9 @@ def _size_limit_gap_error(
 
     Args:
         impossible_reason: The gap, with only size limits in ``exceeded_limits``.
-        request_id: The Horde request id.
+        request_id: The Horde request id, which the front cancelled; None when the Horde refused the request at
+            creation and holds no request.
+        job_may_exist: Whether the Horde holds, or may hold, a request for this call.
         chat_offers: What the online workers serving each requested model declare, for the models the model list has.
         listed_model_count: How many models the request accepts.
     """
@@ -1943,7 +2043,8 @@ def _size_limit_gap_error(
                 f"No worker eligible for this request currently declares a {limit_wording} that high; "
                 f"the largest any online worker for {model_wording} declares now is {largest_declared} tokens.",
             )
-    sentences.append("The request was cancelled.")
+    if request_id is not None:
+        sentences.append("The request was cancelled.")
     param_limit = "max_length" if "max_length" in impossible_reason.exceeded_limits else "max_context_length"
     return ChatFrontError(
         status_code=HTTPStatus.BAD_REQUEST,
@@ -1951,18 +2052,18 @@ def _size_limit_gap_error(
         code=NO_WORKER_SUPPORTS_LIMIT_CODE,
         param=NATIVE_FIELD_PARAMETERS[param_limit],
         metadata=ErrorMetadata(request_id=request_id),
-        job_may_exist=True,
+        job_may_exist=job_may_exist,
     )
 
 
 async def _size_limit_gap_chat_offers(
     front_state: FrontState,
-    error: Exception,
+    impossible_reason: ChatImpossibleReason | None,
     *,
     models: Sequence[str],
     credentials: HordeCredentials,
 ) -> list[HordeChatModelOffer]:
-    """Return the requested models' chat entries from the model list when a wait ended on a size limit gap.
+    """Return the requested models' chat entries from the model list when a request was refused on a size limit gap.
 
     Any requested model may serve the request, so the 400 states the largest limit across all of them. The list is
     read through the process cache. A failed read, or a gap of another kind, returns no entries and leaves the 400
@@ -1970,11 +2071,12 @@ async def _size_limit_gap_chat_offers(
 
     Args:
         front_state: The process state holding the model list cache.
-        error: The error that ended the wait.
+        impossible_reason: The gap a wait ended on or a ``NoEligibleWorker`` refusal reported; None when there is
+            none.
         models: The models the request accepts.
         credentials: The client's credentials, used for the model list read.
     """
-    if not isinstance(error, DeclarationGapError) or not _is_size_limit_gap(error.impossible_reason):
+    if impossible_reason is None or not _is_size_limit_gap(impossible_reason):
         return []
     try:
         active_models = await front_state.model_list_cache.get(front_state.horde_client, credentials)
@@ -2087,6 +2189,7 @@ async def _chat_completions(request: Request) -> Response:
             operation_key=operation_key,
             credentials=credentials,
             deadline=deadline,
+            models=chat_request.requested_models(),
         )
     except ChatFrontError as error:
         inflight_slot.release()

horde/oai/horde_client.py

@@ -208,6 +208,7 @@ class HordeAPIError(Exception):
         message: str,
         forwarded_headers: Mapping[str, str],
         field_errors: Mapping[str, str] | None = None,
+        impossible_reason: ChatImpossibleReason | None = None,
     ) -> None:
         """Create the error.
 
@@ -218,6 +219,8 @@ class HordeAPIError(Exception):
             forwarded_headers: The ``Retry-After`` and rate-limit headers to pass back to the client.
             field_errors: The native payload validation failures by dotted field path, such as
                 ``{"temperature": "5.5 is greater than the maximum of 5.0"}``; empty when the body listed none.
+            impossible_reason: The declaration gap a ``NoEligibleWorker`` refusal reports; None when the body has none
+                or holds a reason of a kind this front does not know.
         """
         super().__init__(f"{status_code} {return_code}: {message}")
         self.status_code: int = status_code
@@ -230,6 +233,8 @@ class HordeAPIError(Exception):
         """The headers to pass back to the client."""
         self.field_errors: dict[str, str] = dict(field_errors) if field_errors is not None else {}
         """The native payload validation failures by dotted field path, in the Horde's order."""
+        self.impossible_reason: ChatImpossibleReason | None = impossible_reason
+        """The declaration gap the refusal reports."""
 
 
 class HordeUnavailableError(Exception):
@@ -819,6 +824,7 @@ def _api_error_from_response(response: httpx.Response) -> HordeAPIError:
     return_code: str | None = None
     message = response.text[:MAX_ERROR_BODY_CHARACTERS]
     field_errors: dict[str, str] = {}
+    impossible_reason: ChatImpossibleReason | None = None
     try:
         error_body = response.json()
     except ValueError:
@@ -833,10 +839,18 @@ def _api_error_from_response(response: httpx.Response) -> HordeAPIError:
         body_field_errors = error_body.get("errors")
         if isinstance(body_field_errors, dict):
             field_errors = {str(field_path): str(field_message) for field_path, field_message in body_field_errors.items()}
+        body_impossible_reason = error_body.get("impossible_reason")
+        if body_impossible_reason is not None:
+            # A reason of a kind this front does not know is read as none, as on the status route.
+            try:
+                impossible_reason = ChatImpossibleReason.model_validate(body_impossible_reason)
+            except ValidationError:
+                impossible_reason = None
     return HordeAPIError(
         status_code=response.status_code,
         return_code=return_code,
         message=message,
         forwarded_headers=forwarded_headers,
         field_errors=field_errors,
+        impossible_reason=impossible_reason,
     )

tests/integration/test_chat.py

@@ -1177,11 +1177,14 @@ def test_a_quote_cached_for_a_funded_requester_does_not_answer_an_anonymous_text
     make_api_user,
     settle_kudos,
 ) -> None:
-    """The quote cache is keyed by the params alone, so it must not answer a dry run the requester would be refused."""
+    """The quote cache is keyed by the params alone, so it must not answer a dry run the requester would be refused.
+
+    No worker serves the model, so the allowance is at its floor, 2048 tokens is above it, and the anonymous dry run is
+    refused ``NoEligibleWorker`` in place of ``KudosUpfront``, without a declaration gap to state.
+    """
     funded = make_api_user(kudos=100000)
     settle_kudos()
     funded_headers = {"apikey": funded.api_key, "Client-Agent": "aihorde_ci_client:1:test"}
-    # No worker serves the model, so the allowance is at its floor and 2048 tokens is above it.
     text_body = {
         "prompt": "Rivers.",
         "params": {"max_length": 2048, "max_context_length": 8192},
@@ -1194,7 +1197,10 @@ def test_a_quote_cached_for_a_funded_requester_does_not_answer_an_anonymous_text
     anonymous_quote = client.post("/api/v2/generate/text/async", json=text_body, headers=ANONYMOUS_HEADERS)
 
     assert anonymous_quote.status_code == 403, anonymous_quote.get_data(as_text=True)
-    assert anonymous_quote.get_json()["rc"] == "KudosUpfront"
+    assert anonymous_quote.get_json() == {
+        "message": "No online worker can serve this request.",
+        "rc": "NoEligibleWorker",
+    }
 
 
 def test_a_repeated_operation_key_is_answered_while_admission_is_off(client, make_api_user, monkeypatch) -> None:

tests/unit/oai/test_errors.py

@@ -600,6 +600,101 @@ async def test_an_impossible_reason_of_an_unknown_kind_waits_out_the_grace(start
     assert response.json()["error"]["code"] == "no_eligible_worker"
 
 
+def no_eligible_worker_refusal(
+    *,
+    missing_declarations: list[str] | None = None,
+    exceeded_limits: list[str] | None = None,
+) -> dict[str, object]:
+    """Return the native body of a submit refused ``NoEligibleWorker``, with ``impossible_reason`` when a gap is given."""
+    if missing_declarations is None and exceeded_limits is None:
+        return {
+            "message": "No online worker can serve this request.",
+            "rc": "NoEligibleWorker",
+        }
+    return {
+        "message": "No online worker can serve this request: a declaration gap.",
+        "rc": "NoEligibleWorker",
+        "impossible_reason": {
+            "kind": "declaration_gap",
+            "missing_declarations": missing_declarations or [],
+            "exceeded_limits": exceeded_limits or [],
+        },
+    }
+
+
+@pytest.mark.parametrize("stream", [False, True])
+async def test_a_submit_refused_for_a_declaration_gap_is_404_without_a_request(start_front: StartFront, stream: bool) -> None:
+    """The Horde refuses the submit and creates no request, so the 404 states the gap, carries no request id and
+    leaves retrying to the client; a streaming request gets the same JSON error before any stream header."""
+    async with start_front() as front:
+        front.horde.submit_route.respond(403, json=no_eligible_worker_refusal(missing_declarations=["vision"]))
+        response = await front.client.post("/v1/chat/completions", json=build_chat_body(stream=stream))
+
+    assert front.horde.status_route.call_count == 0
+    assert front.horde.cancel_route.call_count == 0
+    assert SHOULD_RETRY not in response.headers
+    assert "retry-after" not in response.headers
+    assert response.status_code == 404
+    assert response.headers["content-type"] == "application/json"
+    assert response.json() == {
+        "error": {
+            "message": "No worker can serve this request: no online worker serving the model declares vision.",
+            "type": "not_found_error",
+            "param": None,
+            "code": "no_worker_supports_features",
+            "metadata": {},
+        },
+    }
+    assert front.inflight == 0
+
+
+@pytest.mark.parametrize("stream", [False, True])
+async def test_a_submit_refused_for_a_size_limit_gap_is_400_stating_the_largest_declared(
+    start_front: StartFront,
+    stream: bool,
+) -> None:
+    async with start_front() as front:
+        front.horde.models_route.respond(200, json=MODEL_LIST)
+        front.horde.submit_route.respond(403, json=no_eligible_worker_refusal(exceeded_limits=["max_length"]))
+        response = await front.client.post("/v1/chat/completions", json=build_chat_body(stream=stream))
+
+    assert SHOULD_RETRY not in response.headers
+    assert response.status_code == 400
+    assert response.json() == {
+        "error": {
+            "message": (
+                "No worker eligible for this request currently declares a completion limit that high; the largest any online "
+                "worker for the model declares now is 1024 tokens."
+            ),
+            "type": "invalid_request_error",
+            "param": "max_tokens",
+            "code": "no_worker_supports_limit",
+            "metadata": {},
+        },
+    }
+
+
+@pytest.mark.parametrize("stream", [False, True])
+async def test_a_submit_refused_without_a_gap_is_503_no_eligible_worker_without_a_request(start_front: StartFront, stream: bool) -> None:
+    async with start_front() as front:
+        front.horde.submit_route.respond(403, json=no_eligible_worker_refusal())
+        response = await front.client.post("/v1/chat/completions", json=build_chat_body(stream=stream))
+
+    assert front.horde.cancel_route.call_count == 0
+    assert SHOULD_RETRY not in response.headers
+    assert response.status_code == 503
+    assert response.headers["retry-after"] == "30"
+    assert response.json() == {
+        "error": {
+            "message": "No online worker can serve this request at the moment. Retry later.",
+            "type": "api_error",
+            "param": None,
+            "code": "no_eligible_worker",
+            "metadata": {"retry_after": 30},
+        },
+    }
+
+
 async def test_no_eligible_worker_does_not_fail_while_a_worker_is_processing(start_front: StartFront) -> None:
     fail_fast_settings = FAST_TEST_SETTINGS.model_copy(update={"fail_fast_grace_seconds": 0.1, "fallback_poll_seconds": 0.05})
     async with start_front(fail_fast_settings, with_redis=False) as front:

tests/unit/test_kobold_text_wp.py

@@ -28,12 +28,14 @@ from horde import exceptions as e
 from horde import metrics
 from horde.apis.v2 import kobold as text_api_module
 from horde.apis.v2.kobold import TextAsyncGenerate
+from horde.chat_contract import ChatImpossibleReason
 from horde.classes.base.waiting_prompt import estimate_queue_wait_seconds
 from horde.classes.kobold import processing_generation as text_processing_generation
 from horde.classes.kobold.processing_generation import TextProcessingGeneration
 from horde.classes.kobold.waiting_prompt import TextWaitingPrompt, compute_free_output_allowance
 from horde.classes.kobold.worker import TextWorker
 from horde.config import Config
+from horde.database.functions import RequestWorkerAvailability
 from horde.flask import db
 
 
@@ -199,6 +201,27 @@ class TestCalculateKudos:
 ALLOWANCE_MODEL = "allowance-model"
 """The model every request in the allowance records tests asks for."""
 
+ONE_ELIGIBLE_WORKER = RequestWorkerAvailability(
+    worker_count=1,
+    thread_count=1,
+)
+"""The availability of a request one online worker can serve, which keeps a kudos refusal a ``KudosUpfront``."""
+
+NO_ELIGIBLE_WORKER = RequestWorkerAvailability(
+    worker_count=0,
+    thread_count=0,
+)
+"""The availability of a request no online worker can serve, without a declaration gap."""
+
+VISION_GAP = RequestWorkerAvailability(
+    worker_count=0,
+    thread_count=0,
+    impossible_reason=ChatImpossibleReason(
+        missing_declarations=("vision",),
+    ),
+)
+"""The availability of a request with an image that no online worker serving the model declares ``vision`` for."""
+
 
 class _CounterRecorder:
     """Stands in for a metric counter and keeps every increment with its attributes."""
@@ -230,9 +253,10 @@ def _queue_allowance_request(
     allow_downgrade: bool = False,
     is_chat: bool = False,
     sharedkey: object | None = None,
+    availability: RequestWorkerAvailability = ONE_ELIGIBLE_WORKER,
 ) -> TextAsyncGenerate:
-    """Create a text request through ``TextAsyncGenerate.initiate_waiting_prompt`` with the wait and the kudos
-    reservation fixed, so each allowance outcome is reached deterministically.
+    """Create a text request through ``TextAsyncGenerate.initiate_waiting_prompt`` with the wait, the eligible workers
+    and the kudos reservation fixed, so each allowance outcome is reached deterministically.
 
     A request costs one kudo per token, so any request that needs kudos upfront needs a reservation.
     """
@@ -279,6 +303,7 @@ def _queue_allowance_request(
         },
     )
     monkeypatch.setattr(resource, "estimate_competing_wait_seconds", lambda: estimated_wait_seconds)
+    monkeypatch.setattr(text_api_module.database, "get_worker_availability_for_request", lambda _wp: availability)
     monkeypatch.setattr(
         text_api_module,
         "reserve_kudos",
@@ -498,6 +523,163 @@ class TestFreeAllowanceRecords:
         assert estimated_waits.records == []
 
 
+class TestNoEligibleWorkerRefusal:
+    """A request the kudos rules would refuse while no online worker can serve it is refused ``NoEligibleWorker``, with
+    the declaration gap in its message and body when there is one."""
+
+    @pytest.fixture(autouse=True)
+    def default_allowance_settings(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        monkeypatch.setattr(Config, "text_free_output_ceiling", FREE_OUTPUT_CEILING)
+        monkeypatch.setattr(Config, "text_free_output_floor", FREE_OUTPUT_FLOOR)
+        monkeypatch.setattr(Config, "text_free_output_wait_full_seconds", FULL_WAIT_SECONDS)
+        monkeypatch.setattr(Config, "text_free_output_wait_floor_seconds", FLOOR_WAIT_SECONDS)
+        monkeypatch.delenv("HORDE_UPFRONT_KUDOS_ON_WORKERLIST", raising=False)
+        monkeypatch.setattr(text_api_module.model_reference, "get_text_model_multiplier", lambda _model_name: 21)
+
+    @pytest.fixture
+    def decisions(self, monkeypatch: pytest.MonkeyPatch) -> _CounterRecorder:
+        recorder = _CounterRecorder()
+        monkeypatch.setattr(metrics, "text_free_allowance_decisions", recorder)
+        return recorder
+
+    def test_an_anonymous_request_over_the_floor_with_a_declaration_gap_is_refused_with_the_gap(
+        self,
+        db_session: Any,
+        fake_redis: Any,
+        make_user: Any,
+        monkeypatch: pytest.MonkeyPatch,
+        decisions: _CounterRecorder,
+    ) -> None:
+        """An anonymous 2048-token chat request that no online worker declares ``vision`` for is refused 403
+        ``NoEligibleWorker`` stating the gap, its body carries ``impossible_reason``, its row is deleted, and the
+        decision is counted ``refused``."""
+        with pytest.raises(e.NoEligibleWorker) as refusal:
+            _queue_allowance_request(
+                monkeypatch,
+                make_user(oauth_id="anon"),
+                max_length=2048,
+                estimated_wait_seconds=None,
+                reservation_granted=False,
+                is_chat=True,
+                availability=VISION_GAP,
+            )
+
+        body, status_code = e.handle_bad_requests(refusal.value)
+        assert status_code == 403
+        assert body == {
+            "message": "No online worker can serve this request: no online worker serving the model declares vision.",
+            "rc": "NoEligibleWorker",
+            "impossible_reason": {
+                "kind": "declaration_gap",
+                "missing_declarations": ["vision"],
+                "exceeded_limits": [],
+            },
+        }
+        assert db.session.query(TextWaitingPrompt).count() == 0
+        assert [attributes["horde.outcome"] for _, attributes in decisions.increments] == ["refused"]
+
+    def test_an_anonymous_request_over_the_floor_without_a_worker_or_gap_is_refused_without_a_gap(
+        self,
+        db_session: Any,
+        fake_redis: Any,
+        make_user: Any,
+        monkeypatch: pytest.MonkeyPatch,
+        decisions: _CounterRecorder,
+    ) -> None:
+        """An anonymous 2048-token request for a model no online worker serves is refused 403 ``NoEligibleWorker``
+        with only ``message`` and ``rc`` in its body."""
+        with pytest.raises(e.NoEligibleWorker) as refusal:
+            _queue_allowance_request(
+                monkeypatch,
+                make_user(oauth_id="anon"),
+                max_length=2048,
+                estimated_wait_seconds=None,
+                reservation_granted=False,
+                availability=NO_ELIGIBLE_WORKER,
+            )
+
+        body, status_code = e.handle_bad_requests(refusal.value)
+        assert status_code == 403
+        assert body == {
+            "message": "No online worker can serve this request.",
+            "rc": "NoEligibleWorker",
+        }
+        assert db.session.query(TextWaitingPrompt).count() == 0
+
+    def test_a_shared_key_without_kudos_and_without_a_worker_is_refused_no_eligible_worker(
+        self,
+        db_session: Any,
+        fake_redis: Any,
+        make_user: Any,
+        monkeypatch: pytest.MonkeyPatch,
+        decisions: _CounterRecorder,
+    ) -> None:
+        """A request whose shared key cannot cover it, for a model no online worker serves, is refused
+        ``NoEligibleWorker`` in place of ``SharedKeyInsufficientKudos``."""
+        empty_sharedkey = SimpleNamespace(
+            id="empty-key",
+            kudos=0,
+            is_job_within_limits=lambda **_limits: (True, None),
+        )
+
+        with pytest.raises(e.NoEligibleWorker):
+            _queue_allowance_request(
+                monkeypatch,
+                make_user(),
+                max_length=2048,
+                estimated_wait_seconds=None,
+                reservation_granted=False,
+                sharedkey=empty_sharedkey,
+                availability=NO_ELIGIBLE_WORKER,
+            )
+
+        assert db.session.query(TextWaitingPrompt).count() == 0
+
+    def test_an_anonymous_request_over_the_floor_with_an_eligible_worker_is_refused_for_kudos(
+        self,
+        db_session: Any,
+        fake_redis: Any,
+        make_user: Any,
+        monkeypatch: pytest.MonkeyPatch,
+        decisions: _CounterRecorder,
+    ) -> None:
+        """An anonymous 2048-token request at a 300 s wait that one online worker can serve is refused
+        ``KudosUpfront``."""
+        with pytest.raises(e.KudosUpfront) as refusal:
+            _queue_allowance_request(
+                monkeypatch,
+                make_user(oauth_id="anon"),
+                max_length=2048,
+                estimated_wait_seconds=FLOOR_WAIT_SECONDS,
+                reservation_granted=False,
+                availability=ONE_ELIGIBLE_WORKER,
+            )
+
+        assert refusal.value.rc == "KudosUpfront"
+
+    def test_a_funded_request_without_an_eligible_worker_is_created(
+        self,
+        db_session: Any,
+        fake_redis: Any,
+        make_user: Any,
+        monkeypatch: pytest.MonkeyPatch,
+        decisions: _CounterRecorder,
+    ) -> None:
+        """A registered user's 2048-token request whose kudos are reserved is created and counted ``funded`` although
+        no online worker can serve it."""
+        resource = _queue_allowance_request(
+            monkeypatch,
+            make_user(),
+            max_length=2048,
+            estimated_wait_seconds=None,
+            reservation_granted=True,
+            availability=VISION_GAP,
+        )
+
+        assert db.session.query(TextWaitingPrompt).filter_by(id=resource.wp.id).count() == 1
+        assert [attributes["horde.outcome"] for _, attributes in decisions.increments] == ["funded"]
+
+
 class TestQueueWaitRecord:
     """Every real assignment records the time from its request's creation to the pop."""