mirror of
https://github.com/Haidra-Org/AI-Horde.git
synced 2026-11-06 05:17:31 +00:00
feat(front): return reasoning by default and pace a post's reasoning before its text
A reasoning model can think for most of a minute before its first visible token. Clients that read the DeepSeek `reasoning_content` field show that progress live, and the OpenAI SDK keeps the unknown field in `model_extra` without complaint, so returning the reasoning by default costs a plain client nothing. - `horde.include_reasoning` defaults to true. `false` hides the reasoning. The option changes what the front delivers and not the work, so it stays out of the stored request and the operation fingerprint - `DeltaPacer._spread` schedules every reasoning piece of a worker post before its first content piece. The piece budget is divided between the two fields by length, with at least one piece for each field with text, so a client that renders in arrival order shows the end of the reasoning and then the start of the reply - `DeltaRelay`, `build_completion` and the SSE module read the option as on unless false - Tests cover the default without a `horde` block and the reasoning-first order of a paced post. Relay tests that assert dropped reasoning turn the option off explicitly - ADR 0020, the chat front and chat requests references and the how-to state the default and the pacing order
This commit is contained in:
11 files changed
+98
-70
No files matched your search
@@ -52,7 +52,7 @@ defines it for both the front and the native route, and an unknown key is refuse
|
||||
- `openai_guards`: turns on the request refusals OpenAI applies and the Horde relaxes.
|
||||
- `availability_mode`: wait for a worker, or refuse at once when the online pool cannot serve the request.
|
||||
- `max_kudos`: the most the request may cost at creation.
|
||||
- `include_reasoning`: return the model's reasoning beside the reply.
|
||||
- `include_reasoning`: return the model's reasoning beside the reply, on by default.
|
||||
- `dry_run` and `operation_key`: native callers only. The front refuses both in a public body and supplies the
|
||||
operation key itself ([ADR 0026](0026-classify-client-retries-and-recover-submits-by-operation-key.md)).
|
||||
|
||||
|
||||
@@ -323,10 +323,11 @@ cost less
|
||||
|
||||
A reasoning model writes its thinking before the answer. The worker submits it in `reasoning_content`, separate
|
||||
from `content`, so a client that does not expect reasoning receives a clean answer, and a block cut
|
||||
off by `max_tokens` cannot become the reply. The front hides it unless the request sets `horde.include_reasoning`;
|
||||
the native status shows it either way. It is not priced: the charge and reward are computed on the content and tool
|
||||
calls, and the worker's `reasoning_tokens` count is shown and never charged, as with every other reported count
|
||||
([Reasoning](../reference/chat_requests.md#reasoning)).
|
||||
off by `max_tokens` cannot become the reply. The front returns it as `reasoning_content`, so a client that reads that
|
||||
field shows progress during a long reasoning phase. A request that sets `horde.include_reasoning` to false does not
|
||||
receive it. The native status shows it either way. It is not priced: the charge and reward are computed on the content
|
||||
and tool calls, and the worker's `reasoning_tokens` count is shown and never charged, as with every other reported
|
||||
count ([Reasoning](../reference/chat_requests.md#reasoning)).
|
||||
|
||||
### Admission against the online pool
|
||||
|
||||
|
||||
@@ -201,7 +201,7 @@ The `horde` object carries the Horde options OpenAI does not have a key for:
|
||||
| `context_fit` | `truncate_oldest` | `truncate_oldest` grows the context as `grow` does, then drops whole turns from the oldest forward until the messages fit, keeping every system and developer message and the latest user message with what follows it. `grow` raises the context to fit the messages, up to the largest chat context an online worker for the model declares, and refuses messages that do not fit at that bound. `reject` refuses messages that do not fit the context. |
|
||||
| `conformance` | `strict` | `strict` sends the request only to workers that follow every part of the OpenAI contract it relies on. `best_effort` also accepts workers that may deviate; see [Conformance](#conformance). |
|
||||
| `openai_guards` | `false` | `true` turns on the request refusals OpenAI applies and the Horde relaxes on purpose: a `json_object` response format whose messages never mention "json" is 400 `ChatRequestConflict`. |
|
||||
| `include_reasoning` | `false` | `true` returns the model's reasoning as `reasoning_content`; see [Reasoning](#reasoning). |
|
||||
| `include_reasoning` | `true` | `false` hides the model's reasoning, which is otherwise returned as `reasoning_content`; see [Reasoning](#reasoning). |
|
||||
| `max_kudos` | none | A positive integer cap on the request's kudos estimate. A request whose estimate at current pricing exceeds it is refused with 400 `max_kudos_exceeded` before it is queued, so it is not charged; `metadata.kudos_estimate` and `metadata.max_kudos` hold both values. The estimate assumes the full `max_tokens` is generated, while the charge is for the output estimate, so the check is conservative. |
|
||||
| `availability_mode` | `wait` | `wait` queues the request for the front's budget. `fail_fast` refuses it at once with 503 `queue_unavailable` and a `Retry-After` when the online pool has zero eligible worker threads, or when the Horde's estimated wait exceeds the budget; see [Errors and retries](#errors-and-retries). |
|
||||
|
||||
@@ -228,21 +228,22 @@ for adjustment in completion.x_horde["context_adjustments"] or []:
|
||||
|
||||
## Reasoning
|
||||
|
||||
A reasoning model, such as Qwen3, thinks before it answers. The front hides that reasoning by default, as OpenAI's
|
||||
chat API does, and the answer in `content` never includes it. To receive it, send:
|
||||
A reasoning model, such as Qwen3, thinks before it answers. The answer in `content` never includes that reasoning.
|
||||
When the worker submitted reasoning, the assistant message has it in `reasoning_content` beside `content`, as
|
||||
DeepSeek and OpenRouter return it. A stream sends it as `delta.reasoning_content` chunks before the content of the
|
||||
same worker post, so a client that reads the field shows progress during a long reasoning phase. The OpenAI SDK
|
||||
keeps the field in `model_extra`, streamed and whole. To hide the reasoning, as OpenAI's chat API does, send:
|
||||
|
||||
```json
|
||||
{"horde": {"include_reasoning": true}}
|
||||
{"horde": {"include_reasoning": false}}
|
||||
```
|
||||
|
||||
The assistant message then carries `reasoning_content` beside `content`, as DeepSeek and OpenRouter return it, when
|
||||
the worker submitted reasoning. A stream sends it as `delta.reasoning_content` chunks before the content. Whether or
|
||||
not you ask for it, `usage.completion_tokens_details.reasoning_tokens` reports how many of the completion tokens
|
||||
were reasoning when the worker counted them. The option does not change the work, its cost or the operation the
|
||||
request is for.
|
||||
Either way, `usage.completion_tokens_details.reasoning_tokens` reports how many of the completion tokens were
|
||||
reasoning when the worker counted them. The option does not change the work, its cost or the operation the request
|
||||
is for.
|
||||
|
||||
Verification: a request to a reasoning model with `include_reasoning: true` returns `choices[0].message.reasoning_content`;
|
||||
the same request without it returns a message without a `reasoning_content` key.
|
||||
Verification: a request to a reasoning model returns `choices[0].message.reasoning_content`; the same request with
|
||||
`include_reasoning: false` returns a message without a `reasoning_content` key.
|
||||
|
||||
## Images
|
||||
|
||||
@@ -351,8 +352,8 @@ With `"stream": true` the response is server-sent events in this order:
|
||||
1. Keepalive comment lines while the request waits, such as `: horde queue_position=7 wait_time=42`, at an interval
|
||||
the deployment sets (5 seconds by default).
|
||||
2. A role chunk, `delta: {"role": "assistant"}`.
|
||||
3. A reasoning chunk with the whole `delta.reasoning_content`, when you sent `horde.include_reasoning` and the model
|
||||
reasoned.
|
||||
3. A reasoning chunk with the whole `delta.reasoning_content`, when the model reasoned, unless you sent
|
||||
`horde.include_reasoning: false`.
|
||||
4. A content chunk with the whole `delta.content`, when the reply has text.
|
||||
5. A tool-call chunk with `delta.tool_calls`, when the reply has calls: every call whole, each with its 0-based
|
||||
`index`, `id`, `type` and `function`, and no content in the same chunk.
|
||||
@@ -361,8 +362,8 @@ With `"stream": true` the response is server-sent events in this order:
|
||||
8. `data: [DONE]`.
|
||||
|
||||
A request without `tools` and with no `response_format` other than `text` gets its content live when the worker
|
||||
supports it: after the role chunk, one content chunk per batch the worker posts, about one a second, with reasoning
|
||||
chunks before them under `horde.include_reasoning`, then only the text those chunks did not cover before the finish
|
||||
supports it: after the role chunk, one content chunk per batch the worker posts, about one a second, with a batch's reasoning
|
||||
chunks before its content unless `horde.include_reasoning` is false, then only the text those chunks did not cover before the finish
|
||||
chunk. Otherwise, as with tools or structured output, the worker generates the whole reply before the Horde reports
|
||||
it, and content arrives in one chunk. Every chunk carries the same `id`; the finish and usage chunks carry the served
|
||||
`model`, and every chunk carries the same `system_fingerprint`. An SDK that accumulates tool-call deltas by `index`
|
||||
@@ -510,7 +511,8 @@ The front differs from OpenAI on purpose in these points:
|
||||
- Live content arrives in batches of about a second, and only for requests without tools or structured output.
|
||||
- A `json_object` request whose messages never mention JSON is served, where OpenAI refuses it, unless
|
||||
`horde.openai_guards` is set.
|
||||
- Reasoning text is returned as `reasoning_content` under `horde.include_reasoning`, where OpenAI never returns it.
|
||||
- Reasoning text is returned as `reasoning_content`, where OpenAI never returns it. The OpenAI SDK keeps the field in
|
||||
`model_extra`, and `horde.include_reasoning: false` hides it.
|
||||
|
||||
## Reversal
|
||||
|
||||
|
||||
@@ -130,8 +130,8 @@ calls. `x_horde.context_adjustments` is the native status's list, passed through
|
||||
`worker_max_length` entry when `max_tokens` was lowered to the largest value declared by an online worker. It holds a
|
||||
`grow` entry when the context grew, and a `truncate_oldest` entry with `dropped_messages` when the oldest turns were
|
||||
dropped (see [Context adjustments](chat_requests.md#context-adjustments)). It is left out when the Horde did not make
|
||||
any of these changes. `reasoning_content`, the model's reasoning as DeepSeek and OpenRouter return it, is written only when the
|
||||
request sets `horde.include_reasoning` and the worker submitted reasoning; it is never part of `content`.
|
||||
any of these changes. `reasoning_content`, the model's reasoning as DeepSeek and OpenRouter return it, is written when the
|
||||
worker submitted reasoning, unless the request sets `horde.include_reasoning` to false. It is never part of `content`.
|
||||
`usage.completion_tokens_details.reasoning_tokens`, OpenAI's shape, is the reasoning count the worker reported,
|
||||
written whether or not the reasoning is returned and left out when no worker reported one. Like the other reported
|
||||
counts it is informational: the Horde charges and rewards on the reply it measures. The response `model` is the first
|
||||
@@ -147,8 +147,8 @@ A stream's headers go out once the first status read has passed the checks in
|
||||
events:
|
||||
|
||||
1. `delta: {"role": "assistant"}`.
|
||||
2. A reasoning chunk with the whole `delta.reasoning_content`, when the message carries it (only under
|
||||
`horde.include_reasoning`).
|
||||
2. A reasoning chunk with the whole `delta.reasoning_content`, when the message has it (left out when
|
||||
`horde.include_reasoning` is false).
|
||||
3. A content chunk with the whole `delta.content`, when the reply has text.
|
||||
4. A tool-call chunk with `delta.tool_calls`, when the reply has calls, each call whole with its 0-based `index`,
|
||||
`id`, `type` and `function`. It does not carry content, because some clients read tool calls only from such chunks.
|
||||
@@ -178,10 +178,10 @@ the request is delivered whole when the front has not sent anything, and otherwi
|
||||
stream through the reconciliation below.
|
||||
|
||||
The first delta with `seq` 1 pins its `gen`, and the first delta that sends anything sends the role chunk and exposes
|
||||
the stream. A delta then sends a chunk with `delta.reasoning_content` when it carries reasoning and the request sets
|
||||
`horde.include_reasoning`, and a chunk with `delta.content` when it carries text; each chunk carries the `model` the
|
||||
the stream. A delta then sends a chunk with `delta.reasoning_content` when it has reasoning, unless the request sets
|
||||
`horde.include_reasoning` to false, and a chunk with `delta.content` when it carries text; each chunk carries the `model` the
|
||||
first delta's entry carries and the `system_fingerprint` of that entry's `worker` and `model` (the request's model, or
|
||||
its first listed model, when the entry omits it). Without `horde.include_reasoning` the reasoning is dropped, and a delta carrying only
|
||||
its first listed model, when the entry omits it). With `horde.include_reasoning` set to false the reasoning is dropped, and a delta carrying only
|
||||
reasoning advances the sequence without sending anything, so the text after a reasoning block is still relayed.
|
||||
Keepalive comments continue between deltas.
|
||||
|
||||
@@ -190,7 +190,9 @@ split into pieces, and the pieces are sent evenly over a window. The window is t
|
||||
generation's posts. The estimate is an exponential moving average of the gaps between the posts' stream id times,
|
||||
with a weight of 0.5 on the newest gap. It is kept between 0.05 seconds and
|
||||
`HORDE_OAI_DELTA_PACING_MAX_WINDOW_SECONDS`. A window holds at most `HORDE_OAI_DELTA_PACING_MAX_CHUNKS_PER_SECOND`
|
||||
pieces per second, and the piece count does not exceed the character count.
|
||||
pieces per second, and the piece count does not exceed the character count. Within one delta, every reasoning piece
|
||||
is sent before the first content piece. The piece count is divided between reasoning and content by length, with at
|
||||
least one piece for each field that has text.
|
||||
|
||||
Each cut is at whitespace within half a piece of an even offset. Without such whitespace, the cut is at the next offset
|
||||
that does not split a visible character. A cut does not separate a character from a following Unicode mark (such as
|
||||
@@ -240,7 +242,7 @@ returns 502 `ChatRequestNoGeneration`. The front does not return a completion wi
|
||||
| First streamed chunk | `delta` holds only `role` | `delta` also holds `content` |
|
||||
| Delta granularity | For plain text requests, each worker progress post (about one a second) is paced over the interval until the next post, in up to 20 chunks a second. Other requests get the whole text at once. | One chunk per token or few tokens |
|
||||
| `json_object` without "json" in the messages | Served, unless `horde.openai_guards` is set | Refused |
|
||||
| Reasoning text | Hidden; `horde.include_reasoning` returns it as `message.reasoning_content` and `delta.reasoning_content` | Not returned; `usage.completion_tokens_details.reasoning_tokens` only |
|
||||
| Reasoning text | Returned as `message.reasoning_content` and `delta.reasoning_content`, which the OpenAI SDK keeps in `model_extra`; `horde.include_reasoning: false` hides it | Not returned; `usage.completion_tokens_details.reasoning_tokens` only |
|
||||
|
||||
## No eligible worker
|
||||
|
||||
|
||||
@@ -239,7 +239,7 @@ and values:
|
||||
| `context_fit` | `grow`, `reject` or `truncate_oldest` | `truncate_oldest` | With `grow`, the context grows only up to the largest chat context an online chat worker serving the model declares. With `truncate_oldest`, the context grows as under `grow`; when the messages do not fit even at that bound, whole turns are dropped from the oldest forward until they do. See [Context adjustments](#context-adjustments) |
|
||||
| `conformance` | `strict` or `best_effort` | `strict` | With `strict`, only workers declaring every guarantee the request needs serve it; `best_effort` drops that requirement. See [Guarantees](#guarantees) |
|
||||
| `openai_guards` | boolean | `false` | Turns on the request refusals OpenAI applies and the Horde relaxes on purpose: a `json_object` response format without the word "json" (any case) in some message text returns `400 ChatRequestConflict` with OpenAI's message. Acts on admission only; not stored |
|
||||
| `include_reasoning` | boolean | `false` | Read by the OpenAI front, which returns the model's reasoning only when it is set. The native route accepts it and does not store it; it is not in the operation fingerprint, and the native status shows a submitted `reasoning_content` either way |
|
||||
| `include_reasoning` | boolean | `true` | Read by the OpenAI front, which returns the model's reasoning unless it is false. The native route accepts it and does not store it; it is not in the operation fingerprint, and the native status shows a submitted `reasoning_content` either way |
|
||||
| `availability_mode` | `wait` or `fail_fast` | `wait` | Read by the OpenAI front. With `wait` the request queues for the front's budget; with `fail_fast` the front refuses it at the first status read when the status reports zero `eligible_worker_threads` and nothing processing, or a `wait_time` over the time left in the budget. The native route accepts it and does not store it; it is not in the operation fingerprint |
|
||||
| `max_kudos` | positive integer | none | A cap on the request's kudos estimate, the quote a dry run returns. When the estimate exceeds it, the route deletes the inactive row before any image upload or kudos hold and returns `400 ChatKudosCapExceeded`, whose message states the estimate at current pricing and the cap; a dry run is refused the same way. The estimate assumes the full `max_length`, while the charge is the output estimate ([ADR 0027](../decisions/0027-price-chat-generations-by-the-text-output-estimate.md)), so the check is conservative. Not stored; not in the operation fingerprint |
|
||||
| `dry_run` | boolean | `false` | Return the kudos cost without queueing |
|
||||
|
||||
@@ -288,11 +288,12 @@ class ChatHordeOptions(BaseModel):
|
||||
charge is the output estimate, so the check is conservative. It acts only at creation, so it is not stored and the
|
||||
operation fingerprint leaves it out."""
|
||||
|
||||
include_reasoning: StrictBool = False
|
||||
include_reasoning: StrictBool = True
|
||||
"""Return the model's reasoning as ``reasoning_content`` beside the reply. OpenAI's chat API does not return
|
||||
reasoning text, so the OpenAI front hides it unless asked for. It changes what the front delivers and leaves the
|
||||
work unchanged, so it is not stored and the operation fingerprint leaves it out; the native status shows the
|
||||
reasoning a worker submitted regardless."""
|
||||
reasoning text, but clients that read the DeepSeek ``reasoning_content`` field render it live, and the OpenAI SDK
|
||||
keeps the unknown field in ``model_extra``. Without it a client sees nothing during a long reasoning phase. False
|
||||
hides it. It changes what the front delivers and leaves the work unchanged, so it is not stored and the operation
|
||||
fingerprint leaves it out. The native status shows the reasoning a worker submitted regardless."""
|
||||
|
||||
dry_run: StrictBool = False
|
||||
"""Return the kudos cost without queueing; native callers only."""
|
||||
|
||||
+27
-12
@@ -2629,21 +2629,36 @@ class DeltaPacer:
|
||||
def _spread(self, *, content: str, reasoning_content: str, window_seconds: float, now: float) -> None:
|
||||
"""Mutate the schedule by splitting the pending text into even pieces across ``window_seconds`` from ``now``.
|
||||
|
||||
The first piece is due at ``now``, so a merge does not delay text that was already due.
|
||||
The first piece is due at ``now``, so a merge does not delay text that was already due. Every reasoning piece is
|
||||
scheduled before the first content piece, so a client that renders chunks in arrival order shows all of a post's
|
||||
reasoning before the reply text in the same post. The piece budget is divided between the two fields by length,
|
||||
with at least one piece for each field that has text.
|
||||
"""
|
||||
rate_piece_limit = max(1, math.floor(self._max_chunks_per_second * window_seconds))
|
||||
piece_count = max(1, min(rate_piece_limit, max(len(content), len(reasoning_content))))
|
||||
content_pieces = _split_evenly(content, piece_count)
|
||||
reasoning_pieces = _split_evenly(reasoning_content, piece_count)
|
||||
piece_spacing_seconds = window_seconds / piece_count
|
||||
total_length = len(content) + len(reasoning_content)
|
||||
piece_count = max(1, min(rate_piece_limit, total_length))
|
||||
reasoning_piece_count = 0
|
||||
if reasoning_content:
|
||||
reasoning_piece_count = max(1, round(piece_count * len(reasoning_content) / total_length))
|
||||
if content:
|
||||
reasoning_piece_count = min(reasoning_piece_count, piece_count - 1) if piece_count > 1 else 1
|
||||
content_piece_count = max(1, piece_count - reasoning_piece_count) if content else 0
|
||||
piece_texts: list[DeltaText] = []
|
||||
if reasoning_piece_count:
|
||||
piece_texts += [DeltaText(reasoning_content=piece) for piece in _split_evenly(reasoning_content, reasoning_piece_count)]
|
||||
if content_piece_count:
|
||||
piece_texts += [DeltaText(content=piece) for piece in _split_evenly(content, content_piece_count)]
|
||||
if not piece_texts:
|
||||
piece_texts = [DeltaText()]
|
||||
piece_spacing_seconds = window_seconds / len(piece_texts)
|
||||
# The new pieces go after any pieces still in the schedule, such as a first post read in the same batch.
|
||||
self._pieces += [
|
||||
PacedPiece(
|
||||
content=content_piece,
|
||||
reasoning_content=reasoning_piece,
|
||||
content=piece_text.content,
|
||||
reasoning_content=piece_text.reasoning_content,
|
||||
due_at=now + piece_index * piece_spacing_seconds,
|
||||
)
|
||||
for piece_index, (content_piece, reasoning_piece) in enumerate(zip(content_pieces, reasoning_pieces, strict=True))
|
||||
for piece_index, piece_text in enumerate(piece_texts)
|
||||
]
|
||||
|
||||
|
||||
@@ -2778,8 +2793,8 @@ class DeltaRelay:
|
||||
never spliced into a choice the client already holds part of (ADR 0024). Before exposure such entries unpin the
|
||||
generation, and the next generation's first delta can pin it again, or the request is delivered whole.
|
||||
|
||||
A delta's reasoning is relayed only when the client asked for it with ``horde.include_reasoning``; otherwise it is
|
||||
dropped, and a delta carrying only reasoning advances the sequence without sending anything. Reasoning is never
|
||||
A delta's reasoning is relayed unless the client turned it off with ``horde.include_reasoning: false``. When off it
|
||||
is dropped, and a delta carrying only reasoning advances the sequence without sending anything. Reasoning is never
|
||||
part of the reconciliation with the final reply.
|
||||
"""
|
||||
|
||||
@@ -2792,7 +2807,7 @@ class DeltaRelay:
|
||||
model: The model the client asked for, the first listed for ``models``, which the relayed chunks carry and
|
||||
the fingerprint uses when the pinned generation's first delta omits the model that serves it.
|
||||
include_usage: Whether the client asked for usage, which writes a null ``usage`` in every chunk.
|
||||
include_reasoning: Whether the client asked for reasoning with ``horde.include_reasoning``.
|
||||
include_reasoning: Whether reasoning is relayed, from ``horde.include_reasoning`` (true when absent).
|
||||
"""
|
||||
self.generation_id: str | None = None
|
||||
"""The generation whose deltas are relayed; None before its first delta."""
|
||||
@@ -3132,7 +3147,7 @@ async def _stream_completion(
|
||||
created=int(submitted_at),
|
||||
model=chat_request.requested_models()[0],
|
||||
include_usage=include_usage,
|
||||
include_reasoning=chat_request.horde is not None and chat_request.horde.include_reasoning,
|
||||
include_reasoning=chat_request.horde is None or chat_request.horde.include_reasoning,
|
||||
)
|
||||
wait_task = asyncio.create_task(watcher.wait_until_finished())
|
||||
async for relay_event in _relay_deltas(
|
||||
|
||||
@@ -404,9 +404,9 @@ def build_completion(
|
||||
|
||||
Returns:
|
||||
One choice per generation, in the Horde's order; ``model``, ``system_fingerprint`` and the ``x_horde`` worker
|
||||
details are the first generation's. A message carries ``reasoning_content`` only under
|
||||
``horde.include_reasoning``. ``usage.completion_tokens_details.reasoning_tokens`` is the sum of the reasoning
|
||||
counts the workers reported, present when any reported one, whether or not the reasoning is returned.
|
||||
details are the first generation's. A message has ``reasoning_content`` unless
|
||||
``horde.include_reasoning`` is false. ``usage.completion_tokens_details.reasoning_tokens`` is the sum of the
|
||||
reasoning counts the workers reported, present when any reported one, whether or not the reasoning is returned.
|
||||
|
||||
Raises:
|
||||
CompletionWithoutGenerationError: ``status`` has no finished generation. A completion built from it would be a
|
||||
@@ -420,7 +420,7 @@ def build_completion(
|
||||
prompt_tokens: int | None = None
|
||||
reasoning_tokens: int | None = None
|
||||
has_empty_return = False
|
||||
include_reasoning = chat_request.horde is not None and chat_request.horde.include_reasoning
|
||||
include_reasoning = chat_request.horde is None or chat_request.horde.include_reasoning
|
||||
for generation_index, generation in enumerate(status.generations):
|
||||
generated_text = generation_content(generation)
|
||||
tool_calls = generation_tool_calls(generation)
|
||||
|
||||
+2
-2
@@ -5,8 +5,8 @@
|
||||
"""Render a streamed chat answer as server-sent events.
|
||||
|
||||
A buffered stream sends keepalive comments while the request waits (``: horde queue_position=7 wait_time=42``), then
|
||||
the finished completion as a role chunk, a reasoning chunk when the message carries ``reasoning_content`` (only under
|
||||
``horde.include_reasoning``), a content chunk when the reply has text, a tool-call chunk when it has calls,
|
||||
the finished completion as a role chunk, a reasoning chunk when the message carries ``reasoning_content`` (omitted when
|
||||
``horde.include_reasoning`` is false), a content chunk when the reply has text, a tool-call chunk when it has calls,
|
||||
a finish chunk carrying ``x_horde``, the usage chunk when the client asked for it, and ``[DONE]``. A relayed stream
|
||||
sends the role chunk and, per delta, a reasoning chunk when it relays reasoning and a content chunk when the delta
|
||||
has text, while the worker generates ([`build_delta_events`][horde.oai.sse.build_delta_events]); its finished
|
||||
|
||||
@@ -688,22 +688,21 @@ REASONING_TEXT: str = "The user wants a greeting."
|
||||
"""The reasoning a worker submitted beside the reply."""
|
||||
|
||||
|
||||
@pytest.mark.parametrize("include_reasoning", [True, False])
|
||||
async def test_reasoning_is_returned_on_the_message_only_when_the_client_asks_for_it(
|
||||
@pytest.mark.parametrize("include_reasoning", [True, False, None])
|
||||
async def test_reasoning_is_returned_on_the_message_unless_the_client_turns_it_off(
|
||||
start_front: StartFront,
|
||||
include_reasoning: bool,
|
||||
include_reasoning: bool | None,
|
||||
) -> None:
|
||||
"""OpenAI's chat API does not return reasoning, so the front hides it unless ``horde.include_reasoning`` is set."""
|
||||
"""Reasoning is returned by default, including to a client that does not send a ``horde`` block, and
|
||||
``horde.include_reasoning: false`` hides it."""
|
||||
chat_body = build_chat_body() if include_reasoning is None else build_chat_body(horde={"include_reasoning": include_reasoning})
|
||||
async with start_front() as front:
|
||||
front.horde.status_payload = done_status(reasoning_content=REASONING_TEXT)
|
||||
response = await front.client.post(
|
||||
"/v1/chat/completions",
|
||||
json=build_chat_body(horde={"include_reasoning": include_reasoning}),
|
||||
)
|
||||
response = await front.client.post("/v1/chat/completions", json=chat_body)
|
||||
|
||||
assert response.status_code == 200
|
||||
expected_message: dict[str, object] = {"role": "assistant", "content": GENERATED_TEXT}
|
||||
if include_reasoning:
|
||||
if include_reasoning is not False:
|
||||
expected_message["reasoning_content"] = REASONING_TEXT
|
||||
assert response.json()["choices"][0]["message"] == expected_message
|
||||
|
||||
|
||||
@@ -612,7 +612,7 @@ async def test_reasoning_deltas_are_relayed_before_the_text_when_the_client_asks
|
||||
assert equal_outcomes == 1
|
||||
|
||||
|
||||
async def test_reasoning_deltas_are_dropped_and_the_text_still_relayed_when_the_client_did_not_ask(start_front: StartFront) -> None:
|
||||
async def test_reasoning_deltas_are_dropped_and_the_text_still_relayed_when_the_client_turns_reasoning_off(start_front: StartFront) -> None:
|
||||
"""A delta carrying only reasoning advances the sequence without sending anything, so the text that follows it is
|
||||
relayed as the stream's first delta."""
|
||||
async with start_front(with_deltas=True) as front:
|
||||
@@ -624,7 +624,7 @@ async def test_reasoning_deltas_are_dropped_and_the_text_still_relayed_when_the_
|
||||
await asyncio.sleep(RELAY_SETTLE_SECONDS)
|
||||
await finish(front, reasoning_content=REASONING_TEXT)
|
||||
|
||||
body = await stream_while(front, generate)
|
||||
body = await stream_while(front, generate, horde={"include_reasoning": False})
|
||||
relayed = front.metric_total(RELAYED_NAME)
|
||||
prefix_outcomes = front.metric_total(RELAY_OUTCOMES_NAME, horde_outcome="prefix")
|
||||
|
||||
@@ -709,7 +709,7 @@ async def test_stream_lost_before_any_delta_was_sent_delivers_the_request_whole(
|
||||
await asyncio.sleep(RELAY_SETTLE_SECONDS)
|
||||
await finish(front)
|
||||
|
||||
body = await stream_while(front, generate)
|
||||
body = await stream_while(front, generate, horde={"include_reasoning": False})
|
||||
relayed = front.metric_total(RELAYED_NAME)
|
||||
relay_outcomes = front.metric_total(RELAY_OUTCOMES_NAME)
|
||||
cancels = front.horde.cancels_received
|
||||
@@ -919,7 +919,7 @@ async def test_stream_replaced_with_lower_ids_before_any_delta_was_sent_delivers
|
||||
await asyncio.sleep(RELAY_SETTLE_SECONDS)
|
||||
await finish(front)
|
||||
|
||||
body = await stream_while(front, generate)
|
||||
body = await stream_while(front, generate, horde={"include_reasoning": False})
|
||||
relayed = front.metric_total(RELAYED_NAME)
|
||||
relay_outcomes = front.metric_total(RELAY_OUTCOMES_NAME)
|
||||
cancels = front.horde.cancels_received
|
||||
@@ -1098,15 +1098,23 @@ async def test_a_maximum_window_of_zero_sends_each_post_whole_when_it_arrives()
|
||||
assert [(due_at, piece.content) for due_at, piece in schedule] == [(3.0, "Hello "), (3.0, PACED_TEXT), (4.0, "more")]
|
||||
|
||||
|
||||
async def test_reasoning_is_paced_with_the_text() -> None:
|
||||
pacer = DeltaPacer(max_window_seconds=2.0, max_chunks_per_second=3)
|
||||
async def test_a_post_with_reasoning_and_text_paces_all_the_reasoning_before_the_text() -> None:
|
||||
"""A client that renders in arrival order shows the end of the reasoning, then the start of the reply, without
|
||||
alternating between them. The piece budget is divided between the two by length."""
|
||||
pacer = DeltaPacer(max_window_seconds=2.0, max_chunks_per_second=6)
|
||||
pacer.add([accepted("Hello ", 100.0), accepted("abcdef", 101.0, reasoning_content="thinking")], now=0.0)
|
||||
|
||||
schedule = drain_schedule(pacer)
|
||||
paced_pieces = [piece for _due_at, piece in schedule][1:]
|
||||
|
||||
assert piece_contents(paced_pieces) == ["ab", "cd", "ef"]
|
||||
assert [piece.reasoning_content for piece in paced_pieces] == ["th", "ink", "ing"]
|
||||
assert [(piece.reasoning_content, piece.content) for piece in paced_pieces] == [
|
||||
("th", ""),
|
||||
("ink", ""),
|
||||
("ing", ""),
|
||||
("", "ab"),
|
||||
("", "cd"),
|
||||
("", "ef"),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
|
||||
Reference in new issue
Block a user