diff --git a/docs/decisions/0020-select-chat-service-by-request-body-and-model-variants.md b/docs/decisions/0020-select-chat-service-by-request-body-and-model-variants.md index 2b3fca1c..2583b875 100644 --- a/docs/decisions/0020-select-chat-service-by-request-body-and-model-variants.md +++ b/docs/decisions/0020-select-chat-service-by-request-body-and-model-variants.md @@ -52,7 +52,7 @@ defines it for both the front and the native route, and an unknown key is refuse - `openai_guards`: turns on the request refusals OpenAI applies and the Horde relaxes. - `availability_mode`: wait for a worker, or refuse at once when the online pool cannot serve the request. - `max_kudos`: the most the request may cost at creation. -- `include_reasoning`: return the model's reasoning beside the reply. +- `include_reasoning`: return the model's reasoning beside the reply, on by default. - `dry_run` and `operation_key`: native callers only. The front refuses both in a public body and supplies the operation key itself ([ADR 0026](0026-classify-client-retries-and-recover-submits-by-operation-key.md)). diff --git a/docs/explanation/chat_requests.md b/docs/explanation/chat_requests.md index 240d0640..c85799f7 100644 --- a/docs/explanation/chat_requests.md +++ b/docs/explanation/chat_requests.md @@ -323,10 +323,11 @@ cost less A reasoning model writes its thinking before the answer. The worker submits it in `reasoning_content`, separate from `content`, so a client that does not expect reasoning receives a clean answer, and a block cut -off by `max_tokens` cannot become the reply. The front hides it unless the request sets `horde.include_reasoning`; -the native status shows it either way. It is not priced: the charge and reward are computed on the content and tool -calls, and the worker's `reasoning_tokens` count is shown and never charged, as with every other reported count -([Reasoning](../reference/chat_requests.md#reasoning)). +off by `max_tokens` cannot become the reply. The front returns it as `reasoning_content`, so a client that reads that +field shows progress during a long reasoning phase. A request that sets `horde.include_reasoning` to false does not +receive it. The native status shows it either way. It is not priced: the charge and reward are computed on the content +and tool calls, and the worker's `reasoning_tokens` count is shown and never charged, as with every other reported +count ([Reasoning](../reference/chat_requests.md#reasoning)). ### Admission against the online pool diff --git a/docs/how-to/send_chat_completions.md b/docs/how-to/send_chat_completions.md index c0ded5cc..2e8d5064 100644 --- a/docs/how-to/send_chat_completions.md +++ b/docs/how-to/send_chat_completions.md @@ -201,7 +201,7 @@ The `horde` object carries the Horde options OpenAI does not have a key for: | `context_fit` | `truncate_oldest` | `truncate_oldest` grows the context as `grow` does, then drops whole turns from the oldest forward until the messages fit, keeping every system and developer message and the latest user message with what follows it. `grow` raises the context to fit the messages, up to the largest chat context an online worker for the model declares, and refuses messages that do not fit at that bound. `reject` refuses messages that do not fit the context. | | `conformance` | `strict` | `strict` sends the request only to workers that follow every part of the OpenAI contract it relies on. `best_effort` also accepts workers that may deviate; see [Conformance](#conformance). | | `openai_guards` | `false` | `true` turns on the request refusals OpenAI applies and the Horde relaxes on purpose: a `json_object` response format whose messages never mention "json" is 400 `ChatRequestConflict`. | -| `include_reasoning` | `false` | `true` returns the model's reasoning as `reasoning_content`; see [Reasoning](#reasoning). | +| `include_reasoning` | `true` | `false` hides the model's reasoning, which is otherwise returned as `reasoning_content`; see [Reasoning](#reasoning). | | `max_kudos` | none | A positive integer cap on the request's kudos estimate. A request whose estimate at current pricing exceeds it is refused with 400 `max_kudos_exceeded` before it is queued, so it is not charged; `metadata.kudos_estimate` and `metadata.max_kudos` hold both values. The estimate assumes the full `max_tokens` is generated, while the charge is for the output estimate, so the check is conservative. | | `availability_mode` | `wait` | `wait` queues the request for the front's budget. `fail_fast` refuses it at once with 503 `queue_unavailable` and a `Retry-After` when the online pool has zero eligible worker threads, or when the Horde's estimated wait exceeds the budget; see [Errors and retries](#errors-and-retries). | @@ -228,21 +228,22 @@ for adjustment in completion.x_horde["context_adjustments"] or []: ## Reasoning -A reasoning model, such as Qwen3, thinks before it answers. The front hides that reasoning by default, as OpenAI's -chat API does, and the answer in `content` never includes it. To receive it, send: +A reasoning model, such as Qwen3, thinks before it answers. The answer in `content` never includes that reasoning. +When the worker submitted reasoning, the assistant message has it in `reasoning_content` beside `content`, as +DeepSeek and OpenRouter return it. A stream sends it as `delta.reasoning_content` chunks before the content of the +same worker post, so a client that reads the field shows progress during a long reasoning phase. The OpenAI SDK +keeps the field in `model_extra`, streamed and whole. To hide the reasoning, as OpenAI's chat API does, send: ```json -{"horde": {"include_reasoning": true}} +{"horde": {"include_reasoning": false}} ``` -The assistant message then carries `reasoning_content` beside `content`, as DeepSeek and OpenRouter return it, when -the worker submitted reasoning. A stream sends it as `delta.reasoning_content` chunks before the content. Whether or -not you ask for it, `usage.completion_tokens_details.reasoning_tokens` reports how many of the completion tokens -were reasoning when the worker counted them. The option does not change the work, its cost or the operation the -request is for. +Either way, `usage.completion_tokens_details.reasoning_tokens` reports how many of the completion tokens were +reasoning when the worker counted them. The option does not change the work, its cost or the operation the request +is for. -Verification: a request to a reasoning model with `include_reasoning: true` returns `choices[0].message.reasoning_content`; -the same request without it returns a message without a `reasoning_content` key. +Verification: a request to a reasoning model returns `choices[0].message.reasoning_content`; the same request with +`include_reasoning: false` returns a message without a `reasoning_content` key. ## Images @@ -351,8 +352,8 @@ With `"stream": true` the response is server-sent events in this order: 1. Keepalive comment lines while the request waits, such as `: horde queue_position=7 wait_time=42`, at an interval the deployment sets (5 seconds by default). 2. A role chunk, `delta: {"role": "assistant"}`. -3. A reasoning chunk with the whole `delta.reasoning_content`, when you sent `horde.include_reasoning` and the model - reasoned. +3. A reasoning chunk with the whole `delta.reasoning_content`, when the model reasoned, unless you sent + `horde.include_reasoning: false`. 4. A content chunk with the whole `delta.content`, when the reply has text. 5. A tool-call chunk with `delta.tool_calls`, when the reply has calls: every call whole, each with its 0-based `index`, `id`, `type` and `function`, and no content in the same chunk. @@ -361,8 +362,8 @@ With `"stream": true` the response is server-sent events in this order: 8. `data: [DONE]`. A request without `tools` and with no `response_format` other than `text` gets its content live when the worker -supports it: after the role chunk, one content chunk per batch the worker posts, about one a second, with reasoning -chunks before them under `horde.include_reasoning`, then only the text those chunks did not cover before the finish +supports it: after the role chunk, one content chunk per batch the worker posts, about one a second, with a batch's reasoning +chunks before its content unless `horde.include_reasoning` is false, then only the text those chunks did not cover before the finish chunk. Otherwise, as with tools or structured output, the worker generates the whole reply before the Horde reports it, and content arrives in one chunk. Every chunk carries the same `id`; the finish and usage chunks carry the served `model`, and every chunk carries the same `system_fingerprint`. An SDK that accumulates tool-call deltas by `index` @@ -510,7 +511,8 @@ The front differs from OpenAI on purpose in these points: - Live content arrives in batches of about a second, and only for requests without tools or structured output. - A `json_object` request whose messages never mention JSON is served, where OpenAI refuses it, unless `horde.openai_guards` is set. -- Reasoning text is returned as `reasoning_content` under `horde.include_reasoning`, where OpenAI never returns it. +- Reasoning text is returned as `reasoning_content`, where OpenAI never returns it. The OpenAI SDK keeps the field in + `model_extra`, and `horde.include_reasoning: false` hides it. ## Reversal diff --git a/docs/reference/chat_front.md b/docs/reference/chat_front.md index 10a2838f..88173428 100644 --- a/docs/reference/chat_front.md +++ b/docs/reference/chat_front.md @@ -130,8 +130,8 @@ calls. `x_horde.context_adjustments` is the native status's list, passed through `worker_max_length` entry when `max_tokens` was lowered to the largest value declared by an online worker. It holds a `grow` entry when the context grew, and a `truncate_oldest` entry with `dropped_messages` when the oldest turns were dropped (see [Context adjustments](chat_requests.md#context-adjustments)). It is left out when the Horde did not make -any of these changes. `reasoning_content`, the model's reasoning as DeepSeek and OpenRouter return it, is written only when the -request sets `horde.include_reasoning` and the worker submitted reasoning; it is never part of `content`. +any of these changes. `reasoning_content`, the model's reasoning as DeepSeek and OpenRouter return it, is written when the +worker submitted reasoning, unless the request sets `horde.include_reasoning` to false. It is never part of `content`. `usage.completion_tokens_details.reasoning_tokens`, OpenAI's shape, is the reasoning count the worker reported, written whether or not the reasoning is returned and left out when no worker reported one. Like the other reported counts it is informational: the Horde charges and rewards on the reply it measures. The response `model` is the first @@ -147,8 +147,8 @@ A stream's headers go out once the first status read has passed the checks in events: 1. `delta: {"role": "assistant"}`. -2. A reasoning chunk with the whole `delta.reasoning_content`, when the message carries it (only under - `horde.include_reasoning`). +2. A reasoning chunk with the whole `delta.reasoning_content`, when the message has it (left out when + `horde.include_reasoning` is false). 3. A content chunk with the whole `delta.content`, when the reply has text. 4. A tool-call chunk with `delta.tool_calls`, when the reply has calls, each call whole with its 0-based `index`, `id`, `type` and `function`. It does not carry content, because some clients read tool calls only from such chunks. @@ -178,10 +178,10 @@ the request is delivered whole when the front has not sent anything, and otherwi stream through the reconciliation below. The first delta with `seq` 1 pins its `gen`, and the first delta that sends anything sends the role chunk and exposes -the stream. A delta then sends a chunk with `delta.reasoning_content` when it carries reasoning and the request sets -`horde.include_reasoning`, and a chunk with `delta.content` when it carries text; each chunk carries the `model` the +the stream. A delta then sends a chunk with `delta.reasoning_content` when it has reasoning, unless the request sets +`horde.include_reasoning` to false, and a chunk with `delta.content` when it carries text; each chunk carries the `model` the first delta's entry carries and the `system_fingerprint` of that entry's `worker` and `model` (the request's model, or -its first listed model, when the entry omits it). Without `horde.include_reasoning` the reasoning is dropped, and a delta carrying only +its first listed model, when the entry omits it). With `horde.include_reasoning` set to false the reasoning is dropped, and a delta carrying only reasoning advances the sequence without sending anything, so the text after a reasoning block is still relayed. Keepalive comments continue between deltas. @@ -190,7 +190,9 @@ split into pieces, and the pieces are sent evenly over a window. The window is t generation's posts. The estimate is an exponential moving average of the gaps between the posts' stream id times, with a weight of 0.5 on the newest gap. It is kept between 0.05 seconds and `HORDE_OAI_DELTA_PACING_MAX_WINDOW_SECONDS`. A window holds at most `HORDE_OAI_DELTA_PACING_MAX_CHUNKS_PER_SECOND` -pieces per second, and the piece count does not exceed the character count. +pieces per second, and the piece count does not exceed the character count. Within one delta, every reasoning piece +is sent before the first content piece. The piece count is divided between reasoning and content by length, with at +least one piece for each field that has text. Each cut is at whitespace within half a piece of an even offset. Without such whitespace, the cut is at the next offset that does not split a visible character. A cut does not separate a character from a following Unicode mark (such as @@ -240,7 +242,7 @@ returns 502 `ChatRequestNoGeneration`. The front does not return a completion wi | First streamed chunk | `delta` holds only `role` | `delta` also holds `content` | | Delta granularity | For plain text requests, each worker progress post (about one a second) is paced over the interval until the next post, in up to 20 chunks a second. Other requests get the whole text at once. | One chunk per token or few tokens | | `json_object` without "json" in the messages | Served, unless `horde.openai_guards` is set | Refused | -| Reasoning text | Hidden; `horde.include_reasoning` returns it as `message.reasoning_content` and `delta.reasoning_content` | Not returned; `usage.completion_tokens_details.reasoning_tokens` only | +| Reasoning text | Returned as `message.reasoning_content` and `delta.reasoning_content`, which the OpenAI SDK keeps in `model_extra`; `horde.include_reasoning: false` hides it | Not returned; `usage.completion_tokens_details.reasoning_tokens` only | ## No eligible worker diff --git a/docs/reference/chat_requests.md b/docs/reference/chat_requests.md index ca7beaaa..27572b96 100644 --- a/docs/reference/chat_requests.md +++ b/docs/reference/chat_requests.md @@ -239,7 +239,7 @@ and values: | `context_fit` | `grow`, `reject` or `truncate_oldest` | `truncate_oldest` | With `grow`, the context grows only up to the largest chat context an online chat worker serving the model declares. With `truncate_oldest`, the context grows as under `grow`; when the messages do not fit even at that bound, whole turns are dropped from the oldest forward until they do. See [Context adjustments](#context-adjustments) | | `conformance` | `strict` or `best_effort` | `strict` | With `strict`, only workers declaring every guarantee the request needs serve it; `best_effort` drops that requirement. See [Guarantees](#guarantees) | | `openai_guards` | boolean | `false` | Turns on the request refusals OpenAI applies and the Horde relaxes on purpose: a `json_object` response format without the word "json" (any case) in some message text returns `400 ChatRequestConflict` with OpenAI's message. Acts on admission only; not stored | -| `include_reasoning` | boolean | `false` | Read by the OpenAI front, which returns the model's reasoning only when it is set. The native route accepts it and does not store it; it is not in the operation fingerprint, and the native status shows a submitted `reasoning_content` either way | +| `include_reasoning` | boolean | `true` | Read by the OpenAI front, which returns the model's reasoning unless it is false. The native route accepts it and does not store it; it is not in the operation fingerprint, and the native status shows a submitted `reasoning_content` either way | | `availability_mode` | `wait` or `fail_fast` | `wait` | Read by the OpenAI front. With `wait` the request queues for the front's budget; with `fail_fast` the front refuses it at the first status read when the status reports zero `eligible_worker_threads` and nothing processing, or a `wait_time` over the time left in the budget. The native route accepts it and does not store it; it is not in the operation fingerprint | | `max_kudos` | positive integer | none | A cap on the request's kudos estimate, the quote a dry run returns. When the estimate exceeds it, the route deletes the inactive row before any image upload or kudos hold and returns `400 ChatKudosCapExceeded`, whose message states the estimate at current pricing and the cap; a dry run is refused the same way. The estimate assumes the full `max_length`, while the charge is the output estimate ([ADR 0027](../decisions/0027-price-chat-generations-by-the-text-output-estimate.md)), so the check is conservative. Not stored; not in the operation fingerprint | | `dry_run` | boolean | `false` | Return the kudos cost without queueing | diff --git a/horde/chat_contract.py b/horde/chat_contract.py index 2b6b4c14..b24275ef 100644 --- a/horde/chat_contract.py +++ b/horde/chat_contract.py @@ -288,11 +288,12 @@ class ChatHordeOptions(BaseModel): charge is the output estimate, so the check is conservative. It acts only at creation, so it is not stored and the operation fingerprint leaves it out.""" - include_reasoning: StrictBool = False + include_reasoning: StrictBool = True """Return the model's reasoning as ``reasoning_content`` beside the reply. OpenAI's chat API does not return - reasoning text, so the OpenAI front hides it unless asked for. It changes what the front delivers and leaves the - work unchanged, so it is not stored and the operation fingerprint leaves it out; the native status shows the - reasoning a worker submitted regardless.""" + reasoning text, but clients that read the DeepSeek ``reasoning_content`` field render it live, and the OpenAI SDK + keeps the unknown field in ``model_extra``. Without it a client sees nothing during a long reasoning phase. False + hides it. It changes what the front delivers and leaves the work unchanged, so it is not stored and the operation + fingerprint leaves it out. The native status shows the reasoning a worker submitted regardless.""" dry_run: StrictBool = False """Return the kudos cost without queueing; native callers only.""" diff --git a/horde/oai/app.py b/horde/oai/app.py index f6fb0656..362a8ca7 100644 --- a/horde/oai/app.py +++ b/horde/oai/app.py @@ -2629,21 +2629,36 @@ class DeltaPacer: def _spread(self, *, content: str, reasoning_content: str, window_seconds: float, now: float) -> None: """Mutate the schedule by splitting the pending text into even pieces across ``window_seconds`` from ``now``. - The first piece is due at ``now``, so a merge does not delay text that was already due. + The first piece is due at ``now``, so a merge does not delay text that was already due. Every reasoning piece is + scheduled before the first content piece, so a client that renders chunks in arrival order shows all of a post's + reasoning before the reply text in the same post. The piece budget is divided between the two fields by length, + with at least one piece for each field that has text. """ rate_piece_limit = max(1, math.floor(self._max_chunks_per_second * window_seconds)) - piece_count = max(1, min(rate_piece_limit, max(len(content), len(reasoning_content)))) - content_pieces = _split_evenly(content, piece_count) - reasoning_pieces = _split_evenly(reasoning_content, piece_count) - piece_spacing_seconds = window_seconds / piece_count + total_length = len(content) + len(reasoning_content) + piece_count = max(1, min(rate_piece_limit, total_length)) + reasoning_piece_count = 0 + if reasoning_content: + reasoning_piece_count = max(1, round(piece_count * len(reasoning_content) / total_length)) + if content: + reasoning_piece_count = min(reasoning_piece_count, piece_count - 1) if piece_count > 1 else 1 + content_piece_count = max(1, piece_count - reasoning_piece_count) if content else 0 + piece_texts: list[DeltaText] = [] + if reasoning_piece_count: + piece_texts += [DeltaText(reasoning_content=piece) for piece in _split_evenly(reasoning_content, reasoning_piece_count)] + if content_piece_count: + piece_texts += [DeltaText(content=piece) for piece in _split_evenly(content, content_piece_count)] + if not piece_texts: + piece_texts = [DeltaText()] + piece_spacing_seconds = window_seconds / len(piece_texts) # The new pieces go after any pieces still in the schedule, such as a first post read in the same batch. self._pieces += [ PacedPiece( - content=content_piece, - reasoning_content=reasoning_piece, + content=piece_text.content, + reasoning_content=piece_text.reasoning_content, due_at=now + piece_index * piece_spacing_seconds, ) - for piece_index, (content_piece, reasoning_piece) in enumerate(zip(content_pieces, reasoning_pieces, strict=True)) + for piece_index, piece_text in enumerate(piece_texts) ] @@ -2778,8 +2793,8 @@ class DeltaRelay: never spliced into a choice the client already holds part of (ADR 0024). Before exposure such entries unpin the generation, and the next generation's first delta can pin it again, or the request is delivered whole. - A delta's reasoning is relayed only when the client asked for it with ``horde.include_reasoning``; otherwise it is - dropped, and a delta carrying only reasoning advances the sequence without sending anything. Reasoning is never + A delta's reasoning is relayed unless the client turned it off with ``horde.include_reasoning: false``. When off it + is dropped, and a delta carrying only reasoning advances the sequence without sending anything. Reasoning is never part of the reconciliation with the final reply. """ @@ -2792,7 +2807,7 @@ class DeltaRelay: model: The model the client asked for, the first listed for ``models``, which the relayed chunks carry and the fingerprint uses when the pinned generation's first delta omits the model that serves it. include_usage: Whether the client asked for usage, which writes a null ``usage`` in every chunk. - include_reasoning: Whether the client asked for reasoning with ``horde.include_reasoning``. + include_reasoning: Whether reasoning is relayed, from ``horde.include_reasoning`` (true when absent). """ self.generation_id: str | None = None """The generation whose deltas are relayed; None before its first delta.""" @@ -3132,7 +3147,7 @@ async def _stream_completion( created=int(submitted_at), model=chat_request.requested_models()[0], include_usage=include_usage, - include_reasoning=chat_request.horde is not None and chat_request.horde.include_reasoning, + include_reasoning=chat_request.horde is None or chat_request.horde.include_reasoning, ) wait_task = asyncio.create_task(watcher.wait_until_finished()) async for relay_event in _relay_deltas( diff --git a/horde/oai/mapping.py b/horde/oai/mapping.py index 4a07a684..b307c235 100644 --- a/horde/oai/mapping.py +++ b/horde/oai/mapping.py @@ -404,9 +404,9 @@ def build_completion( Returns: One choice per generation, in the Horde's order; ``model``, ``system_fingerprint`` and the ``x_horde`` worker - details are the first generation's. A message carries ``reasoning_content`` only under - ``horde.include_reasoning``. ``usage.completion_tokens_details.reasoning_tokens`` is the sum of the reasoning - counts the workers reported, present when any reported one, whether or not the reasoning is returned. + details are the first generation's. A message has ``reasoning_content`` unless + ``horde.include_reasoning`` is false. ``usage.completion_tokens_details.reasoning_tokens`` is the sum of the + reasoning counts the workers reported, present when any reported one, whether or not the reasoning is returned. Raises: CompletionWithoutGenerationError: ``status`` has no finished generation. A completion built from it would be a @@ -420,7 +420,7 @@ def build_completion( prompt_tokens: int | None = None reasoning_tokens: int | None = None has_empty_return = False - include_reasoning = chat_request.horde is not None and chat_request.horde.include_reasoning + include_reasoning = chat_request.horde is None or chat_request.horde.include_reasoning for generation_index, generation in enumerate(status.generations): generated_text = generation_content(generation) tool_calls = generation_tool_calls(generation) diff --git a/horde/oai/sse.py b/horde/oai/sse.py index 32d0f303..8a2ba420 100644 --- a/horde/oai/sse.py +++ b/horde/oai/sse.py @@ -5,8 +5,8 @@ """Render a streamed chat answer as server-sent events. A buffered stream sends keepalive comments while the request waits (``: horde queue_position=7 wait_time=42``), then -the finished completion as a role chunk, a reasoning chunk when the message carries ``reasoning_content`` (only under -``horde.include_reasoning``), a content chunk when the reply has text, a tool-call chunk when it has calls, +the finished completion as a role chunk, a reasoning chunk when the message carries ``reasoning_content`` (omitted when +``horde.include_reasoning`` is false), a content chunk when the reply has text, a tool-call chunk when it has calls, a finish chunk carrying ``x_horde``, the usage chunk when the client asked for it, and ``[DONE]``. A relayed stream sends the role chunk and, per delta, a reasoning chunk when it relays reasoning and a content chunk when the delta has text, while the worker generates ([`build_delta_events`][horde.oai.sse.build_delta_events]); its finished diff --git a/tests/unit/oai/test_chat_front.py b/tests/unit/oai/test_chat_front.py index 61c9cce1..3d1bec5c 100644 --- a/tests/unit/oai/test_chat_front.py +++ b/tests/unit/oai/test_chat_front.py @@ -688,22 +688,21 @@ REASONING_TEXT: str = "The user wants a greeting." """The reasoning a worker submitted beside the reply.""" -@pytest.mark.parametrize("include_reasoning", [True, False]) -async def test_reasoning_is_returned_on_the_message_only_when_the_client_asks_for_it( +@pytest.mark.parametrize("include_reasoning", [True, False, None]) +async def test_reasoning_is_returned_on_the_message_unless_the_client_turns_it_off( start_front: StartFront, - include_reasoning: bool, + include_reasoning: bool | None, ) -> None: - """OpenAI's chat API does not return reasoning, so the front hides it unless ``horde.include_reasoning`` is set.""" + """Reasoning is returned by default, including to a client that does not send a ``horde`` block, and + ``horde.include_reasoning: false`` hides it.""" + chat_body = build_chat_body() if include_reasoning is None else build_chat_body(horde={"include_reasoning": include_reasoning}) async with start_front() as front: front.horde.status_payload = done_status(reasoning_content=REASONING_TEXT) - response = await front.client.post( - "/v1/chat/completions", - json=build_chat_body(horde={"include_reasoning": include_reasoning}), - ) + response = await front.client.post("/v1/chat/completions", json=chat_body) assert response.status_code == 200 expected_message: dict[str, object] = {"role": "assistant", "content": GENERATED_TEXT} - if include_reasoning: + if include_reasoning is not False: expected_message["reasoning_content"] = REASONING_TEXT assert response.json()["choices"][0]["message"] == expected_message diff --git a/tests/unit/oai/test_delta_relay.py b/tests/unit/oai/test_delta_relay.py index c8416920..4b0def70 100644 --- a/tests/unit/oai/test_delta_relay.py +++ b/tests/unit/oai/test_delta_relay.py @@ -612,7 +612,7 @@ async def test_reasoning_deltas_are_relayed_before_the_text_when_the_client_asks assert equal_outcomes == 1 -async def test_reasoning_deltas_are_dropped_and_the_text_still_relayed_when_the_client_did_not_ask(start_front: StartFront) -> None: +async def test_reasoning_deltas_are_dropped_and_the_text_still_relayed_when_the_client_turns_reasoning_off(start_front: StartFront) -> None: """A delta carrying only reasoning advances the sequence without sending anything, so the text that follows it is relayed as the stream's first delta.""" async with start_front(with_deltas=True) as front: @@ -624,7 +624,7 @@ async def test_reasoning_deltas_are_dropped_and_the_text_still_relayed_when_the_ await asyncio.sleep(RELAY_SETTLE_SECONDS) await finish(front, reasoning_content=REASONING_TEXT) - body = await stream_while(front, generate) + body = await stream_while(front, generate, horde={"include_reasoning": False}) relayed = front.metric_total(RELAYED_NAME) prefix_outcomes = front.metric_total(RELAY_OUTCOMES_NAME, horde_outcome="prefix") @@ -709,7 +709,7 @@ async def test_stream_lost_before_any_delta_was_sent_delivers_the_request_whole( await asyncio.sleep(RELAY_SETTLE_SECONDS) await finish(front) - body = await stream_while(front, generate) + body = await stream_while(front, generate, horde={"include_reasoning": False}) relayed = front.metric_total(RELAYED_NAME) relay_outcomes = front.metric_total(RELAY_OUTCOMES_NAME) cancels = front.horde.cancels_received @@ -919,7 +919,7 @@ async def test_stream_replaced_with_lower_ids_before_any_delta_was_sent_delivers await asyncio.sleep(RELAY_SETTLE_SECONDS) await finish(front) - body = await stream_while(front, generate) + body = await stream_while(front, generate, horde={"include_reasoning": False}) relayed = front.metric_total(RELAYED_NAME) relay_outcomes = front.metric_total(RELAY_OUTCOMES_NAME) cancels = front.horde.cancels_received @@ -1098,15 +1098,23 @@ async def test_a_maximum_window_of_zero_sends_each_post_whole_when_it_arrives() assert [(due_at, piece.content) for due_at, piece in schedule] == [(3.0, "Hello "), (3.0, PACED_TEXT), (4.0, "more")] -async def test_reasoning_is_paced_with_the_text() -> None: - pacer = DeltaPacer(max_window_seconds=2.0, max_chunks_per_second=3) +async def test_a_post_with_reasoning_and_text_paces_all_the_reasoning_before_the_text() -> None: + """A client that renders in arrival order shows the end of the reasoning, then the start of the reply, without + alternating between them. The piece budget is divided between the two by length.""" + pacer = DeltaPacer(max_window_seconds=2.0, max_chunks_per_second=6) pacer.add([accepted("Hello ", 100.0), accepted("abcdef", 101.0, reasoning_content="thinking")], now=0.0) schedule = drain_schedule(pacer) paced_pieces = [piece for _due_at, piece in schedule][1:] - assert piece_contents(paced_pieces) == ["ab", "cd", "ef"] - assert [piece.reasoning_content for piece in paced_pieces] == ["th", "ink", "ing"] + assert [(piece.reasoning_content, piece.content) for piece in paced_pieces] == [ + ("th", ""), + ("ink", ""), + ("ing", ""), + ("", "ab"), + ("", "cd"), + ("", "ef"), + ] @pytest.mark.parametrize(