Update funding and usage guidance, clarify generation polling and webhook payloads, and modernize the stable landing page with current apps, API links, and community resources.
- Add a guide covering the native text API: headers, model selection, generation parameters, context_fit, worker restrictions, dry-run cost checks, submitting, polling, reading and cancelling requests, webhooks, styles, and error handling.
- Also links the new page from the how-to index, topics page and zensical.toml navigation.
- Rewrite README.md to reflect the current service: quick-start links, OpenAI-compatible chat endpoint, updated tech stack, funding, docs, contributing and license sections.
- Replace the long FAQ.md with a pointer to aihorde.net/faq. Shrink README_StableHorde.md and README_KoboldAIHorde.md to short pointers to the website and main readme. Add a private-horde intro to README_docker.md.
- Remove the now-unused FAQ and GUI images (and their .license files), and update the join-the-horde and FAQ links in index_stable.md.
- sql_statements/5.1.16.txt: section 3 sets lock_timeout and statement_timeout to 0 and resets both at the end. Before it marks an index valid, a concurrent build waits for every transaction with an older snapshot, on any table. Under a lock timeout, one such transaction fails the build after the whole table has been read, and the build leaves an INVALID index. The builds and VALIDATE CONSTRAINT take SHARE UPDATE EXCLUSIVE, which does not conflict with writes, so their waits do not hold up requests.
- pg_temp.kudos_add_legacy_range sets lock_timeout to 5s as a function setting. Adding the NOT VALID CHECK takes ACCESS EXCLUSIVE, and writes queue behind its wait. The swap transactions keep their SET LOCAL timeouts.
- Tests: a concurrent build waits for a transaction on users that stays open longer than any lock timeout in the file, and the run then completes without an INVALID index.
- Docs: the how-to's build and CHECK steps, with a query for the transactions that hold up a build and a check of each table's freeze age.
- select_next_unit catalogues the legacy partition as soon as it is attached and takes a unit when its own upper bound is past the window, so the catch-up of pre-swap history starts when retention is enabled instead of one window after the swap time. The partition drop still waits for every sub-range, so the last one, ending at the swap time, holds the detach for a window.
- retention_health computes oldest_eligible_age by sub-range for a legacy partition with catalog rows (_eligible_uppers), and by partition bound otherwise.
- Tests: the early catalogue and the per-sub-range selection.
- Docs: the how-to, reference, explanation and ADR 42 state the per-unit rule.
- retention_health reports pending_units_by_dataset, and the tick exports it as horde.kudos.retention.pending_units with horde.kudos.dataset. The units of the legacy partition count as pending until the whole partition is dropped, so the gauge stays high for a whole catch-up. An alert can use it to tell a catch-up from steady growth, which the age of the oldest eligible partition does not show when a catch-up starts on time.
- pending_units is the sum of the per-dataset counts.
- Tests: the health test reads the per-dataset counts.
- Docs: the metric and health tables of the reference.
- sql_statements/5.1.16.txt: the kudos_ledger swap takes the applier advisory lock (KUDOS_APPLIER_LOCK) before it locks users. The applier claims ledger rows and then updates users, the reverse of the swap's order. The applier only tries that lock, so it skips its cycles until the swap commits and does not deadlock with it.
- After the advisory lock, each lock wait of the swap ends after 500 ms, below deadlock_timeout. A request that inserts a posting before it updates users can still form a lock cycle with the swap. The swap then fails with lock_not_available before the deadlock check runs in the request, so the request continues and the file is rerun. The file stops with an error when deadlock_timeout is 500 ms or less.
- archive evict runs evict_statistics_dumps. It keeps the dumps of statistics units that are not dropped, because their raw rows are still online. It sets the object_key of each evicted unit to NULL, also after a failed batch. The command exits 1 when the kept dumps hold the tier above the budget. KudosArchive.enforce_budget takes kept_keys for this.
- The tick does not clear the key of a missing statistics dump without a trace. A dumped unit whose object is missing goes to dump_failed and is dumped again on the next tick. A detached one blocks with archive object missing.
- A dropped statistics unit keeps the key of a missing object, so archive audit reports the loss on every run.
- _check_statistics_dumps runs with the currency check before the detach and before the drop. The raw rows of a unit stay online when its dump was removed after the last listing.
- archive object missing is waivable for statistics. retention unblock --waive-dump refuses a unit of a detached partition, so the drop after a waiver starts from an attached partition.
- A dataset with an attached partition that has dropped catalog units does not make a transition, and the tick returns blocked with RESTORED_PARTITION_REFUSAL. Such a partition was restored under its original name, and the catalog does not describe its rows. retention_health lists restored_partitions, and the gauge horde.kudos.retention.restored_partitions exports the count.
- Tests: the migration tests run the swap against a concurrent applier cycle and a concurrent request, and against a short deadlock_timeout. The retention tests cover a vanished dump in each state and an eviction that keeps online dumps. A further test restores a dropped partition under its name. The reconcile test expects dump_failed for a vanished dump, and the rehearsal grace test expects the restored-partition block after a future reference time drops the current week.
- Docs: the how-to's swap, eviction, missing object, rehearsal and rollback text, the reference's block reason, eviction, reconcile, metric, log and health tables, and the explanation's diagnosis table.
- sql_statements/5.1.16.txt creates eight consecutive weekly partitions from the swap time instead of one, each with the same name pattern, bounds and autovacuum storage parameters. A release without the retention tick does not create weekly partitions, so a rollback to such a release keeps inserting for eight weeks after the swap instead of one. A rerun creates only the weeks that are missing: a week that an existing partition overlaps is skipped, as is a week that ended before the current one, so a rerun neither collides with a partition the tick created at another width nor brings back a week retention dropped.
- DEFAULT_DROP_GRACE_DAYS is seven. A deployment that loses KUDOS_RETENTION_DROP_GRACE_DAYS in a render keeps a reattach window, and a grace of zero has to be set explicitly. retention_health reports drop_grace_days, so retention status shows the grace in force and a deploy can be gated on it.
- Tests: the migration tests expect the legacy partition plus eight weeks after a run and after a rerun, the tick's bootstrap continues from the eighth week, every week keeps the storage parameters, and a rerun recreates a dropped week while leaving a hand-made wider partition untouched. The unit and end-to-end fixtures pass a zero grace explicitly, the policy test reads the default of seven and explicit values, and the health tests read drop_grace_days.
- Docs: the how-to's conversion end state, catalog step and rerun paragraph, the reference's partition layout and environment and health tables, and the explanation's grace text.
- KudosArchive.version_usage lists the object versions under a tier's prefix and sums the noncurrent sizes and the delete markers, manifests included. tier_bytes and object_sizes count current objects only, so in a versioned bucket this listing is the only place where the storage left behind by an eviction or a re-export is visible. A store that does not list versions, a key without that permission or an unreachable store returns None after a warning, because the figure is informational.
- The archive audit summary line gains noncurrent_bytes and delete_markers, each a map from tier to the figure or null. Null differs from zero, and neither figure changes the exit status.
- Tests: the fake store serves list_object_versions when versioned and refuses it otherwise, so the existing audit tests cover the unsupported path. Two evictions out of three objects yield the expected noncurrent bytes and markers while tier_bytes is unchanged, and an unversioned store yields None for both tiers.
- Docs: the archive audit row of the reference, the module symbol table, and the audit step of the how-to.
- The eviction step of the how-to says that a deleted object stays stored as a noncurrent version the tier listing does not count, so the space is freed by the deployment's procedure as well
- Manifest format version 2 adds column_types, each column's name and PostgreSQL type as format_type reports it with modifiers. The export reads them through a temporary view over the export query inside the export transaction, because the driver reports types without modifiers, and it aborts the upload when the CSV header differs from the described names (ArchiveColumn, _describe_copy_columns in horde/kudos_archive.py).
- restore_archive_unit creates the scratch table from column_types and never consults the live parent table, so a later rename, removal or retype of a column does not break the restore of an older object. A manifest type must match the allowlist of built-in type spellings before it enters CREATE TABLE.
- A version 1 manifest does not record column types. Verification, audit and the drop pipeline read it as before, and restore refuses it with an error that states the version.
- archive restore exits with the error message on an archive error.
- Tests: one fixture object with a version 1 and a version 2 manifest. The version 2 object restores against PostgreSQL with its types and values checked, the version 1 object is refused, a tampered type is refused, and the fake connection of the measured-bytes test answers the describe queries.
- Docs: the manifest field table, the restore command row and the module symbol table in the reference, and the restore step in the how-to.
- Currency retention checks every unit of the partition against the archive before the detach and again immediately before the drop (_check_currency_archive in kudos_retention.py). The object must exist, its manifest must be readable, the stored size must equal the manifest and the manifest row count must equal the unit's recorded count. A missing object or key blocks the unit, a mismatch blocks it with "archive mismatch", and an unreachable store fails the tick so a later one retries. The drop guard previously checked only for a recorded key, so an object lost after verification was noticed by the post-step listing once the partition was gone.
- create_balance_snapshot runs under the retention lock and raises StaleCheckpointsError while a currency checkpoint is stale. A snapshot taken then stores applied totals short of the late rows, and a repair against it after the recheckpoint posts a false correction.
- recheckpoint_from_unit records one kudos_snapshot_invalidations row, from the earliest reset unit's catalog creation to the reset, in the transaction that deletes the stale checkpoints. reconcile_balances refuses a snapshot created inside any interval in read-only and repair modes, and the snapshot rows are kept as evidence. The model is KudosSnapshotInvalidation, and 5.1.16 creates the table in a rerun-safe statement.
- Archive uploads are bounded by their measured compressed size. _tier_allowance measures the tier once, the estimate is checked against it, and _MultipartSink raises ArchiveBudgetExceeded when the compressed bytes, including the final flush, would pass the allowance. The multipart upload is aborted, an object already at the key is kept, and the unit blocks with the tier's budget reason. The estimate alone let a tier pass its budget.
- archive evict takes the retention lock, because an export measures its allowance from the tier size.
- Tests cover the drop's archive check for a missing manifest, an unreachable store and a size mismatch, an underestimated upload for both tiers, the measured limit at one byte under, at and one byte over, the recheckpoint cascade refusing the pre-incident snapshot, and snapshot creation refused while the lock is held or a checkpoint is stale. The vanishing-object test is parametrized over hold_drops. The safety test commits and snapshots before the stale unit exists.
- The reference's reconciliation section describes the lock, the invalidation interval and its width on legacy units. The schema and interface tables list the new table and the refusal. The archive section describes the measured budget and the pre-drop check. The how-to says when snapshot exits with an error. The ADR's confirmation list gains both behaviors.
- A recheckpoint cascade deletes the checkpoint rows and daily statements of every reset unit in the same transaction, so reconciliation falls back to the last valid boundary; reconcile --apply refuses while a currency unit is blocked for late rows or online drift
- The detachment is recorded on every unit before the post-detach recount, so a late row found then leaves the partition reattachable; reattach accepts a partition whose units are all blocked with a detachment time
- The drop's autocommit connection is discarded after use, so its lock timeout never reaches the pool
- Each archive tier has its own budget and nothing is evicted automatically: a tier whose next upload would exceed its budget blocks its unit; eviction is the operator command archive evict, which reports the keys deleted even when a later batch fails; the tick clears catalog keys whose object is missing and blocks a currency unit awaiting its drop with "archive object missing"; the drop refuses a currency unit without an object key
- archive audit verifies every object of a tier against its manifest and reports catalog units whose object is gone; archive restore refuses an existing table unless --append; a dump blocked by the statistics budget can be waived
- Reference, how-to, explanation and the record describe the per-tier budgets, the new block reasons and the commands
- A new weekly partition is created as a standalone table (LIKE the parent, with a bounds CHECK and the autovacuum settings) and attached, which takes SHARE UPDATE EXCLUSIVE on the parent instead of the ACCESS EXCLUSIVE that CREATE TABLE ... PARTITION OF takes, so a long reader no longer queues every insert behind partition maintenance
- The listing lock and each attach run under a short lock timeout; a timeout logs a warning, leaves no standalone table behind and defers that partition to the next tick; a range whose table name is already taken is skipped
- The idempotency key backfill in 5.1.16 covers key_lifetime_days (psql variable, default 30) so a deployment with longer windows backfills every key still inside its promised lifetime
- docs/explanation/kudos_retention.md is the entry point: the problem, the three readers of old rows and their replacements, the rejected alternative, partitions by age, the per-dataset replacement and archive table, one state diagram per dataset, properties and costs, a symptom table and a code map
- docs/reference/kudos_accounting.md gains the retention sections: catalog columns, states and step table, block reasons, configuration, archive objects, metrics and log lines, health fields, operator commands
- docs/how-to/kudos_ledger_operations.md gains the partition conversion, enabling retention with a held-drops soak and a grace period, rehearsing removal on a restored copy, handling blocked units, recovering from late rows or a chain gap, rolling back a removed partition, user statements and archive restore
- The explanation of kudos accounting links to the retention explanation instead of describing retention itself; the class index, topics page and site navigation list the new document
- The quorum node runs kudos_retention_tick every 60 seconds on its own advisory lock, so a slow export does not delay the applier
- The tick records step outcomes, dump failures, blocked units by reason, partition coverage, held partitions, the oldest eligible age and the archive tier and budget bytes; rare outcome series get a zero add each tick so alerts have a series before the first event
- Blocked and failed dump ticks log a warning line with a stable prefix for log-based alerts
- The compatibility prune_kudos_ledger thread function is removed
- tools/kudos_ledger_admin.py gains retention status, step, unblock and reattach commands, a daily statement report and archive restore into a scratch table
- The privacy policy discloses the accounting retention windows and, with an archive configured, the archive provider; the windows come from AccountingRetentionDisclosure so the document follows the applied policy
- The transfer idempotency header documents that a key is honored for the online retention window
- transfer_kudos records a kudos_idempotency_keys row for client-keyed transfers and detects retries and conflicting parameters from it, so retry detection no longer scans ledger postings by event_id and a retry after the key expires is a new transfer
- Reconciliation repairs record a repair key with their postings, and a repair from a snapshot older than the key lifetime is refused
- The all-time applied total is the latest ledger checkpoint plus the applied postings at or above its boundary, so reconciliation stays correct after retention drops old postings
- The compatibility no-op prune_applied_kudos_ledger and its tests are removed
- Tests read partitioned event rows by id with a query, since the primary key is now (id, created), and the quarantine migration test runs against an unpartitioned table in a throwaway schema
- horde/database/kudos_retention.py moves one partition one step per tick under the retention lock: statistic partitions are folded into daily rollups, their quarantined rows moved and an archive dump written or waived; currency partitions are checkpointed per user with daily statements, exported, verified against the manifest and the online rows, then dropped
- KUDOS_RETENTION_POLICY is loaded from the environment at import and refuses windows below 30 days, partition widths outside 1 to 31 days, and a configured archive without a byte budget
- Nothing is removed unless KUDOS_RETENTION_ENABLED is set; a drop grace period detaches a partition before dropping it, and KUDOS_RETENTION_HOLD_DROPS keeps drop-ready partitions attached
- A unit that fails a guard is blocked with a reason until an operator unblocks it, waives its dump, recheckpoints from it or reattaches its partition; restore_archive_unit loads an archived unit back into a table
- expire_idempotency_keys removes keys older than the key lifetime in batches
- .env_template documents the retention and archive variables
- tests/unit/test_kudos_retention.py and tests/integration/test_kudos_retention_end_to_end.py cover the step table, guards, operator paths and a full run against the managed object store
- horde/kudos_archive.py writes zstd-compressed archive objects with manifests to an S3-compatible bucket under a normalized deployment prefix, keeps statistic dumps within a byte budget by evicting only that tier, and verifies a read-back against the manifest
- KudosArchive.from_environment returns None unless the KUDOS_ARCHIVE_* endpoint, bucket and both token variables are set
- zstandard is a runtime dependency; kudos_archive and kudos_retention are under strict mypy
- The managed Garage test store and the CI test stack create a kudos-archive bucket and export the KUDOS_ARCHIVE_* variables
- tests/unit/test_kudos_archive.py covers the codec, manifests, budget eviction and restore verification
- kudos_ledger and kudos_stat_events are range-partitioned on created with (id, created) primary keys, so retention removes whole partitions by age
- KudosPartitioning and ensure_kudos_event_partitions create upcoming partitions with the autovacuum storage parameters; init_db and the test fixtures call it after create_all
- The retention catalog, stat rollup, quarantine, ledger checkpoint, daily statement and idempotency key models and their enums are declared alongside the event tables, so one migration and the ORM build the same schema
- kudos_db gains a session-level retention advisory lock taken on a dedicated autocommit connection
- sql_statements/5.1.16.txt creates the retention tables, backfills idempotency keys inside the online window and converts the existing tables in place by attaching each as a legacy partition behind a validated range CHECK
- tests/integration/test_kudos_retention_migration.py checks the conversion, reruns, refusal guards and ORM parity with the migration
- Replace the sealed-ID-range, archive-everything-forever proposal in ADR 42 with time partitions by created, eligibility by age, statistic rollups that are deleted after verification, and currency checkpoints plus a verified archive
- Record the previous form as a rejected option and add the erasure and disclosure position
- The record states behavior only; capacity figures and probes live outside the repository
- Update actions/checkout to v7 and actions/setup-python to v7 across workflows and the setup-test-python composite action.
- Pin astral-sh/setup-uv to the v10.3.3 commit SHA in the composite action and docs workflow, replacing the v5 and v10 tags.
-This aligns action versions with those already used in docs.yml.
Broad editorial pass over docs/ plus a nav update in zensical.toml.
- Wording and style: replace "carries"/"answers" phrasing with plainer terms, use periods instead of semicolons, and split long paragraphs.
- Structure: add subheadings and tables in the chat, moderation, kudos, model reference and style contract docs. Add "Code map" and "Contract tests" sections to several reference pages. Move the code maps in chat_front.md and chat_requests.md to the end of those pages. Reorder the minimal chat worker page so the procedure comes before the source listing.
- Content fixes:
- Document the 404 ChatJobAborted vs ChatProgressNotStreaming progress responses.
- Document the availability refusal table (503/404/400).
- Document fail-fast admission and queue-position refresh.
- Fix the stream parser example to check response.ok and error on a missing [DONE].
- Fix the style contract's max_length ceiling (16384).
- Note that the kudos heartbeat metrics are quorum-node only.
- Clarify tool_choice, strict conformance and capability caveats in the tutorial.
- Navigation: reorder Tutorial and Topics in the docs landing page and zensical nav. Exclude /openai-chat-local/ from builds.
- Chat doc summaries were rewritten in several front matter blocks and the generated README tables updated to match.
The pop queries did not read `trusted_workers`, so a request for trusted-workers-only reached the page of every worker and was refused afterwards in `can_generate` for an owner that is not trusted. Such a request at the front of the queue then cost every untrusted worker a page slot on *every* pop until a trusted worker took it.
- `text_wp_offer_clauses` in `horde/database/text_functions.py` and the image pop query in `horde/database/functions.py` offer a request only when it is not trusted-only or the popping worker's owner is trusted, folded in as the unsafe-IP and NSFW clauses are. The Python check in `can_generate` stays as the guard behind the page
- the text pop's TODO for this filter is removed
- unit tests show a trusted-only request is not paged to a worker whose owner is untrusted and is paged once the owner gains the trusted role, for text and image workers
The route handlers read and validate bodies by hand, so FastAPI does not derive a schema from them and the front had no API description. The document is generated from the models used by the handlers for validation and serialization, so it describes the handlers without changing how they behave.
- `horde/oai/openapi.py` builds the document: component schemas from `ChatCompletionRequest` as validated and from the response, chunk, error envelope and model list models as serialized; each route's request body, responses by status with the front's codes and the `x-should-retry` and `Retry-After` headers; a front description that states the differences from OpenAI (refused and no-op keys, the `max_tokens` range, default and lowering, `reasoning_content`, `models`, the `horde` and `x_horde` objects, `system_fingerprint`, tool-call chunks, the error envelope) and links the tutorial, how-to, reference and explanation pages on the published documentation site
- the request schema lists the no-op keys at their default-only values, the schemas of models whose unknown keys are dropped allow other keys, and FastAPI's 422 response is removed since the front never returns it
- `GET /` redirects to `/docs`; `/healthz` is left out of the document
- the field docstrings in `horde/oai/schema.py` and the `horde` options in `horde/chat_contract.py` are the public field descriptions (`use_attribute_docstrings`), rewritten to state the front's behaviour and where it differs from OpenAI; maintainer notes moved to comments; mkdocstrings cross-references are rendered as plain names in the document
- tests check the routes and models in the document, that every accepted request key and `horde` option is listed with a description and the reasoning default is true, that the header names match the app's constants, and the description rendering
- the chat front reference describes the document and its adjustments, and the run how-to lists the routes
A ratio of completions to admitted requests cannot show a fault, since most admitted requests are refused for reasons the front cannot serve (an unknown model, no online chat worker, a client error). One bounded label sorts every finished request so a served share of servable requests can be read.
- `OutcomeClass` in `horde/oai/metrics.py`: `served`, `client_gave_up`, `capacity`, `service_fault`, `unservable`, `client_error`, `front_capacity` and `unclassified`, recorded as `horde.outcome_class` on `horde.oai.requests.refused`, `horde.oai.requests.errors`, `horde.oai.stream.failures` and `horde.oai.request.duration`
- `OUTCOME_CLASS_BY_CODE` and `outcome_class_for` in `horde/oai/app.py` decide the classes beside the code constants. `no_eligible_worker` is `capacity` at the `grace` stage and `unservable` at `submit`. A code outside the table, including a new Horde `rc` passed through, is `unclassified` rather than given a default class, so it stays visible
- the Horde return codes the front passes through without a constant of its own (`UnknownChatModel`, `PromptExceedsContext`, `InvalidAPIKey`, `KudosUpfront`) get constants in `horde/oai/app.py`
- unit tests check that every front code constant has a class, both stages, unknown and missing codes, and the label on each of the four instruments
- the chat front reference lists the label on the four rows and describes the classes, the stage rule and the served share
The Locust jobs drove only image and text users, so the chat routes and the OpenAI front had no coverage in CI beyond the pytest suite.
- the stress smoke jobs in `prtests.yml`, `maintests.yml` and `trusted-fork-integration.yml` start the chat front from source with `python -m horde.oai` on port 7002, pointed at the Horde and the test stack's Redis, and wait on `/healthz` before the Locust run
- `docker-publish.yml` starts the front as a second container from the built image, as the compose service does, and removes it at teardown
- every one of these runs adds three chat workers, two native clients and three front clients to `locustfile.py`; `--users` rises from 24 to 32 since the chat counts are pinned and taken from it. Chat workers pop under generated names from the bootstrapped worker keys
- the front's log is scanned for tracebacks, uploaded with the server's and dumped on failure; `check_smoke_results.py` already gates on contract violations and environment conditions, so chat violations fail the job
- `tests/stress/README.md` describes the chat mix in CI and the gate's full scope
- `max_tokens_over_worker_limit` in `tests/stress/locustsuite/chat_scenarios.py` expects a text reply: the Horde lowers a `max_tokens` above every online worker's `max_length` at submit and serves the request. The persona's payload check fails a job whose `max_length` is above its declaration, so a request that was not lowered is still a violation. The unused `no_worker_supports_limit` code constant is removed
- `LIMITED_CARD_PARAMETERS` in `tests/stress/locustsuite/chat_front_client.py` lists `parallel_tool_calls`, which the front adds to a model card for any worker that declares `tools`
- Reworded the chat worker how-to, tutorial, wire examples and the explanation page so each sentence states one thing in plain terms
- The same pass over the docstrings, comments and log messages of the minimal chat worker tool and the chat test helpers
- Renamed the wire-example tests' answered_by field to response_example
- _fake_chat_worker.py holds the request constants, worker feature lists and helpers (serve_reply, image_body, png_data_url, submit_image_chat_request, post_progress) that several chat test modules used, so no test module imports another.
- _front_stack.py holds the front conformance worker, reply picker and constants; its fixtures and the progress and offload fixtures are in conftest.py.
- The wire-example modules declare their own autouse fixtures and import only shared modules; the front SDK CHAT_MESSAGES is FRONT_CHAT_MESSAGES so one name has one meaning.
- test_minimal_chat_worker_example.py uses LOOKUP_TOOL in place of its duplicate WEATHER_TOOL.
- test_chat_wire_examples.py is marked integration.
- docs/reference/chat_wire_examples.md holds the JSON bodies of native requesters, OpenAI front clients and text workers, with a sequence diagram of one chat job
- test_chat_wire_examples.py and test_chat_front_wire_examples.py capture each response body live and compare it with the page: keys, the values of rc, code, type, state, finish_reason, kind, reason and booleans, and the JSON type of every other scalar; documented requests are posted, and worker request keys are checked against the route's parser and models
- tests/integration/_wire_examples.py holds the page parsing and comparison both test modules share
- serve_chat_jobs.md opens with one chat job in full; it, chat_requests.md and chat_front.md include wire examples through pymdownx.snippets sections, so a shown body cannot drift from the checked one
- chat_requests.md and chat_front.md link each major section to its examples; the 202 description states when message and warnings appear, and the front's error descriptions carry the 404 type and metadata.rc
- tools/chat_worker_example/minimal_chat_worker.py pops chat jobs only, sends each to the backend's /v1/chat/completions and submits the reply, or faults with prompt_overflow, generation_error or reply_refused
- the fit check counts the prompt exactly through the backend's tokenizer route (TokenCounter llamacpp, koboldcpp, vllm) or bounds it by UTF-8 bytes; the worker declares prompt_overflow_refused only when a startup probe shows the count reaches generation's usage.prompt_tokens, so an undercounting counter cannot let an overflowing prompt through
- check_token_counter.py compares a backend's counter with generation on four conversations and shows whether the backend refuses an oversized prompt or trims it
- test_minimal_chat_worker_example.py drives the worker against the in-process horde with a stub backend: reply, request body, fit boundary, counters, refusal and tool calls
- docs/how-to/write_a_minimal_chat_worker.md includes the script through pymdownx.snippets, enabled in zensical.toml, so the page shows the code the tests run
- submit_chat_request, chat_status, cancel_request, take_worker_offline and build_tool_call move from test_chat.py into tests/integration/_fake_chat_worker.py under public names, with CHAT_MODEL and CHAT_MESSAGES, so other chat test modules import them from the shared helper module instead of a test module's private names
- test_chat.py imports them from there; its tests and assertions are unchanged
- serve_chat_jobs.md: the koboldcpp worked example follows release 1.124, which declares chat on every pop, pops from aihorde.net, declares only the prompt_overflow_refused guarantee, estimates fit from the request JSON and faults with prompt_overflow or reply_refused; step 1 and the example verify the declaration through worker details
- first_chat_worker.md: the tutorial runs on aihorde.net with a model name no other worker serves, lists the release feature set, and sends its forced tool request as best_effort, since no release koboldcpp worker declares tool_choice_enforced
- chat_requests.md: koboldcpp faults a refused reply only after ChatSubmitInvalid, ChatSubmitNonconformant or ChatModelMismatch
- send_chat_completions.md: a strict request no worker can serve ends with 404 no_worker_supports_features
- `capable_chat_worker_clauses` in `horde/classes/kobold/worker.py` gains the SQL form of `serves_chat_images`: a request that requires `vision` has a capable worker only when that worker declares the `url` or `base64` image transport. A worker that declares `vision` with no way to receive images, or with `messages_url` alone, no longer keeps an image request from a slower worker that can take it. The clause is built from `TextArrayContainsAll`, so it compiles on both dialects
- the pop clause comment in `horde/database/text_functions.py`, ADR 0041 and the chat request reference state the transport condition
- a parametrized unit test in `tests/unit/test_text_job_ttl.py` covers a faster vision worker with no transport, an empty list, `messages_url` only, `url` and `base64`
- `text_wp_speed_gate_clause` offers a chat request above the popping worker's lease capacity when no capable chat worker is online, through a correlated NOT EXISTS over the workers table. A capable worker is online, accepts chat jobs for one of the request's models, is not in maintenance or paused, declares chat limits that admit the request, declares every chat feature the request requires and has a capacity of at least its `max_length` (`capable_chat_worker_clauses`, `text_worker_lease_capacity_covers_clause` in `horde/classes/kobold/worker.py`)
- `TextArrayContainsAll` in `horde/classes/base/sql_constructs.py` compiles the feature test per dialect: `@>` over coalesced `text[]` on PostgreSQL, and a `json_each` anti-join on SQLite that reads SQL NULL and JSON `null` as an empty array. A faster worker without a required feature therefore does not keep a request from a slower worker that has it
- `TextWorker.can_generate_with_eligibility_inputs` returns `lease_speed` only while a capable worker is online. It reads the fact through `TextWaitingPrompt.has_capable_chat_worker`, which keeps the result on the request object for the request's current lengths. Availability counts the slower workers, so such a request is possible and is not refused as `no_eligible_worker`
- `CHAT_WORKER_ONLINE_WINDOW` is defined in `horde/classes/kobold/worker.py`, so the capable worker test and the chat model queries share one online window
- an over-capacity chat assignment stores `over_capacity_at_assignment` and is counted by the `horde.text.gate_shadow.*` instruments as a legacy one is. Those instruments gain a `horde.chat` attribute
- the front's finished-request line ends with `max_tokens` and `max_tokens_source`, and `horde.oai.requests.errors` gains `horde.max_tokens_bucket`, the upper bound of the limit's `BUCKETS_MAX_TOKENS` bucket or `over`, so refusals can be read by the requested limit
- ADR 0041 records the preference. ADR 0038, ADR 0040, the chat request and chat front references and the chat worker how-to describe it
- unit tests cover the pop clause and the eligibility check with and without a capable worker, faster workers that cannot take the request, the unmeasured speed, availability, the required features, the containment on both dialects, the chat shadow records and the front label and log fields. Integration tests cover a slow sole worker receiving the request and a slow worker passed over for a capable one
The reasoning_effort default is a policy whose effect is only visible in production: whether declaring workers receive it, how often koboldcpp's budget fires, how often a reply ends as reasoning only, and what completion limits clients run under. These instruments make each of those countable per model.
- `horde.chat.reasoning_effort_default` counts chat job pops whose client omitted `reasoning_effort`, by model and whether the worker declares the feature (`horde.applied`); `apply_worker_chat_defaults` returns a `ReasoningEffortDefault` and `get_pop_payload` emits the count
- `horde.chat.reasoning_marker` counts stored chat replies with koboldcpp's budget-exceeded marker in `reasoning_content` or `content`, or a raw `<think` tag left in `content`, by model and `horde.where` (`ReasoningMarkerPlace`); `find_reasoning_markers` in `chat_request.py` is the pure match and `TextJobSubmit.record_reasoning_markers` emits it once per stored reply. The marker text is an instrument, not a worker contract
- `horde.oai.completions` gains `horde.content` (`text`, `tool_calls`, `reasoning_only`, `empty`, read from the native status so `include_reasoning: false` does not hide reasoning) and `horde.reasoning_effort` (the client's value or `omitted`)
- `horde.oai.request.max_tokens` histogram records the effective completion limit of each request the Horde accepted at submit, by `horde.source` (`client`, `default`, `lowered`), at the request's end since a lowered limit is only visible in a status
- `horde.oai.reasoning_share` histogram records reasoning tokens over completion tokens per delivered completion with reasoning, by model and finish reason
- A completion without visible text logs one INFO record with its limit, the limit's source, the effort, the reasoning tokens and the finish reason
- `RequestTrace` keeps the client's request instead of only its models
- Reference docs for the front metrics and the native counters; unit and integration tests for each instrument
OpenAI's default `reasoning_effort` is `medium`. koboldcpp maps the value to a thinking budget as a fraction of `max_length` and treats an absent value as unlimited, so a request with a small `max_tokens` spent the whole budget on reasoning and returned an empty reply. Reasoning cannot be resumed through the OpenAI API, so a reply cut off while thinking is lost, and `max_tokens` has to cover the reasoning and the answer in one request.
- `DEFAULT_CHAT_REASONING_EFFORT` in `horde/chat_contract.py` is `medium`
- `apply_worker_chat_defaults` adds it to one job's payload copy in `TextWaitingPrompt.get_pop_payload`, only for a worker that declares the `reasoning_effort` feature and only when the client sent no value. An omitted value does not require the feature, so routing, the stored request, the operation fingerprint and the kudos estimate do not change. An explicit value, `none` included, is passed unchanged
- Unit tests cover the default, the untouched explicit values and non-declaring workers. Integration tests pop a declaring and a non-declaring worker
- ADR 0033 and 0036, the pop payload and request field descriptions, and the chat request, serve-chat-jobs and send-chat-completions docs state the behaviour
- The send-chat-completions how-to, the chat request reference and explanation, ADR 0033 and the native and front request field descriptions publish the budget policy: reasoning is generated within `max_tokens`, an omitted effort gives a declaring worker a budget of about half on koboldcpp, the budget applies per thinking block and does not guarantee an answer, and a reply cut off while thinking cannot be continued. The `include_reasoning` swagger field states its true default
Reasoning is output the worker generated, and OpenAI bills it as completion tokens. A reply whose whole budget went to reasoning was charged and rewarded at the 20-token floor, and its speed sample was the floor over the full elapsed time, so a reasoning model pushed its own worker under the chat speed gate.
- `chat_reply_characters` counts `reasoning_content` with the content and the tool calls. `get_things_count` keeps the floor of 20 and the cap at the requested `max_length`, so a reasoning-only reply is charged and rewarded for the budget it used
- `TextJobSubmit.get_submitted_things_count` is the speed sample's count. The chat submit override passes the verified reply as `message` to `get_things_count`, so the sample and the charge are one count. Text jobs and faulted chat submits keep the generation string path
- `build_completion` estimates `usage.completion_tokens_details.reasoning_tokens` from the reasoning characters when the worker did not report a count, with the formula of the other estimated counts. An estimated `completion_tokens` includes the reasoning, as OpenAI's does. Both estimates are capped at the effective completion limit from `effective_completion_limit`: the lowered `worker_max_length`, else `max_completion_tokens`, else `max_tokens`, else the native default. Reported counts are used as reported, and the stored `reasoning_tokens` column keeps its reported-only meaning
- `DEFAULT_CHAT_MAX_TOKENS` moves to `horde/chat_contract.py` so the front can read the native default. `horde/classes/kobold/chat_request.py` re-exports it
- The integration test that asserted reasoning is not charged now asserts that a reasoning reply earns and samples what a same-length plain reply does
- ADR 0027 and the chat pricing, reasoning and usage docs state the charge basis, the speed sample path and the estimate
A reasoning model can think for most of a minute before its first visible token. Clients that read the DeepSeek `reasoning_content` field show that progress live, and the OpenAI SDK keeps the unknown field in `model_extra` without complaint, so returning the reasoning by default costs a plain client nothing.
- `horde.include_reasoning` defaults to true. `false` hides the reasoning. The option changes what the front delivers and not the work, so it stays out of the stored request and the operation fingerprint
- `DeltaPacer._spread` schedules every reasoning piece of a worker post before its first content piece. The piece budget is divided between the two fields by length, with at least one piece for each field with text, so a client that renders in arrival order shows the end of the reasoning and then the start of the reply
- `DeltaRelay`, `build_completion` and the SSE module read the option as on unless false
- Tests cover the default without a `horde` block and the reasoning-first order of a paced post. Relay tests that assert dropped reasoning turn the option off explicitly
- ADR 0020, the chat front and chat requests references and the how-to state the default and the pacing order
A streamed completion was relayed as one SSE chunk per worker progress post, so a worker's once-a-second posts arrived as lumps. Worker cadence is the wrong lever, since faster posts cost the Horde progress writes.
- DeltaPacer spreads each post's text over the estimated post interval (EMA of stream id append times, clamped to a floor and delta_pacing_max_window_seconds), in at most delta_pacing_max_chunks_per_second pieces per second; the first post is sent at once, a post arriving mid-drain is merged with the pending text, and pending pieces are flushed when the wait ends or the relay breaks
- pieces are cut at whitespace when nearby and never inside a visible character (Unicode marks, variation selectors, zero-width joiner sequences, skin-tone modifiers, flag pairs)
- DeltaRelay.send records relayed text as each event is yielded, so reconciliation sees exactly what the client received; a window of 0 restores one chunk per post
- DeltaEntry gets appended_at from the stream id; metrics horde.oai.stream.post_gap, pieces_per_post, pacing_lag and pacing_flushes record the cadence the pacer follows and the lag it adds
- docs: chat front reference, run how-to settings rows, explanation streaming paragraph
- tests: pacer scheduling, boundary rule, splitter, relay flush and bookkeeping, front-level paced stream, metric observations
A known model without an online chat worker could be admitted but the context bound would go back to the requested 2048, and the request would either be refused as PromptExceedsContext or hold a front slot for the fail-fast grace which is confusing and likely surprising.
A max_tokens above all online worker's declared chat max_length would be refused, which is different to how max_tokens is an upper bound in the OpenAI API.
- ChatAsync.validate refuses a request whose models are not served by any online chat worker with NoEligibleWorker before sizing (ensure_model_served); the front returns 503 no_eligible_worker with Retry-After 30 and x-should-retry: false, which every no_eligible_worker now sets
- clamp_max_length_to_workers lowers max_length to the largest declared chat max_length before the fit and the kudos estimate, recorded as a worker_max_length entry in context_adjustments (ChatContextAdjustmentReason) and counted by horde.chat.max_length_lowered with TEXT_LENGTH_BUCKETS
- get_highest_text_worker_limits returns the worker count and both maxima in one statement (TextWorkerLimits), read once per submission; the legacy text route is unchanged
- front: horde.stage label (submit, grace) on request errors and stream failures, horde.oai.fail_fast.rescues counter for waits the grace saved, finish_reason and max_length_lowered on horde.oai.completions
- COMMON_NON_OPENAI_SAMPLER_PARAMETERS widens the horde.parameter label vocabulary; each UnsupportedChatParameter refusal logs its truncated request path so the keys behind other are visible
- ADRs 0029, 0034, 0036 and 0037 amended; reference, how-to and explanation pages updated
- tests: unserved model refused for submit and dry run under every context fit, legacy route still queues, clamp ordering and counters, stage and rescue metrics, sampler label and log line
A freshly loaded worker kept the default upfront-kudos flag when check-in bookkeeping was debounced. The priority exception also read prioritized_users while the pop callers pass priority_usernames.
- WorkerTemplate.check_in assigns require_upfront_kudos and prioritized_users before the debounce return, reading the priority list from priority_usernames.
- The 30-second debounce still governs persisted check-in bookkeeping.
- tests/unit/test_image_worker_eligibility.py covers fresh workers with and without debounce, trusted and priority exceptions, minimum-kudos boundaries, replacement of previous policy inputs and the absence of bookkeeping writes.
- tests/unit/test_text_worker_details.py derives the text worker columns from the SQLAlchemy mapper (TextWorker minus Worker), so a new column fails until it is given a details path or listed in TEXT_WORKER_COLUMNS_KEPT_OUT_OF_DETAILS
- each column's distinct value is checked at its path after marshalling get_details() through WorkerDetails with skip_none, as the endpoints do, so a key that get_chat_details and WorkerChatDetails spell differently fails instead of dropping out of the response
- `chat` on `WorkerDetails` (`WorkerChatDetails`, `WorkerChatLimits`) carries the last pop's `model_id`, `chat_features`, `chat_limits` and `image_transports` under the pop's names, so a worker's details read back what it declared
- `TextWorker.get_chat_details` builds it; the key is absent when the worker does not accept chat jobs, since worker details skip null fields
- docs/reference/chat_requests.md gains a worker details section mapping each key to its pop field and noting the 30 second details cache
- tests in test_chat.py cover a full declaration and a worker that declined or predates chat, each on a fresh worker taken offline afterwards
tazlin