mirror of
https://github.com/Haidra-Org/AI-Horde.git
synced 2026-11-06 05:17:31 +00:00
main
92
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tazlin
|
1f17469c70 |
fix(kudos): the migration's index builds and CHECK validation run without lock or statement timeouts
- sql_statements/5.1.16.txt: section 3 sets lock_timeout and statement_timeout to 0 and resets both at the end. Before it marks an index valid, a concurrent build waits for every transaction with an older snapshot, on any table. Under a lock timeout, one such transaction fails the build after the whole table has been read, and the build leaves an INVALID index. The builds and VALIDATE CONSTRAINT take SHARE UPDATE EXCLUSIVE, which does not conflict with writes, so their waits do not hold up requests. - pg_temp.kudos_add_legacy_range sets lock_timeout to 5s as a function setting. Adding the NOT VALID CHECK takes ACCESS EXCLUSIVE, and writes queue behind its wait. The swap transactions keep their SET LOCAL timeouts. - Tests: a concurrent build waits for a transaction on users that stays open longer than any lock timeout in the file, and the run then completes without an INVALID index. - Docs: the how-to's build and CHECK steps, with a query for the transactions that hold up a build and a check of each table's freeze age. |
||
|
tazlin
|
4580ec70a2 |
fix(kudos): the ledger swap holds the applier lock and bounds its lock waits, and retention guards lost statistics dumps and restored partitions
- sql_statements/5.1.16.txt: the kudos_ledger swap takes the applier advisory lock (KUDOS_APPLIER_LOCK) before it locks users. The applier claims ledger rows and then updates users, the reverse of the swap's order. The applier only tries that lock, so it skips its cycles until the swap commits and does not deadlock with it. - After the advisory lock, each lock wait of the swap ends after 500 ms, below deadlock_timeout. A request that inserts a posting before it updates users can still form a lock cycle with the swap. The swap then fails with lock_not_available before the deadlock check runs in the request, so the request continues and the file is rerun. The file stops with an error when deadlock_timeout is 500 ms or less. - archive evict runs evict_statistics_dumps. It keeps the dumps of statistics units that are not dropped, because their raw rows are still online. It sets the object_key of each evicted unit to NULL, also after a failed batch. The command exits 1 when the kept dumps hold the tier above the budget. KudosArchive.enforce_budget takes kept_keys for this. - The tick does not clear the key of a missing statistics dump without a trace. A dumped unit whose object is missing goes to dump_failed and is dumped again on the next tick. A detached one blocks with archive object missing. - A dropped statistics unit keeps the key of a missing object, so archive audit reports the loss on every run. - _check_statistics_dumps runs with the currency check before the detach and before the drop. The raw rows of a unit stay online when its dump was removed after the last listing. - archive object missing is waivable for statistics. retention unblock --waive-dump refuses a unit of a detached partition, so the drop after a waiver starts from an attached partition. - A dataset with an attached partition that has dropped catalog units does not make a transition, and the tick returns blocked with RESTORED_PARTITION_REFUSAL. Such a partition was restored under its original name, and the catalog does not describe its rows. retention_health lists restored_partitions, and the gauge horde.kudos.retention.restored_partitions exports the count. - Tests: the migration tests run the swap against a concurrent applier cycle and a concurrent request, and against a short deadlock_timeout. The retention tests cover a vanished dump in each state and an eviction that keeps online dumps. A further test restores a dropped partition under its name. The reconcile test expects dump_failed for a vanished dump, and the rehearsal grace test expects the restored-partition block after a future reference time drops the current week. - Docs: the how-to's swap, eviction, missing object, rehearsal and rollback text, the reference's block reason, eviction, reconcile, metric, log and health tables, and the explanation's diagnosis table. |
||
|
tazlin
|
c98476c921 |
feat(kudos): the migration creates eight weekly partitions and the drop grace defaults to seven days
- sql_statements/5.1.16.txt creates eight consecutive weekly partitions from the swap time instead of one, each with the same name pattern, bounds and autovacuum storage parameters. A release without the retention tick does not create weekly partitions, so a rollback to such a release keeps inserting for eight weeks after the swap instead of one. A rerun creates only the weeks that are missing: a week that an existing partition overlaps is skipped, as is a week that ended before the current one, so a rerun neither collides with a partition the tick created at another width nor brings back a week retention dropped. - DEFAULT_DROP_GRACE_DAYS is seven. A deployment that loses KUDOS_RETENTION_DROP_GRACE_DAYS in a render keeps a reattach window, and a grace of zero has to be set explicitly. retention_health reports drop_grace_days, so retention status shows the grace in force and a deploy can be gated on it. - Tests: the migration tests expect the legacy partition plus eight weeks after a run and after a rerun, the tick's bootstrap continues from the eighth week, every week keeps the storage parameters, and a rerun recreates a dropped week while leaving a hand-made wider partition untouched. The unit and end-to-end fixtures pass a zero grace explicitly, the policy test reads the default of seven and explicit values, and the health tests read drop_grace_days. - Docs: the how-to's conversion end state, catalog step and rerun paragraph, the reference's partition layout and environment and health tables, and the explanation's grace text. |
||
|
tazlin
|
f89f00b8fd |
fix(kudos): archive checks before the drop, snapshot invalidation across a recheckpoint and a measured upload budget
- Currency retention checks every unit of the partition against the archive before the detach and again immediately before the drop (_check_currency_archive in kudos_retention.py). The object must exist, its manifest must be readable, the stored size must equal the manifest and the manifest row count must equal the unit's recorded count. A missing object or key blocks the unit, a mismatch blocks it with "archive mismatch", and an unreachable store fails the tick so a later one retries. The drop guard previously checked only for a recorded key, so an object lost after verification was noticed by the post-step listing once the partition was gone. - create_balance_snapshot runs under the retention lock and raises StaleCheckpointsError while a currency checkpoint is stale. A snapshot taken then stores applied totals short of the late rows, and a repair against it after the recheckpoint posts a false correction. - recheckpoint_from_unit records one kudos_snapshot_invalidations row, from the earliest reset unit's catalog creation to the reset, in the transaction that deletes the stale checkpoints. reconcile_balances refuses a snapshot created inside any interval in read-only and repair modes, and the snapshot rows are kept as evidence. The model is KudosSnapshotInvalidation, and 5.1.16 creates the table in a rerun-safe statement. - Archive uploads are bounded by their measured compressed size. _tier_allowance measures the tier once, the estimate is checked against it, and _MultipartSink raises ArchiveBudgetExceeded when the compressed bytes, including the final flush, would pass the allowance. The multipart upload is aborted, an object already at the key is kept, and the unit blocks with the tier's budget reason. The estimate alone let a tier pass its budget. - archive evict takes the retention lock, because an export measures its allowance from the tier size. - Tests cover the drop's archive check for a missing manifest, an unreachable store and a size mismatch, an underestimated upload for both tiers, the measured limit at one byte under, at and one byte over, the recheckpoint cascade refusing the pre-incident snapshot, and snapshot creation refused while the lock is held or a checkpoint is stale. The vanishing-object test is parametrized over hold_drops. The safety test commits and snapshots before the stale unit exists. - The reference's reconciliation section describes the lock, the invalidation interval and its width on legacy units. The schema and interface tables list the new table and the refusal. The archive section describes the measured budget and the pre-drop check. The how-to says when snapshot exits with an error. The ADR's confirmation list gains both behaviors. |
||
|
tazlin
|
59e6defc97 |
fix(kudos): attach new partitions without an exclusive lock; size the key backfill to the key lifetime
- A new weekly partition is created as a standalone table (LIKE the parent, with a bounds CHECK and the autovacuum settings) and attached, which takes SHARE UPDATE EXCLUSIVE on the parent instead of the ACCESS EXCLUSIVE that CREATE TABLE ... PARTITION OF takes, so a long reader no longer queues every insert behind partition maintenance - The listing lock and each attach run under a short lock timeout; a timeout logs a warning, leaves no standalone table behind and defers that partition to the next tick; a range whose table name is already taken is skipped - The idempotency key backfill in 5.1.16 covers key_lifetime_days (psql variable, default 30) so a deployment with longer windows backfills every key still inside its promised lifetime |
||
|
tazlin
|
aef5f44d15 |
feat(kudos): partition the event tables by created and convert in place
- kudos_ledger and kudos_stat_events are range-partitioned on created with (id, created) primary keys, so retention removes whole partitions by age - KudosPartitioning and ensure_kudos_event_partitions create upcoming partitions with the autovacuum storage parameters; init_db and the test fixtures call it after create_all - The retention catalog, stat rollup, quarantine, ledger checkpoint, daily statement and idempotency key models and their enums are declared alongside the event tables, so one migration and the ORM build the same schema - kudos_db gains a session-level retention advisory lock taken on a dedicated autocommit connection - sql_statements/5.1.16.txt creates the retention tables, backfills idempotency keys inside the online window and converts the existing tables in place by attaching each as a legacy partition behind a validated range CHECK - tests/integration/test_kudos_retention_migration.py checks the conversion, reruns, refusal guards and ORM parity with the migration |
||
|
tazlin
|
d120b922e9 |
feat(text): enforce the speed gate for chat requests and shadow it for legacy text
Legacy text requests keep reaching the slow workers that complete them, and each assignment the gate would refuse is recorded so the cost of enforcing it can be read from production. - TextWorker.can_generate_with_eligibility_inputs refuses lease_speed only for a chat request. - text_wp_speed_gate_clause passes every legacy request, so the pop query and the eligibility check agree for both request families. - Status, worker availability and the wait estimate read the same eligibility check, so slow workers count toward legacy requests. - worker_may_be_speed_gated reads only the chat declaration, so horde.pop.lease_capacity_limited counts chat pops only. - A legacy job assigned over the worker's lease capacity stores TextProcessingGeneration.over_capacity_at_assignment, increments horde.text.gate_shadow.assignments and logs its ids. The marker is written by the commit that creates the row. - horde.text.gate_shadow.outcomes counts such a job as completed, faulted or aborted, and horde.text.gate_shadow.service_seconds records a completed job's pop to submit time. - The shadow records carry horde.worker_trusted and a bounded horde.size_bucket from TEXT_LENGTH_BUCKETS in horde/metrics.py. - ProcessingGeneration.fault_and_restart returns whether the call faulted the generation, so a repeated fault submit is counted once. - sql_statements/5.1.15.txt adds the processing_gens column. - ADR 0041 records the decision with its evidence. ADRs 0038 and 0040, docs/reference/chat_requests.md and the decisions index follow it. |
||
|
tazlin
|
919dd78c5f |
perf(eligibility): select a request's eligible workers and their collections in one statement
- `_iter_eligible_workers_for_request` selects each candidate's model names, blacklist words and softprompt names as aggregate result columns and the owner's trust as an `EXISTS`, and yields `EligibleWorker(worker, eligibility_inputs)`, so the availability pass does not hydrate a per-worker collection - `Worker.can_generate` builds `RequestEligibilityInputs` and `WorkerEligibilityInputs` from the ORM and delegates to `can_generate_with_eligibility_inputs`, which holds the rules in one order with the same reasons for both paths; `can_generate_with_model_names` and `can_generate_with_softprompt_names` are removed; the direct path always loads softprompts, so its inputs match the query path's - `worker_ordered_names_clause` holds both dialects: Postgres as an ordered `array_agg`, SQLite as `json_group_array` over a subquery ordered by row id, typed JSON so the result decodes to a list - The nested trusted checks in the image worker are single conditions, and the inpainting comparison keeps its always-false result - The eligibility statement and every reader of a worker's blacklist and softprompts look rows up by `worker_id`, so `WorkerBlackList.worker_id` and `TextWorkerSoftprompts.worker_id` declare `index=True` and `sql_statements/5.1.15.txt` creates both indexes concurrently if absent, under SQLAlchemy's default names - Tests cover rule parity between the paths, the aggregate clauses on both dialects with a SQLite round trip, the per-request statement bound and both indexes; the migration test scopes its `worker_id` exclusion to the two stand-in tables |
||
|
tazlin
|
ffb26ba6b5 |
feat(chat): settle a cancelled streamed generation on the text it posted
A worker that follows the cancel is never penalised for it, and a client cannot work a volunteer for free: the Horde measures the turned-in work through the progress posts, so the cancel pays and charges on it. - The native cancel route settles each in-flight generation of a streamed chat request through `TextProcessingGeneration.settle_cancelled`: the row is locked and reloaded, the content the worker posted (`read_posted_progress`, joined from the retained stream in `seq` order, reasoning excluded, the lease's byte count when the stream lost the entries) is priced by the output estimate, and `ProcessingGeneration.cancel` pays the worker and charges the requester once without a dropped job, whichever of submit or cancel commits first; a late submit is refused as `AbortedGen` - `processing_gens.settled_character_count` records the count the settlement priced, written before `cancel` commits, so cancels can be analysed while the posted text is not kept; `get_things_count` reads it after a stored reply, so a stored reply stays authoritative; the status route does not show it - `cancel_progress_streams` sets `cancelled` on every lease that still exists in one watched transaction and appends each generation's ended marker, best effort, so a cancel succeeds without Redis and the leases stay for the acknowledgement - `chat_progress.lua` returns the lease's `cancelled` flag with `accepted` and `duplicate` replies; the progress route's 200 body (`ChatProgressAccepted`) carries it beside `accepted_seq`, so a worker stops generating and does not submit - Requests that do not stream are paid in full on submit, since the Horde does not measure partial work for them; maintenance mode's abort does not flag leases, since it lets in-flight jobs settle - Tests cover the flag, the ended marker and the kept lease, the ordered join and the byte fallback, a Redis failure, payment and charge on posted content, the floor before any post, the refused late submit, the untouched stale pass, the recorded count and the unchanged non-streaming cancel; the references, explanation and how-to describe the acknowledgement and the settlement - The explanation shows the progress lease's states and the settlement of a cancel, `docs/how-to/serve_chat_jobs.md` opens with the worker's pop loop as a diagram keyed to its steps, and `docs/explanation/kudos_accounting.md` describes the settlement of a cancelled streamed generation |
||
|
tazlin
|
28e78103ae |
feat(chat): carry a reply's reasoning in a separate field, hidden unless asked for
- A worker submits reasoning as `message.reasoning_content`, separate from `content`, and may report `reasoning_tokens`; both are stored on the generation and shown on the native status, and neither changes the charge or the reward, which stay on the server's output estimate - The progress delta accepts `reasoning_content` beside `content`; the Lua append stores it on the entry, counts it against the byte cap and includes it in duplicate detection - The front hides reasoning by default, as OpenAI's chat API does; under `horde.include_reasoning` it returns reasoning on the message and as `reasoning_content` deltas before the content, and usage carries `completion_tokens_details.reasoning_tokens` whenever a worker reported it - `include_reasoning` changes what the front delivers and leaves the work alone, so it is excluded from the operation fingerprint - The relay pins an attempt at its first delta and exposes the stream only when it relays something, so a reasoning-only first delta does not block relay for a client that did not opt in; an ended marker or a gap before exposure unpins the attempt - Reasoning alone is capped at `MAX_CHAT_TEXT_BYTES` at submit, since a kudos cost does not bound what a worker stores as reasoning; migration 5.1.15 adds `processing_gens.reasoning_tokens` |
||
|
tazlin
|
5de56325e8 |
feat(chat): offload a large request's messages to one object in a configurable bucket
A worker fetches one self-contained JSON body, and large payloads stay off the pop response. - With `HORDE_CHAT_MESSAGES_OFFLOAD_ENABLED`, a request whose messages exceed `HORDE_CHAT_MESSAGES_OFFLOAD_BYTES` has the whole array, images inline as JPEG data URLs, stored as one object at submit; the row records `chat_messages_offloaded`, so assignment does not need a store call - The `messages_url` transport: a worker that declares it receives a 120 s signed URL in place of `messages`; a vision worker still declares `url` or `base64` to receive images, and a worker that does not declare `messages_url` receives an offloaded request inline - Compression is a transport encoding of the whole object and the JSON fields are plain; `HORDE_CHAT_MESSAGES_OFFLOAD_GZIP` is off by default, and every object carries `Cache-Control: no-transform` - `R2_CHAT_MESSAGES_BUCKET` holds the objects, defaulting to `R2_SOURCE_IMAGE_BUCKET`; the key suffix `_chat_messages.json` (`CHAT_MESSAGES_KEY_SUFFIX`, `is_chat_messages_key`) routes upload, presigned download and bulk delete to the right bucket, since delete-retry keys are bare - The object is deleted with the request's images, on expiry and on a failed activation - Tests cover the threshold, both encodings, the transport choice, the bucket routing and deletion; the reference, ADR 0032 and the worker how-to describe the fetch cap and the `Content-Encoding` contract - `docs/explanation/chat_requests.md` explains the offload and shows how the pop chooses between `messages_url`, signed image URLs and inline bytes |
||
|
tazlin
|
0dd3aef41c |
feat(chat): refusal reasons, worker fault reasons and nonconformant reply accounting
- Strict tool definitions (`function.strict: true`) pass the structural rules of a strict `response_format`, with OpenAI's wording at `tools[i].function.parameters` - The front bounds every OpenAI sampling parameter with OpenAI's ranges and messages before the native call (temperature 0 to 2, top_p 0 to 1, penalties -2 to 2, `n` 1, at most 4 stops, at least one message); `max_tokens` below the native floor of 16 is refused with the floor stated; a native validation failure maps to the failing `param` - Responses carry `system_fingerprint`, a short hash of the serving worker and model; the front reference and how-to list the deliberate deviations from OpenAI - A refused worker reply records its message and return code on the request and the return code on the generation; a faulted request reports `fault_reason` and `fault_rc` on status and in the fault event, and the front's 502 states the refusal with the code in metadata - A fault that follows a `ChatSubmitNonconformant` refusal counts in the worker's `nonconformant_replies`, exposed in worker details and the moderation report, and leaves the dropped-job counts unchanged, since the worker generated and the reply failed the contract; structural refusals count as dropped jobs - A chat fault submit accepts an optional `fault_reason` from a fixed vocabulary (`prompt_overflow`, `unbuildable_grammar`, `tool_violation`, `reply_refused`, `generation_error`); a known value stores the server's sentence for it with return code `ChatWorkerFault`, and does not overwrite a reason the server recorded when it refused the same generation; an unknown value is ignored; a reason does not change how the fault is counted - The front waits while jobs remain and checks a done status without a generation again before answering `ChatRequestNoGeneration`; cancelled requests without a result settle promptly, and completion mapping requires a generation - Migration 5.1.15 adds `chat_fault_reason`, `chat_fault_rc`, `processing_gens.chat_refusal_rc` and `workers.nonconformant_replies`; docs and tests per item |
||
|
tazlin
|
c60273ee98 |
feat(chat): route by declared guarantees, fault repeated faults at submit and verify replies
A capability says a worker acts on an input; a guarantee says the reply follows that part of the contract. Requests are routed by both under a conformance mode, and the submit checks what the reply itself shows (ADR 0036). - Adopt horde_sdk's chat policy vocabulary (`ChatFeature`, `ChatGuarantee`, `ChatConformance`, `ChatWorkerLimits`) for the server's feature enum - Workers declare guarantees in `chat_features` and `tools_max_temperature` in `chat_limits`; `prompt_jobs: false` makes a chat worker chat-only, which drops the plain branch of its pop query, refuses plain prompts in `can_generate` and leaves its threads out of the text model count - Requests carry `horde.conformance`, `strict` by default; required guarantees are stored beside required capabilities in `chat_required_features`, and the forwarded tools temperature is stored once and compared with the worker's bound in the pop query and in `can_generate` under every mode - Dispatch does not forward tool definitions under `tool_choice` `none`, and a choice of one function forwards only that function as `required`; stored extras keep the client's values; model cards list `tool_choice` and `parallel_tool_calls` only when an online worker declares the matching guarantee - `FAULTED_GENERATIONS_LIMIT` is the one threshold for faulting a request with too many faulted generations; `ProcessingGeneration.fault_and_restart` claims a pending generation, requeues its job and applies the limit in one commit, for text and image fault submits and the stale-generation cleanup alike, so only the claim's winner restarts and records the job; stale image jobs keep their timeout retry policy - Status reads completion from one snapshot of the request and its generation counts; a request completes successfully only once every result is stored and every job has ended, and cancellation stops dispatch and settles after in-flight jobs finish, keeping partial results; the pop commits its taken jobs with the first generation - Native status reports `impossible_reason` when every worker the eligibility pass found was refused for its chat declaration, with the missing declarations and the exceeded limits; the front answers such a gap with 404 `no_worker_supports_features` at once, and a model without an online worker keeps the grace and the 503 - `verify_chat_reply` checks a reply against the stored extras: under both modes a call to a function the request did not offer or two calls sharing an id is `ChatSubmitInvalid`; under `strict` a call under `none`, a missing or other call under a forced choice, and content that is not the requested JSON or does not match a strict `json_schema` are `ChatSubmitNonconformant` - Several calls under `parallel_tool_calls: false` are reduced to the first, and a reply to a forced function finishes with `stop`, both OpenAI's shapes; strict schemas are validated with jsonschema (a runtime dependency), with the validator the schema declares or Draft 2020-12 and references resolved only inside the schema - Migration 5.1.15 adds `accepts_prompt_jobs`, `chat_tools_max_temperature`, `chat_conformance` and `chat_tools_temperature`; ADR 0036, the references, how-tos, explanation and return-code table describe the policy - Tests cover routing per guarantee and mode, the fault limit and concurrent claims, cancellation, image faults, the gap on status and in the front, and each verification rule per mode |
||
|
tazlin
|
279593fa8c |
feat(chat): tools and structured output, native and front
- Chat requests accept `tools`, `tool_choice`, `parallel_tool_calls` and `response_format`, `tool` messages and assistant `tool_calls`; the shapes and bounds live in `horde/chat_contract.py`, so the front validates the document the native route stores - `waiting_prompts.chat_extras` holds the applied keys as the client sent them and reaches the pop payload unchanged; `tool_choice` `none` beside tools is kept, since dropping it would leave the tools under the worker's default - `chat_required_features` is computed once at creation by `required_chat_features` and compared with the declaration in `chat_refusal_reason`, so eligibility does not load a deferred column; `tools`, `json_object` and `json_schema` join `seed`, the penalties and `vision`, and a tool turn in the history needs `tools` as well, since the worker's template renders it - Submit takes `message.tool_calls` with a nullable content and `finish_reason` `tool_calls`; a tool-only reply is a result and is charged on the content plus each call's name and arguments - The flattened prompt includes the tools, schema and tool-call JSON, so the estimate and the context fit account for what the worker renders - The front forwards the keys and messages as sent; replies carry `message.tool_calls`, and the emulated stream sends text and indexed `delta.tool_calls` in separate chunks, as OpenAI does, because KoboldAI Lite reads tool calls only from content-free chunks; the output estimate counts the tool-call JSON - Model cards list `tools`, `tool_choice`, `parallel_tool_calls` and `response_format` from the declared features - Browser clients call the front with a bearer key, so preflights are answered and every response allows any origin, exposing `Retry-After` and `x-should-retry` - Tests cover the request shapes, the stored extras, eligibility, tool-call submits and their charge, a tool round with the result sent back, a streamed tool call and a `json_schema` reply through the OpenAI SDK; the migration and reference pages describe the columns, keys, features and return codes |
||
|
tazlin
|
095703225b |
feat(chat): image inputs as signed URLs, native and front
- A user message may carry `image_url` parts (a data URL or a public http(s) URL, optional `detail`); at submit each image is decoded or fetched through the guarded client (`image.fetch_public_image`), checked against the shared caps (`MAX_CHAT_IMAGE_BYTES`, `MAX_CHAT_IMAGE_COUNT`, `MAX_CHAT_IMAGE_COUNT_UNTRUSTED`), normalised and stored as `<request id>_chat_<n>.jpg`; the stored part references the key, the row does not hold bytes or the client's URL, and the flattened prompt renders `[image]` so the estimate and moderation see it - Workers declare `image_transports` (`url`, `base64`) and the `vision` feature; a request with images is offered only to a worker declaring both, through a JSONB containment test in the queue query so the deferred messages are not loaded; the transport used is recorded on the generation - At pop, after the generation row commits and the queue lock is released, a `url` worker receives a download URL signed for at least the job TTL and 600 seconds, and a `base64` worker receives a `data:image/jpeg` URL (ADR 0032) - Each image is taxed as an extra source image through `GenerateTemplate.get_activation_kudos_adjustment`, counted in the dry-run quote - Uploads happen after the inactive row exists, so an image does not exist without a row; a failed activation, a lost operation-key race and a store failure delete the submission's images, row deletion and expiry delete them, and a failed delete is queued in the bounded Redis set `horde:chat:image_delete_retry` and retried by the cleanup tick, so the bucket is not listed - `MAX_CHAT_BODY_BYTES` is 36 MiB so five base64 images fit beside the text allowance; return code `ChatImageStorageUnavailable`; the migration adds `workers.image_transports` and `processing_gens.image_transport` - The front forwards image parts verbatim, refuses them on any role other than `user` by path, and refuses an oversized data URL or too many parts with the native codes `SourceImageSizeExceeded` and `TooManyExtraSourceImages` before forwarding; image bytes do not count toward the text allowance - The front takes the in-flight slot before reading the body and builds the credentials before it too, so a source at its cap or a malformed authorization does not cost body bytes; a client that disconnects during the upload is answered 499 and counted as a refusal - Native 400s that quote a message path surface it as `param` - Tests cover eligibility, both transports, the lock order, the address guard with a fake resolver, the tax, deletion and the retry set, JPEG encoding of RGBA, LA and palette images, front screening and the SDK conformance module's signed URL with `detail` preserved; `docs/reference/chat_requests.md` and `outbound_requests.md` describe the format, the transport and the fetch limits - Input images on a chat request are not screened by the server or by a chat worker, while image workers screen their output and report `worker_csam`; the chat requests and moderation operations references state this |
||
|
tazlin
|
22f8740e2e |
feat(chat): chat requests as a text request subtype
A chat request carries an OpenAI `messages` array to a worker that renders it with its model's chat template; every other part of it is a text request, so queueing, pricing, moderation and limits run on the text path unchanged (ADR 0019).
- `horde/chat_contract.py` holds `ChatHordeOptions`, the model of a request's `horde` object, and the caps `MAX_CHAT_MESSAGE_COUNT`, `MAX_CHAT_TEXT_BYTES` and `MAX_CHAT_BODY_BYTES`; it imports only pydantic, so the native route and the separate front validate against one model; `INTERNAL_HORDE_OPTION_KEYS` (`dry_run`, `operation_key`) are refused in a public body
- `KNOWN_RC` and `README_return_codes.md` add `UnsupportedChatParameter`, `ChatRequestConflict`, `ChatAdmissionDisabled`, `ChatDeclarationInvalid`, `ChatSubmitInvalid`, `ChatModelMismatch`, `UnknownChatModel`, `OperationKeyConflict` and `RequestTooLarge`; `exceptions.ServiceUnavailable` is a 503 with a return code for a route a deployment switches off
- `sql_statements/5.1.15.txt` (version 5.1.15) adds `waiting_prompts.is_chat` and `messages`, the worker's chat declaration columns and the `processing_gens` reply columns; every statement is idempotent and a metadata-only add under a lock timeout; `is_chat` and `accepts_chat_jobs` default to false, so existing rows stay text requests and workers that predate chat jobs are not handed one, and `messages` is deferred, so only an assigned request loads it
- The `assigned_*` generation columns hold the worker's declaration as it stood at assignment, so a later declaration change does not redefine work already handed out
- `POST /v2/generate/chat/async` (`ChatAsyncGenerate`) takes an OpenAI chat-completions body; `horde/classes/kobold/chat_request.py` validates the supported subset, refuses any other key, role, value or content part with `UnsupportedChatParameter` and the key in the message, and maps the request onto text params, model names and worker targeting; the text params are checked against the text payload model, so a chat request accepts the ranges a text request does
- `ChatCompletionRequest` refuses NaN and infinite sampling values, and `parse_chat_request` refuses NUL in every stored string with `NulByteInPayload` before a row exists, because PostgreSQL rejects NUL in text columns and in JSONB
- The flattened messages are stored as the prompt and feed only the token estimate, kudos and moderation; `mirror_prompt_rewrites_onto_messages` applies a moderation replacement to every message text, so a worker does not receive text validation did not see
- `TextAsyncGenerate` splits into `parse_request_args`, `queue_generation` and `get_waiting_prompt_arguments`, which the chat route overrides; a chat request's context grows only up to the chat context of the workers serving its model
- `ChatAsyncGenerate.ensure_model_known` refuses a model that neither the text model reference lists nor an online chat worker serves with 404 `UnknownChatModel`, after authentication, maintenance and the concurrency check and before context sizing, so a client can tell an unknown model from one without a worker online
- `waiting_prompts.operation_key` and `operation_fingerprint`, unique per user through a partial index built concurrently, let a native caller claim one submission: a repeated key with the same fingerprint (`build_operation_fingerprint`, canonical JSON of the normalised request) returns 200 with the existing id and does not create anything, admission off included; a changed request is 400 `OperationKeyConflict`; `TextWaitingPrompt._activate` writes the claim in the commit that activates and taxes the request, and `activate_waiting_prompt` resolves the unique-violation race by reading the claim again and dropping the losing row
- `read_chat_request_body` refuses a body above `MAX_CHAT_BODY_BYTES` with 413 before parsing, chunked bodies included; `waiting_prompts.context_adjustments` records how `context_fit` grew the context
- `GET` and `DELETE /v2/generate/chat/status/<id>` (`ChatAsyncStatus`) report `RequestStatusChat`; `get_text_wp_by_id` filters on `is_chat`, so each route family reports the other family's ids as not found; `HORDE_CHAT_ADMISSION_ENABLED` refuses chat submits with 503 `ChatAdmissionDisabled` while queued requests are still served
- Text pop takes a chat declaration (`chat_jobs`, `chat_features`, `chat_limits`, `model_id`); `TextJobPop.validate_chat_declaration` refuses one without limits or `model_id`, or with a `model_id` outside the worker's models; `TextWorker.check_in` assigns the declaration on every pop before the check-in debounce returns, so a pop without `chat_jobs` withdraws it, and unchanged values do not write the row
- Chat jobs run under the chat limits and text jobs under the text limits: the pop query offers a chat request only to a worker whose `chat_model_id` the request lists and whose limits fit it, `TextWorker.chat_refusal_reason` refuses a request needing a feature the worker did not declare, and the availability query applies the same filter, so a request's status counts only workers that can serve it
- A chat job's payload carries the messages and every sampling field with its default filled (`build_chat_job_payload`), added at pop from the deferred column; the pop response carries `chat: true` under `GenerationPayloadChat`, and a text job's response is unchanged
- A chat job is submitted with `message`, `finish_reason` and `model_id`, plus optional `native_finish_reason`, `prompt_tokens` and `completion_tokens`; `prepare_chat_submission` refuses a missing or invalid message or finish reason with `ChatSubmitInvalid` and another model with `ChatModelMismatch`, and replaces NUL with U+FFFD as a text submission does
- The message content is stored as `generation`, so every predicate that reads `generation` treats chat and text alike; `get_submission_columns` writes the reply in the compare-and-set update that claims the generation, so a refused submission does not write any of it
- `TextProcessingGeneration.get_charge_basis` is `OUTPUT_ESTIMATE`: chat generations are charged and rewarded by the text formula, and reported token counts are stored and shown without changing the charge (ADR 0027)
- `horde/chat_signal.publish_chat_event` publishes `{"id", "event", "generations"}` on `horde:chat:<request id>` for each `ChatEvent` (generation, done, cancelled, faulted) from logical database 7, after the commit that settled it; a message does not carry text, and publishing is best effort, since a subscriber that misses one still reaches the result through the status route
- `horde/image.py` and `horde/r2.py` normalise and store chat images as JPEG (`load_chat_image`, `normalize_chat_image` with the longest side at most `MAX_CHAT_IMAGE_SIDE` 2048, quality 90 with transparency flattened onto white, since the multimodal loaders workers run do not decode WebP), with upload, download and presigned-URL helpers; the request route refuses image parts, and the helpers serve the signed-URL transport of ADR 0032
- Tests: `tests/unit/test_chat_request.py`, `tests/unit/test_text_worker_eligibility.py`, `tests/integration/test_chat.py` (queueing, pop, submit, status, cancel, expiry, fault, charge, admission, signals, the model check, operation key claims and races, the body bound), `tests/integration/test_chat_migration.py` and the fake chat worker in `tests/integration/_fake_chat_worker.py`
|
||
|
tazlin
|
5e0456b616 |
feat(moderation): filter prompt events by reason
- `GET /operations/moderation/prompts` takes a repeatable `reason`; an unknown value is `400 InvalidModerationReason` - `(reason, id)` index on `prompt_moderation_events` in the model and `sql_statements/5.1.14.txt`, so a page of a rare reason does not walk the common ones - `hold_source` on each listed event (`note`, `account_status` or null), computed in the page query from the same predicates `_under_moderation_action` applies, so the listing never disagrees with retention; nothing is stored and there is no release - the migration test applies every migration that changes a moderation table, now 5.1.12 and 5.1.14 - version 5.1.14 |
||
|
tazlin
|
238d52de80 |
feat(styles): record the style each generation ran under
- WaitingPrompt.style_id records the style a request runs under, the drawn style for a collection, and record_image_statistic and record_text_statistic copy it to image_gen_stats.style_id and text_gen_stats.style_id, so style use can be grouped per style - the columns are nullable UUIDs with no foreign key, so a deleted style leaves its statistics in place; sql_statements/5.1.13.txt adds them as catalogue-only changes - integration tests check a queued text and image request records its style, the drawn style under a collection, and none when unstyled; unit tests check the text statistic copies it |
||
|
tazlin
|
69324c1c3e |
feat(styles): add the style contract module and columns
- A style can carry two optional declarations. A parameter policy sets which of a request's params are kept when the style is applied: `none` keeps the style's params and ignores the request's, `listed` takes only the params the policy lists, `all` takes any of them. Template fields declare the placeholders the style's prompt accepts beyond `{p}`, each with a description and whether a request has to supply it.
- A policy's `ceilings` is a map from param to the largest value a request may set. It limits the request's value only; the style's own params are not checked against it, and a value above a ceiling is rejected rather than clamped, because clamping would change the cost of a request silently. A ceiling on a param the current mode does not let a request set is accepted and has no effect, except for `n`, which always comes from the request and is capped under every mode.
- `StyleContractVocabulary` holds, for one style type, the params a policy may make overridable and the params it may cap, with the range each ceiling may take. It is passed to `StyleParameterPolicy` through the pydantic validation context, so `horde/classes/base/style_contract.py` does not import the API layer and a stored policy can be loaded without one. Text styles cap `max_length`, `max_context_length` and `n`; image styles cap `width`, `height`, `steps` and `n`.
- `styles.parameter_policy` and `styles.template_fields` are nullable JSON columns, so a style that declares neither behaves exactly as before. `sql_statements/5.1.13.txt` adds them by hand; a nullable column with no default is a catalogue-only change and can be applied before the code that reads it is deployed.
- Seven return codes are added to `KNOWN_RC` and `README_return_codes.md` for the rejections the endpoints and the request path raise.
- No endpoint reads the columns yet.
|
||
|
tazlin
|
448ee7e1f5 |
feat(database): widen extra_priority column to BIGINT
This prevents very high kudos balance accounts from seeing 500s when trying to generate. |
||
|
tazlin
|
3703c4da8b |
feat(api): expose paused-worker review and worker suspicion history to moderators
- Record each worker suspicion in worker_suspicion_events from report_suspicion, keyed by worker UUID without foreign keys, so the history outlives suspicion resets, worker deletion and account wipes; the stored name is cut to the workers table's length (MAX_WORKER_NAME_CHARACTERS), since check_for_bad_actor reports an overlong name before it shortens it. - Keep the history unless HORDE_WORKER_SUSPICION_RETENTION_DAYS is set, then delete rows past that window in a bounded step of apply_evidence_retention (RetentionPolicy.worker_suspicion), keeping the history of flagged or suspicious owners like every other step; the window is at most 36,500 days (MAX_WORKER_SUSPICION_RETENTION_DAYS), so the cutoff stays inside the date range. - Disclose what the history records and its period in the privacy document, rendered from the same policy (RetentionPolicy.worker_suspicion_days). - Add GET /api/v2/operations/moderation/worker_suspicion_events, filtered by worker or owner and paged by before_id; each row carries worker_id, user_id and suspicion_id, since worker names are not unique over time. - Define the online window (WORKER_ONLINE_SECONDS) and the name limit (MAX_WORKER_NAME_CHARACTERS) once in worker.py, used by is_stale, check_for_bad_actor and set_name as well as the overview. - Add the paused-worker list to the moderation overview, filtered by worker type, suspicion reason and online state, with a sort and paused_workers_total. - Add InvalidWorkerID and InvalidSuspicionID return codes. |
||
|
tazlin
|
eeac473439 |
feat(api): expose promotion review to moderators
- Add User.promotion_status as the automatic trust decision; check_for_trust promotes on it and skips accounts already trusted. - Add User.promotion_queue_criteria, the same criteria as a SQL filter beside it, so each queue is one bounded query; the equivalence test holds the two forms to the same result. - Add the ix_users_evaluating_kudos_rank partial index so each queue is an index range scan ordered by evaluating kudos. - Add moderator-only GET /api/v2/operations/moderation listing promotion-eligible and suspicion-blocked accounts with their suspicion reasons (horde/database/moderation.py). - Report promotion_threshold, suspicion_threshold and worker_suspicion_threshold once at the top of the overview rather than on every row; each row carries only what differs between rows. |
||
|
tazlin
|
a98cd39776 |
feat(moderation): keep evidence text in object storage behind per-subject countermeasures
- Capture keeps the row as an outbox (text_state pending, text_sha256, text_chars); the quorum applier moves text to evidence/{id}.json every 5 s under an advisory lock and sets stored in one conditional update, deleting objects of rows retention cleared meanwhile and keeping objects another run recorded
- Retention deletes a batch's objects in one bulk call inside the row lock and flags only successes; a store outage holds back only rows with a stored object
- The listing returns text_state, a 10-minute presigned text_url for stored events, and the digest and length; stages stay inline while pending
- The evidence store client needs its own endpoint, bucket and credentials and is off without all four; legacy prompt and image buckets are untouched
- Rejections are counted per IP subject per hour: past HORDE_MODERATION_SUBJECT_TEXT_CAP_PER_HOUR events keep digest and length only; past HORDE_MODEL_REJECTION_TIMEOUT_THRESHOLD model rejections time out the address, every one in raid mode; moderators are exempt; one moderator notice per subject, action and hour carrying the latest event id and review link; countermeasures degrade to none on Redis errors
- Capture, backlog, rejection-rate, distinct-subject and observation-timestamp instruments for flood detection
- ADR 0016, reference doc with the edge rule text, privacy document gated on the configured store, env template
|
||
|
tazlin
|
13e7c281fa |
feat(moderation): decay evidence under an anonymizing ceiling
- Decay evidence in bounded passes instead of deleting it: prompt text after HORDE_MODERATION_TEXT_RETENTION_DAYS (none by default, so text stays until the ceiling and unreviewed evidence stays reviewable), the address after HORDE_MODERATION_IPADDR_RETENTION_DAYS (30), and every identifying value and the address pseudonym at HORDE_MODERATION_EVIDENCE_CEILING_DAYS (365, never higher) (RetentionPolicy, load_retention_policy). Notes are never deleted. - Resolve MODERATION_RETENTION_POLICY when apply_evidence_retention runs, so the pass applies the policy in force; apply_moderation_retention runs it once an hour on the primary node, since rows under moderation action never leave the pending indexes and are rescanned on every pass. - EVENT_COLUMN_RETENTION assigns every event column a retention fate and the passes derive their updates from it; import refuses a column without one. - Flag each removal (text_redacted, ipaddr_redacted, anonymized) so a null reads as removed, with a partial index per step that keeps each bounded pass off finished rows; user_id becomes nullable because the ceiling removes it. - Bound user_problem_jobs by the same passes: the address and origin_text are nulled past the address window through ix_user_problem_jobs_ipaddr_pending and the row deleted past the ceiling; origin_text on events follows the address fate. The problem-job alerts read only the last hour or day. - Exempt records under moderation action from every step, with no cap (_under_moderation_action, _account_restricted): an event with a note, or whose account is flagged or suspicious as User.is_suspicious judges it; worker report records use the account part. A restriction cannot be enforced against an account whose record is gone. Each step repeats its due predicate in its own UPDATE, so a note committed while the anonymize batch waited on its row lock exempts the event. - User.wipe() never touches evidence, notes or user_problem_jobs; the account row is renamed and the records keep their user_id. - Document the retention variables in .env_template; lock the parsing, the column fates, each step, its batch bound and idempotence, the exemption, and evidence surviving a wipe in tests. |
||
|
tazlin
|
1fbc7e448e |
feat(moderation): record prompt evidence when a submission is rejected
- Record filter, NSFW-model and worker CSAM rejections in prompt_moderation_events through record_prompt_evidence on a separate connection, so a rolled-back rejection keeps its evidence; a repeated worker report returns the existing event through the unique job_id. - Capture from GenerateTemplate._record_prompt_rejection and User.record_problem_job, keeping each prompt stage distinct and never filling one from another; the effective prompt is null on a rejection. - Store the request origin as its IP subject and its pseudonym (ip_subject_key); an origin with no subject (free text in a trusted proxy's Proxied-For header) keeps its text as origin_text, clipped to MAX_ORIGIN_TEXT_CHARACTERS (subjectless_origin_text), on the event and on the worker report record. Clip each prompt stage at 16,000 characters (text_truncated). - Drop the user_problem_jobs.worker_id foreign key and both ORM relationships, so deleting a worker keeps the reports it made: they are evidence about the reported account, not about the worker. 5.1.12 drops the constraint. - Name the reasons and outcomes as PromptModerationReason and PromptModerationOutcome, and default capture times to UTC on every dialect, matching the migration, through UtcNow in horde/classes/base/sql_constructs.py, which holds per-dialect SQL expressions any model can use. - Add the prompt_moderation_events and prompt_moderation_notes tables to 5.1.12 with an index per subject filter (account, proxied account, pseudonym, unkeyed address, worker) and on the capture time; MAX_NOTE_CHARACTERS sizes the note column. - Lock the models to the migration (test_orm_models_build_the_same_moderation_tables_as_the_migration) and the capture paths in tests. - Document in .env_template that secret_key keys the address pseudonym and must not rotate. - Rewrite the problem-job alerts (User.record_problem_job) to carry the event ID, the count, the models and LoRAs and a frontpage link (HORDE_MODERATION_FRONTPAGE_URL, moderation_event_url) instead of the prompt and the address, since a Discord message outlives evidence retention and Discord drops content over 2,000 characters; IP alerts carry the first 12 hex characters of the pseudonym, and a report whose origin has no subject sends no IP alert, since it would pool with every other subjectless report. Count the IP daily window over a day. - Bump HORDE_VERSION to 5.1.12. |
||
|
tazlin
|
f98b6d478b |
feat: judge every IP origin as one subject
- Add CounterMeasures.parse_ip_subject and CounterMeasures.ip_subject, the one normalization of a request origin: an IPv4 address as itself, an IPv4-mapped address as its IPv4 address, an IPv6 address or a network of /64 or narrower as its /64, since IPv6 clients rotate addresses inside it; any other value, such as free text in a trusted proxy's Proxied-For or a wider network, has no subject. - Add CounterMeasures.ip_subject_key, an HMAC-SHA256 pseudonym of a subject under the deployment's secret_key with a domain-separation label; IP_SUBJECT_KEY_SECRET is None, with one startup warning, while the secret is unset or a published placeholder (PLACEHOLDER_SECRETS), since the IPv4 space is enumerable. - Block worker IPs through ip_subject, so a mapped IPv4 worker address times out that IPv4 address rather than ::/64, and a worker origin with no subject sets no timeout and logs a warning. - Record user_problem_jobs addresses through ip_subject and drop NOT NULL on user_problem_jobs.ipaddr, since an origin with no subject stores no address. - Remove extract_ipv6_subnet; tests/unit/test_ip_subject.py covers the subject and the key, and test_worker_ops the worker IP block. |
||
|
tazlin
|
ac541c7ae4 |
feat(api): return the original submission from the request endpoints
The request endpoints added for the submitting key returned WaitingPrompt.prompt, which is the worker input after the replacement filter has rewritten it. A client could diff its own submission against the response and learn exactly what the filter replaces, which cannot be allowed. The original submission is now stored on its own column and returned to workers, but not to users querying their own request details. - Add WaitingPrompt.submitted_prompt, the parsed submission before styles or moderation. It is nullable, deferred, and raiseload, so ordinary queries, joins and relationship loads never fetch it, and an accidental lazy access raises instead of silently pulling the text into worker payloads, webhooks or share metadata - Capture it in GenerateTemplate._post_inner before the subclass validate() applies styles, and pass it into every waiting prompt constructor. The constructor commits, so setting it afterwards would leave a committed row without provenance - Read only through read_privileged_submitted_prompt(purpose=...), a scalar select that leaves the attribute unloaded on the instance. SubmittedPromptPurpose is a required keyword identifying the reading consumer, so every caller that handles the text is findable by grep - get_submitted_request returns the original prompt. A row without provenance omits the prompt entirely. The effective prompt is never a fallback because that would reopen the leak for rows the filter touched - Both request endpoints send Cache-Control: private, no-store - ALTER waiting_prompts in sql_statements/5.1.12.txt: nullable, no default, so the add is metadata-only, under a 5s lock_timeout that fails instead of queueing an ACCESS EXCLUSIVE lock behind a long transaction. Existing rows stay unknown; provenance is never backfilled from prompt - Document the contract in docs/reference/prompt_provenance.md, and cover deferral, the privileged read, the original surviving a filter rewrite, the omission on legacy rows and absence from every existing export in tests, with a migration test that the script is additive and repeatable |
||
|
tazlin
|
579f40d6c0 |
feat(api): let the submitting API key read a request's parameters back
A request ID is a bearer token: anyone holding it can poll progress and collect results, and that is intentional. It also meant the prompt and parameters behind a request were unreadable to the client that submitted them, and that the owner of a shared key could enumerate every request ID funded through that key from their own account view. This adds a credentialed read of the submission, and closes the enumeration path at the same time so the new read cannot widen it. - Add GET /v2/generate/request/<id> and /v2/generate/text/request/<id>. The response has the shape of the generation input so it can be posted back to the async endpoint as-is; defaults the Horde filled in come back filled, and source images come back as their storage references rather than base64 - assert_submitting_key decides who may read: a personal key reads requests it submitted and requests submitted through its own shared keys, a shared key reads only what it submitted, and the anonymous key is refused. Any other key gets 403 NotRequestOwner - WaitingPrompt.get_submitted_request rebuilds the body from the row, with image and text overrides restoring n, seed, seed_variation and softprompt from their own columns. derive_submitted_request_model derives the response models from the input models, dropping dry_run, allow_downgrade, replacement_filter and style, which leave no trace on the row - Account views no longer list shared-key-funded request IDs in active_generations, even for moderators. Cache keys for those views are versioned so entries written before this change are ignored, and refresh_cache clears the new keys - GET /v2/sharedkeys/<id> accepts the owner's personal key and then adds an active_usage summary (request, queued, processing and finished counts) without IDs or contents. Credentialed lookups bypass the shared response cache and every response is Cache-Control: private, no-store - Add a partial index on waiting_prompts.sharedkey_id, declared in the ORM and built concurrently in sql_statements/5.1.12.txt, so the usage summary is one indexed SELECT - Document both endpoints in README_integration.md, register NotRequestOwner in KNOWN_RC and README_return_codes.md, and cover the key check, body reconstruction, privacy and cache isolation in tests |
||
|
tazlin
|
e0b0c361fe | perf: index kudos ledger on hot path | ||
|
tazlin
|
6fde1004f6 |
perf: flatten per-row lookups on the pop, teams, user and applier paths
Several read paths issued one query per row they walked, so their cost scaled with queue depth, worker count or batch size rather than with the number of relationships involved. Each is now loaded as a set. - Job pop (image and text) eager-loads the models, tricked_workers and workers relationships can_generate() reads, instead of three lazy SELECTs per candidate - GET /v2/teams loads each team's owner, workers, their owners and their models up front, and get_all_models reads the models relationship rather than the per-worker redis cache - assign_monthly_kudos loads every user's roles in one query - store_known_image_models looks the whole reference up in one query; the single-record path is now lookup, upsert, commit with the upsert shared by both - User.get_details reads live request ids with a two-column query instead of loading whole waiting_prompts rows - The kudos applier consumes a cycle's reservations with one locked read and one set-based write (consume_reservations) rather than one locked SELECT per hold - Add partial indexes for the applier health aggregate over quarantined stat events and the evaluation drain's distinct-user scan, with sql_statements/5.1.10.txt to build them concurrently and set fixed-count autovacuum thresholds on the two queue tables |
||
|
tazlin
|
cf90cd700c |
fix(perf): use explicit IS FALSE predicates in drainable index
Prior to this, the index was effectively a NOOP and led to the multi-million row kudos ledger to be scanned for just the pending events. This brings an observed 30s kudos projection applier lag down to 2s. |
||
|
tazlin
|
c4e90c40fe |
fix/telemetry: expose current kudos applier health and index drainable work
- replace histogram-sampled queue, lag, and reservation state with synchronous gauges so telemetry reflects the quorum applier's current state - split pending counts and oldest-row age between currency and statistic queues, expose recent quarantine age, batch and catch-up configuration, and cycles used per tick - refresh the quorum observation timestamp and lag after every committed catch-up cycle so long drains remain visibly healthy and former-quorum gauges can be excluded - replace the old unapplied-stat index concurrently with a drainable predicate that keeps retained quarantine evidence out of the hot queue scan - test migration idempotency from the previous production index state and verify telemetry remains fresh through multi-cycle catch-up - record the deterministic quarantine and worker-deletion failure policy in ADR 14 and link its amendments from the existing projector and foreign-key decisions |
||
|
tazlin
|
6928bf1e1c |
fix: isolate invalid kudos projections and worker deletion races
- quarantine deterministic malformed stat events with bounded reason codes instead of allowing one poison event to roll back the global projector; quarantine every claimed row from the same business event to prevent partially materialized statistics - add the v5.1.9 additive quarantine schema and expose retained quarantine evidence, age, reason-specific counters, and post-commit fold telemetry for alerting and operational diagnosis - serialize worker deletion with claimed stat events and worker projection locks, preserving immutable audit history and surviving team attribution while skipping worker_stats materialization after a worker disappears - validate projection dimensions and reject non-finite amounts at both kudos emitters before invalid data can enter the accounting queues - keep success metrics behind the projection commit so rolled-back batches cannot appear as productive throughput - document that quarantined_rows is cumulative retained evidence rather than unresolved incident state, and use newly quarantined events for paging - cover poison-event isolation, migration idempotency, failed-transaction telemetry, missing targets, and the exact claimed-event versus worker-delete PostgreSQL interleaving |
||
|
tazlin
|
fbff5c79db |
feat(scheduling): observe worker dispatch lifecycles
- Assign one dispatch ID to every generation returned by the same worker pop so a batched response represents one occupied worker thread. - Record the queue-selection context and compatible worker state at assignment time so scheduling forecasts can be compared with actual outcomes. - Emit one capacity return only after the complete dispatch batch becomes terminal, avoiding inflated clearance measurements for multi-generation requests. - Add the supporting schema and indexes needed to keep active dispatch lookups bounded. |
||
|
tazlin
|
4c30e959d3 |
feat(metrics): add request lifecycle metrics and validation tracking
- Introduced new metrics for tracking request lifecycle stages including time to first start, time to completion, and time to expiry. - Added counters for request outcomes and validation of scheduling estimates. - Implemented functions to record request estimate validations and stall validations with appropriate error handling. - Updated existing code to integrate new metrics into the request processing flow. test(metrics): enhance tests for request lifecycle metrics - Added tests to validate the recording of request lifecycle metrics and outcomes. - Ensured that metrics are correctly recorded during text generation requests. - Verified that request estimate validations handle various scenarios and edge cases. docs: add documentation for request feasibility and queue pressure - Created a new document detailing the concepts of request feasibility, queue pressure, and related metrics. - Provided guidance on interpreting API response fields and recommendations for client behavior. - Included validation criteria for scheduling estimates and stall signals. feat(scheduling): implement shadow forecasts for request scheduling milestones feat: Enhance request scheduling and terminal outcome tracking - Introduced `RequestTerminalOutcome` enum to classify request states (completed, cancelled, faulted, etc.). - Updated `TextWorker` and `ImageWorker` classes to evaluate generation eligibility using softprompt and model names respectively. - Modified `check_waiting_prompts` to handle terminal outcomes and ensure proper telemetry recording for request expirations. - Implemented scheduling event queue for handling request forecasts and observations, including start, completion, expiry, and cancellation events. - Added tests to validate the new scheduling features and ensure that cancellations do not affect terminal validations. - Updated database schema to include `terminal_outcome` and `terminal_recorded_at` columns in `waiting_prompts` table. - Enhanced Redis interactions for storing and processing scheduling forecasts. - Bumped version to 5.1.8 and added SQL statements for schema changes. |
||
|
tazlin
|
3761401381 |
refactor: sampler work policy and execution contracts
- Replaces evaluation-per-step accounting with SDK sampler work primitives across request usage, TTL, upfront gating, and `limit_max_steps` matching. - Adds a new sampler work policy module, publishes execution contracts/work profiles in sampler constraints, and threads worker-advertised `sampler_execution_contract_version` through check-in/pop with fail-closed handling for adaptive ceilings. - Adds a migration for persisting worker execution contract version, updates API schema/docs messaging. - Expands integration/unit coverage for adaptive work estimation, downgrade behavior, and forecasting consistency. - Bumps `horde_sdk` to 0.29 |
||
|
tazlin
|
cbe6b9cffc |
fix(alchemy): enforce unique annotation capabilities
- adds a unique worker-and-type index that also supports capability lookups and makes the ORM schema match the production migration - normalizes nullable and duplicate rows from earlier migration drafts, deduplicates live check-ins and accurately qualifies concurrent-index rerun behavior - covers repeated advertised values through the PostgreSQL-backed persistence path while retaining the existing 100-capability limit |
||
|
tazlin
|
f57a799de9 |
feat: additional annotation control types and gating
- Introduces unified ControlNet/annotation type constants and extends API models/parsers to support `annotation` forms, `annotation_types`, and `allow_extended_controlnet`. - Dispatch logic now gates extended control types by bridge capability/version and worker opt-in, while preserving legacy `hough` behavior. - Alchemy annotation requests now validate `payload.control_type`, return image-output handling via R2, and use detector-class kudos buckets for pricing. - Worker persistence/details were extended for advertised annotation types, - Unit/integration coverage for relevant behavior was added, including a cross-repo SDK wire-contract test for request/response compatibility. fix: include sql migration commands tests: reconcile annotator tests with torch->numpy change |
||
|
db0
|
fbd5ed465e | fix: Set wp_models sequence to BigInt | ||
|
tazlin
|
9f84937f72 |
perf: supporting indexes for the pop and validity EXISTS probes
- wp_models, wp_allowed_workers, and worker_models carried no index beyond their integer pkeys, so the per-WP/per-worker EXISTS subqueries introduced for the semi-join conversions planned against full scans; the inflated cost estimates also pushed every pop candidate query past jit_above_cost, adding JIT compilation per query (observed 67ms with 31 JIT functions vs the sub-10ms join it replaced) - composite indexes (wp_id, model), (wp_id, worker_id), (worker_id, model) declared on the ORM models and in sql_statements/5.1.4.txt as CREATE INDEX CONCURRENTLY for live application - the pop candidate queries walk ix_waiting_prompts_extra_priority backward discarding the expired majority row by row (measured ~3.3k buffers to find ~230 live rows); the partial index (wp_type, extra_priority DESC, created ASC) WHERE active AND NOT faulted AND n > 0 contains only the live queue and streams it presorted |
||
|
tazlin
|
9ac31434ef |
perf(stats): watermark-incremental all-time counts
- compile_imagegen_stats_models/compile_textgen_stats_models carry per-model all-time counts forward from the previous snapshot and fold only rows past last_stat_id; day/month windows and model_id/model_state are recomputed each run since windows slide and model registration can reclassify - merge is a UNION ALL of baseline, id-range delta, and zero-contribution rows for every model in the 30-day window, so a model whose history sits inside the watermark gap still appears and self-corrects next run - measured on production data: 92.5s + 111s nightly becomes ~0.6s + ~1.5s (the irreducible 30-day windowed recompute); verified against ground-truth recomputes across bootstrap, no-op rerun, gap-fold, registration-flip, and pruning scenarios on a disposable postgres 16 - sql_statements/5.1.3.txt (last_stat_id columns on both compiled models tables, catalog-only ALTERs) must be applied before the release that carries it |
||
|
tazlin
|
73c3ea8737 |
fix(stats compiles): watermark-incremental totals and BIGINT token accumulators (5.1.2)
- compile_imagegen_stats_totals/compile_textgen_stats_totals carry all-time totals forward from the previous snapshot and aggregate only rows past last_stat_id, replacing ~10 full-table scans per minute over the never-pruned 55/78 GB stats tables (~32s -> ~2.7s DB time/min); window aggregates collapse into one FILTER pass - compiled_text_gen_stats_totals token columns and total_requests widen to BIGINT: the 30-day token sum passed the int32 ceiling on 2026-07-18 and every run since aborted on INSERT, freezing the text totals endpoints on an 11-day-old snapshot - sql_statements/5.1.2.txt (watermark columns, BIGINT widenings, ix_users_contact, ix_users_proxy_passkey_present recorded idempotently) must be applied before this release starts, since stored procedures auto-apply on startup and the new bodies reference last_stat_id - HORDE_VERSION 5.1.2 |
||
|
tazlin
|
d9faa90061 |
perf: index the suspicion tables by owner column, migration 5.1.1
- user_suspicions (713k rows) and worker_suspicions (570k rows) carried only their primary key, so every suspicion lookup was a sequential scan: User.is_suspicious/Worker.is_suspicious on request paths, and the kudos applier promotion-eligibility scan, whose per-candidate suspicion count did one full-table scan per candidate (EXPLAIN on prod: 22ms and 16.5M buffer hits per subplan loop) - index=True on UserSuspicions.user_id and WorkerSuspicions.worker_id; sql_statements/5.1.1.txt creates ix_user_suspicions_user_id and ix_worker_suspicions_worker_id for existing databases (operator-applied, matching db.create_all() names); HORDE_VERSION bumped to 5.1.1 |
||
|
tazlin
|
edf0a8ac3c |
feat: event-sourced kudos accounting with reservations and shadow cutover
Kudos mutations previously updated shared user/worker/stat rows inside request transactions, making hot user rows part of many unrelated lock graphs (the production pop/submit convoy). Movements are now appended as typed postings and folded asynchronously. - add append-only kudos_ledger (currency) and kudos_stat_events (display/counter facts), folded into users/workers/teams/stats by a single advisory-lock-serialized projector (apply_pending_kudos) with bounded batches and a bounded per-tick catch-up loop - add per-payer reservations (kudos_reservations) under payer advisory locks; available_kudos reads balance, holds, and queued debits in one statement snapshot so admission cannot overspend during projector lag - add shadow/ledger dual mode via kudos_ledger_control: shadow retains inline projection (kudos_legacy_projection, the deletion boundary after cutover) while recording audit events; mode reads take a control-row key-share pin memoized per transaction; the ledger-to-shadow transition drains the final tail atomically - transfers accept an optional Idempotency-Key header; zero-amount transfers are rejected (InvalidKudosTransferAmount); upfront admission now includes the get_min_kudos floor - expired prompt/interrogation cleanups release their admission holds; snapshot/reconcile/repair (kudos_reconciliation, tools/kudos_ledger_admin.py) makes recovery compensating postings rather than history edits - migration sql_statements/5.1.0.txt; heartbeat exposes kudos_ledger health and reports DEGRADED past 30s pending age; strict mypy allowlist extended to the accounting modules - accounting semantics are observable: structured logs for mode transitions, floor adjustments, reconciliation, and trust promotion; horde.kudos.* counters for fold volume, catch-up saturation, admission rejections, and idempotent replays; a horde.kudos.applier.tick span wraps each applier tick |
||
|
tazlin
|
6c12cf8743 |
fix: worker speed zero-average fallback
- Align speed materialization with runtime behavior by treating an average performance of 0 like no samples and falling back to the worker baseline, preventing downstream division-by-zero on speed reads. - Add an index on worker_performances.worker_id in the model and migration path to avoid repeated full-table scans during backfill and ongoing performance updates. - Refactor 5.0.4 SQL rollout guidance, and add temporary insert-trigger scaffolding for mixed 5.0.3/5.0.4 fleets with a follow-up cleanup script. - Add unit tests covering zero-average fallback and recovery once nonzero samples arrive. |
||
|
tazlin
|
96bf72c2a3 |
perf: materialize worker speed into a maintained column
- Replace the `Worker.speed` hybrid property, whose SQL form ran a correlated avg(worker_performances) subquery per candidate worker per pop (the database's single largest cumulative-time query), with a plain NOT-NULL `speed` column maintained at the one write site (`record_performance`) and seeded on construction. - Preserve read semantics exactly: a worker with no samples reports the per-type baseline `1 * thing_divisors[wtype]` via COALESCE(avg, baseline), reproducing the old CASE expression, so fresh image workers still clear the `>= 500000` pop threshold while fresh text and interrogation workers still fall below theirs. Every `worker.speed >= N` filter site and Python reader becomes a plain column access with no call-site change. - Seed the column in `__init__` (the single construction chokepoint) rather than only in `create()`, so every path that persists a worker satisfies the NOT-NULL column, closing the gap for callers that build and commit a worker directly. - Add sql_statements/5.0.4.txt for existing databases: ALTER TABLE, a type-aware backfill from worker_performances, NOT NULL, and the ix_workers_speed index; bump the version to 5.0.4. - Add behavioural unit tests covering the running average, the prune interaction, baseline behaviour for both Python and pop-filter-shaped reads, and the team/status readers. |
||
|
tazlin
|
dde7d910d7 |
fix: allow outpainting a source_processing type
`outpainting` is allowed as a `source_processing` value, but prior to this change setting this value will cause a 500 stemming from a `sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(10)` error. This change allows the intended number of characters, plus an extra one for posterity. |
||
|
tazlin
|
868a94f010 |
feat: initial telemetry & logfire instrumentation
Introduce application telemetry using logfire/OTel and add spans/metrics across the codebase. - New horde/telemetry.py: configures logfire, optional Pyroscope profiling, helper to capture W3C traceparent, and defines OTel metrics (generate/pop/submit durations, pop query duration, candidates evaluated, pop skipped, submit kudos, IP check duration). - Wire telemetry into server startup (init_telemetry(HORDE)) and add logfire to requirements. - Instrument API flows and core operations with logfire spans and metric recording: generate, job_pop, job_submit, waiting prompt lifecycle (init/activate/start_generation), processing_generation webhook, worker contribution recording, image upload/download and prompt upload (r2), detection checks, and countermeasures IP checks. - Propagate traceparent on WaitingPrompt and include it in pop payloads when present. - Record metrics and attach span attributes (e.g. candidates evaluated, procgen counts, wp IDs) to enable trace→metric linking. - Add lightweight refactors/private wrappers (e.g. _post_inner, _ensure_source_image_uploaded, _upload_image/_download_image, _set_generation_inner) to keep logic unchanged while adding spans. - Database tweaks: instrument get_sorted_wp_filtered_to_worker and get_wp_queue_stats, measure pop query duration, and refactor workers_exist for correctness and fewer queries. refactor: explict logfire calls; addtl. spans/metrics; performance fixes telemetry: defer instrument binding until post-configure telemetry: use OTel meter proxy so custom metrics export chore: include pyroscope by default lint: bump ruff, apply auto-formatting+linting refactor: app init, background threads & testing - Move Flask app creation and side-effectful initialization out of package import into create_app()/get_app(), making the horde package safe to import. - Add explicit init functions for DB (init_db), background threads (start_background_threads), countermeasures, limiter, and Redis connection; replace direct HORDE references and HORDE.app_context() with get_app().app_context() throughout. - Introduce a cached_passkeys accessor (get_cached_passkeys/set_cached_passkeys) and update request_utils to use it. - Harden code for testing and optional deps (e.g. patreon/stripe), fix several boolean/raw-value checks and logging formatting, and adjust argparser to avoid SystemExit during import. - Update GitHub Actions to add a separate unit test job and run integration tests where appropriate, and add new/renamed tests and SQL statement file. |
||
|
tazlin
|
9cbd787edc | chore: add missing license file to migration script | ||
|
tazlin
|
24d41778d0 |
fix: known image description, stats columns type/length
- Change compiled image stats columns minute_pixels, hour_pixels, and day_pixels from Integer to BigInteger. - Increase KnownImageModel.description varchar length from 512 to 1024 to allow longer model descriptions. - Add SQL migration file (sql_statements/4.50.0.txt) with corresponding ALTER TABLE statements and a license file for the migration. |