- Document the promotion review queues in docs/reference/moderation_operations.md: the automatic promotion criteria, the two forms of them in User.promotion_status and User.promotion_queue_criteria, and the ix_users_evaluating_kudos_rank range scan each queue reads.
- Document the paused-worker list, its filters and sort, and the worker suspicion history: what report_suspicion records, the name cut to MAX_WORKER_NAME_CHARACTERS, the optional window bounded by MAX_WORKER_SUSPICION_RETENTION_DAYS, its retention step, and its disclosure on the privacy page.
- Describe both surfaces in the reference index.
- The per-subject text cap is removed: once text leaves the row a capped row costs the same, and past the threshold every rejection was both capped and timed out, so the cap did the timeout's work twice
- The counter counts model rejections only, so the threshold means what it says; filter rejections keep their own first-rejection timeout
- Evidence text_state none now means only that no stage existed at capture or retention removed it
- The notice reports the timeout alone; the privacy document states the timeout for repeated NSFW-model and flagged-account rejections
- The retention tick runs every minute and repeats bounded passes up to RETENTION_MAX_CATCHUP_CYCLES while a step fills its batch, so a backlog of millions of worker report rows clears in hours and a fresh row meets its window within minutes of becoming due; each pass keeps its short transactions
- A tick that uses every cycle with a full final batch counts a saturation and logs it; rows per step and cycles per tick are instruments
- The no-store warning in the text steps repeats at most once an hour under the faster cadence
- Reference doc and ADRs state the per-minute cadence
- Capture keeps the row as an outbox (text_state pending, text_sha256, text_chars); the quorum applier moves text to evidence/{id}.json every 5 s under an advisory lock and sets stored in one conditional update, deleting objects of rows retention cleared meanwhile and keeping objects another run recorded
- Retention deletes a batch's objects in one bulk call inside the row lock and flags only successes; a store outage holds back only rows with a stored object
- The listing returns text_state, a 10-minute presigned text_url for stored events, and the digest and length; stages stay inline while pending
- The evidence store client needs its own endpoint, bucket and credentials and is off without all four; legacy prompt and image buckets are untouched
- Rejections are counted per IP subject per hour: past HORDE_MODERATION_SUBJECT_TEXT_CAP_PER_HOUR events keep digest and length only; past HORDE_MODEL_REJECTION_TIMEOUT_THRESHOLD model rejections time out the address, every one in raid mode; moderators are exempt; one moderator notice per subject, action and hour carrying the latest event id and review link; countermeasures degrade to none on Redis errors
- Capture, backlog, rejection-rate, distinct-subject and observation-timestamp instruments for flood detection
- ADR 0016, reference doc with the edge rule text, privacy document gated on the configured store, env template
- Add docs/reference/moderation_operations.md: the recorded events and prompt stages, the IP subject and pseudonym, the moderator evidence API and its limits, the retention variables, steps and their effect on every reader, the account wipe, and the sharp edges operators need.
- Record the tiered retention design under an anonymizing ceiling in ADR 0015, with the pseudonym and worker report records, and add it to the decision index.
- Link the evidence stages from the prompt provenance reference and add the page to the reference index, the topics and the navigation.
The request endpoints added for the submitting key returned WaitingPrompt.prompt,
which is the worker input after the replacement filter has rewritten it. A
client could diff its own submission against the response and learn exactly
what the filter replaces, which cannot be allowed. The original submission
is now stored on its own column and returned to workers, but not to users
querying their own request details.
- Add WaitingPrompt.submitted_prompt, the parsed submission before
styles or moderation. It is nullable, deferred, and raiseload, so ordinary
queries, joins and relationship loads never fetch it, and an accidental lazy
access raises instead of silently pulling the text into worker payloads,
webhooks or share metadata
- Capture it in GenerateTemplate._post_inner before the subclass validate()
applies styles, and pass it into every waiting prompt constructor. The
constructor commits, so setting it afterwards would leave a committed row
without provenance
- Read only through read_privileged_submitted_prompt(purpose=...), a scalar
select that leaves the attribute unloaded on the instance. SubmittedPromptPurpose
is a required keyword identifying the reading consumer, so every caller that
handles the text is findable by grep
- get_submitted_request returns the original prompt. A row without provenance
omits the prompt entirely. The effective prompt is never a fallback because
that would reopen the leak for rows the filter touched
- Both request endpoints send Cache-Control: private, no-store
- ALTER waiting_prompts in sql_statements/5.1.12.txt: nullable, no default, so
the add is metadata-only, under a 5s lock_timeout that fails instead of
queueing an ACCESS EXCLUSIVE lock behind a long transaction. Existing rows
stay unknown; provenance is never backfilled from prompt
- Document the contract in docs/reference/prompt_provenance.md, and cover
deferral, the privileged read, the original surviving a filter rewrite, the
omission on legacy rows and absence from every existing export in tests, with
a migration test that the script is additive and repeatable
- The quorum publisher job now polls every 60 s and publishes only when the served snapshot is degraded or older than FLEET_PUBLISH_INTERVAL_SECONDS (publish_fleet_snapshot_if_due), so a newly elected owner or a fleet on a degraded document refreshes within a minute instead of after a full hourly interval
- refresh_text_reference drops individual entries without a numeric parameter count with a warning instead of rejecting the whole payload; only an unusable or empty payload retains the previous text reference
- Document both behaviors in docs/reference/model_reference.md
- Add horde/model_reference_snapshot.py: one complete, hash-verified ImageReferenceSnapshot per publication, replaced atomically under a fenced lease so a slow publisher cannot overwrite a newer revision, with sequence-guarded fan-out to every REDIS_SERVERS master
- ModelReference no longer fetches per process: the quorum owner publishes hourly via publish_model_reference_snapshot, every process polls Redis every 5 s, and cold start contends for the bootstrap lease instead of fetching independently
- Publication refuses an empty reference, an empty catalog, or HMR's GitHub fallback so the last-known-good document is never replaced by a degraded read; a fleet with no snapshot at all may bootstrap one degraded document from fallback sources after the strict window, logged as errors and flagged in the document
- Unknown top-level snapshot fields are ignored with a warning so older consumers keep loading during a rolling deploy
- Report snapshot age and degraded state as gauges on every poll and warn once the served snapshot is three hours old
- Require PrefetchStrategy.NONE on the HMR manager; reject a pre-existing manager built with another strategy
- Validate the text reference before replacing it and expose loader state through immutable per-type state objects
- store_known_image_models guards on a non-empty reference now that reference is never None
- Document the fleet distribution in docs/reference/model_reference.md and add the bootstrap and lease knobs to .env_template
- Refresh and stage the served baseline catalog before loading models, then expose both through one copy-on-write snapshot.
- Keep the previous complete snapshot serving when either fetch or validation fails, rather than leaking a new model against stale baseline policy.
- Add loader coverage for coherent publication and failed-refresh rollback; document the snapshot contract.
- Fixes baseline policy to treat uncatalogued baselines conservatively: plain generation is still allowed, but baseline-dependent workflows (such as QR code/remix and other gated features) are rejected until explicit catalog/bridge support exists.
- Renames the default capability constant to `UNCATALOGUED_CAPABILITIES`, updates parity and policy tests to reflect the new behavior.
- Adds Anima baseline coverage, added by bumping `horde_model_reference` to 8.5.0, which also supports the new "baseline" record semantics on models.aihorde.net.
- model_reference.py loads image_generation through horde_model_reference's ModelReferenceManager (REPLICA, GitHub fallback) with the PRIMARY's pending queue overlaid via PendingModelProvider, so beta models are known models; HORDE_BETA_MODEL_CATEGORIES and HORDE_BETA_MODELS_API_KEY control the overlay, the raw GitHub URL overrides and the expired zmodel branch gate are gone, and stable_diffusion_names is every record rather than a hard-coded baseline allowlist.
- Baselines are the horde_model_reference enum spelling end to end; get_model_baseline returns a record's baseline verbatim (unknown strings pass through) with the customizer name-suffix fallbacks, get_all_model_baselines shares them.
- baseline_policy.py: baseline_violation composes the request rules from two authorities, the served baseline record (weights and mechanisms that exist: remix, transparent, qr_code, unavailable control types, flow matching) and bridge_supports (which bridge releases render hires_fix, control_type, flow_shift, transparent); a feature no known worker kind renders is refused with the existing return codes. policy() reads kudos, batching, TTL and resolution floor from the record with par defaults, replacing the duplicated kudos ladders, BASELINE_BATCHING_MULTIPLIERS, FLOW_SHIFT_BASELINES and the per-baseline branches in validation.py, stable.py, worker.py, waiting_prompt.py and processing_generation.py.
- ImageAsyncGenerate.validate validates against the effective model list, which a style may have substituted; get_highest_model_batching_multiplier resolves baselines instead of looking model names up in a baseline-keyed dict; the dry-run quote applies the same qr_code and hires factors the charge does; resolution floors cover every large architecture.
- The KnownImageModel mirror takes ImageGenerationModelRecord objects; the manager's baseline store is refreshed on the hourly reference refresh.
- Tests seed the reference with real records (model_reference_seed.py); docs for the loader, the policy table and adding a baseline.
- replace histogram-sampled queue, lag, and reservation state with synchronous gauges so telemetry reflects the quorum applier's current state
- split pending counts and oldest-row age between currency and statistic queues, expose recent quarantine age, batch and catch-up configuration, and cycles used per tick
- refresh the quorum observation timestamp and lag after every committed catch-up cycle so long drains remain visibly healthy and former-quorum gauges can be excluded
- replace the old unapplied-stat index concurrently with a drainable predicate that keeps retained quarantine evidence out of the hot queue scan
- test migration idempotency from the previous production index state and verify telemetry remains fresh through multi-cycle catch-up
- record the deterministic quarantine and worker-deletion failure policy in ADR 14 and link its amendments from the existing projector and foreign-key decisions
- quarantine deterministic malformed stat events with bounded reason codes instead of allowing one poison event to roll back the global projector; quarantine every claimed row from the same business event to prevent partially materialized statistics
- add the v5.1.9 additive quarantine schema and expose retained quarantine evidence, age, reason-specific counters, and post-commit fold telemetry for alerting and operational diagnosis
- serialize worker deletion with claimed stat events and worker projection locks, preserving immutable audit history and surviving team attribution while skipping worker_stats materialization after a worker disappears
- validate projection dimensions and reject non-finite amounts at both kudos emitters before invalid data can enter the accounting queues
- keep success metrics behind the projection commit so rolled-back batches cannot appear as productive throughput
- document that quarantined_rows is cumulative retained evidence rather than unresolved incident state, and use newly quarantined events for paging
- cover poison-event isolation, migration idempotency, failed-transaction telemetry, missing targets, and the exact claimed-event versus worker-delete PostgreSQL interleaving
Each requested LoRA now adds ~107 seconds (400 MB at 30 Mbps) to the job lease after the 150-second floor. This covers cache-miss download time, which does not scale with resolution or sampler work. The extra-slow worker multiplier is applied after the LoRA addition. Docs and tests updated accordingly.
- Document three accepted architecture decisions covering distinct sampler work units, versioned sampler contract publication, and image job TTL as a conservative prefetch lease.
- Adds links to the new ADRs from the decisions index and sampler/scheduler reference so the public docs point to the governing design decisions.
Adds a new how-to document covering integration of modern image controls (sampler discovery, schedules, solver controls, expanded ControlNet types, and annotation) into an existing frontend. Also updates the samplers/schedulers reference with formatting fixes and registers the new doc in the nav and topic index.
- Clarify the worker-assignment TTL as a lease rather than an inference estimate, and document how sampler work, minimums, slow-model baselines, ControlNet, and extra-slow workers affect it.
- Update image job TTL calculation to round up to whole seconds and apply slow-model multipliers only to the assigned model, and tighten API text.
- Integration, unit, and stress coverage for the returned `ttl` value.
- Replaces evaluation-per-step accounting with SDK sampler work primitives across request usage, TTL, upfront gating, and `limit_max_steps` matching.
- Adds a new sampler work policy module, publishes execution contracts/work profiles in sampler constraints, and threads worker-advertised `sampler_execution_contract_version` through check-in/pop with fail-closed handling for adaptive ceilings.
- Adds a migration for persisting worker execution contract version, updates API schema/docs messaging.
- Expands integration/unit coverage for adaptive work estimation, downgrade behavior, and forecasting consistency.
- Bumps `horde_sdk` to 0.29
- explains why deterministic samplers agree once converged, which is the thing that makes side-by-side comparisons at 25 steps look uninformative: they integrate the same ODE, so agreeing is correct and only stochastic samplers keep producing different images
- separates per-step cost from fixed per-render overhead by fitting time against a 4/6/8/12/16/24/60 step ladder, because comparing wall time at one step count dilutes a higher-order sampler's real cost with the ~1.2s fixed part
- records the measured ordering: first-order solvers 1.0-1.4x, second-order 2.1-2.7x, heunpp2 3.2x, matching evaluations per step
- notes that SECOND_ORDER_SAMPLERS is a boolean where the cost is a ratio: it fits k_heun and k_dpm_2, under-states heunpp2 and k_dpmpp_sde, and prices uni_pc and dpmpp_2m_sde as if they were free
- tabulates schedule behaviour at 4 steps, where the differences are large: simple and sgm_uniform cleanest, karras artefacted despite its low-step reputation, linear_quadratic unusable
- records the known-failing combinations and where local measurement contradicts the published compatibility matrix
- adds the `generation` topic to the controlled vocabulary and the nav entry the docs CI requires
docs(reference): describe solver controls and constraints
- expands the sampler catalog with solver families, evaluation costs, accepted controls and client-facing presentation tiers
- distinguishes hard backend constraints from quality recommendations so clients know which combinations to reject and which merely deserve guidance
- documents the constraints endpoint, structured validation failures and the difference between trajectory steps and billable model evaluations
- records measured SD 1.5 and SDXL cost ratios with their provenance because fixed render overhead makes single-step-count comparisons misleading
- The inline trust-promotion path credited released escrow through modify_kudos(amount, "accumulated"), updating the user's per-action statistic. The ledger-mode drain path (_drain_trusted_escrow) only emitted the currency pair, leaving user_stats understating the balance's provenance.
- This change emits a matching EVALUATION_PROMOTION statistic event during the drain so both paths are consistent.
- Adds a unit test verifying the stat event is created with the correct fields and reflected in UserStats.
- Updates docs and reference tables accordingly.
- The /status/heartbeat endpoint backs per-node load balancer health checks. Surfacing kudos applier lag (shared DB state) caused fleet-wide health check failures simultaneously. This removes kudos_ledger from the response and moves health sampling to before the fold loop in apply_kudos_ledger so metrics reflect live inflow rather than a drained queue.
- Adds unit tests verifying the endpoint is node-local only.
- Reworked the service docs landing pages to emphasize the documentation classes and topic-based navigation, and added generated overview indexes for Explanation, Reference, and How-to sections.
- Added front matter and generated topic links to kudos docs so they can be indexed consistently, updated the decisions intro text, and documented index regeneration requirements in CONTRIBUTING.
- Updated zensical navigation to include Topics and section Overview pages, plus stub-status tooltip support for planned documents.
- Replace mode pinning/transition coordination from control-row key-share/update locks to a shared/exclusive advisory mode-gate lock.
- This preserves transaction pinning semantics while preventing transition starvation under continuous overlapping writer pins through fair lock queueing.
- Add dedicated mode-gate lock helpers, update mode read/transition code and related comments.
- Add a deterministic concurrency test that reproduces gapless writer pins to verify transitions still complete.
- Document the decision in ADR 10 and update accounting/ADR references
- Harden writer-thread cleanup in transition stress tests.
- add docs/explanation/kudos_accounting.md (design rationale and guarantees), docs/reference/kudos_accounting.md (maintainer contract: schemas, producer/consumer inventories, lock order, review checklist), and docs/how-to/kudos_ledger_operations.md (cutover, rollback, recovery procedure)
- add docs/README.md index and link it from the repository README
- add the haidra-assets submodule for shared user-facing kudos documentation
tazlin