dilim/AI-Horde
1
0 Fork 0
mirror of https://github.com/Haidra-Org/AI-Horde.git synced 2026-11-06 05:17:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: kudos accounting Diataxis set and haidra-assets submodule

Browse Source 
- add docs/explanation/kudos_accounting.md (design rationale and guarantees), docs/reference/kudos_accounting.md (maintainer contract: schemas, producer/consumer inventories, lock order, review checklist), and docs/how-to/kudos_ledger_operations.md (cutover, rollback, recovery procedure)
- add docs/README.md index and link it from the repository README
- add the haidra-assets submodule for shared user-facing kudos documentation
This commit is contained in:
tazlin committed 2026-07-28 13:23:45 -04:00
1 parent 7ed5b46f69
commit 01e769de0f
8 files changed
+848
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitmodules
+7
View File
@@ -0,0 +1,7 @@
# SPDX-FileCopyrightText: 2026 Tazlin
#
# SPDX-License-Identifier: AGPL-3.0-or-later
[submodule "docs/haidra-assets"]
path = docs/haidra-assets
url = https://github.com/Haidra-Org/haidra-assets.git
.prettierrc
+19
View File
@@ -0,0 +1,19 @@
{
"tabWidth": 2,
"printWidth": 119,
"trailingComma": "all",
"bracketSpacing": true,
"semi": true,
"singleQuote": true,
"endOfLine": "lf",
"htmlWhitespaceSensitivity": "css",
"overrides": [
{
"files": "*.md",
"options": {
"tabWidth": 4,
"proseWrap": "always"
}
}
]
}
README.md
+3
View File
@@ -72,6 +72,9 @@ The AI Horde middleware can also be deployed privately within any enterprise env
For a high-level overview of how the AI Horde operates, including diagrams of the request/job lifecycle, see the [request/job lifecycle explanation](https://github.com/Haidra-Org/haidra-assets/blob/main/docs/workers.md).
Maintainers and contributors can find repository-specific architecture documentation in [the local documentation
index](docs/README.md), including the kudos accounting explanation, reference, and operational procedure.
There are also individual readmes for each specific mode supported by the AI Horde.
- [Image generation Readme](README_StableHorde.md)
docs/README.md
+23
View File
@@ -0,0 +1,23 @@
<!--
SPDX-FileCopyrightText: 2026 Tazlin
SPDX-License-Identifier: AGPL-3.0-or-later
-->
# AI Horde service documentation
These pages describe the implementation and operation of this repository. General, user-facing AI Horde
documentation remains in [`haidra-assets/docs`](haidra-assets/docs/README.md).
The local documentation follows [Diataxis](https://diataxis.fr/):
- **Explanation** builds a mental model and records why the system has its present shape.
- **Reference** states the exact contracts, schemas, producers, consumers, and invariants.
- **How-to** material gives operators a sequence to follow for a concrete task.
## Kudos accounting
- [Explanation: kudos accounting, projection, and concurrency](explanation/kudos_accounting.md)
- [Reference: kudos mutation and consumption](reference/kudos_accounting.md)
- [How-to: kudos ledger operations](how-to/kudos_ledger_operations.md)
- [General concept: what kudos are](haidra-assets/docs/kudos.md)
docs/explanation/kudos_accounting.md
+336
View File
@@ -0,0 +1,336 @@
<!--
SPDX-FileCopyrightText: 2026 Tazlin
SPDX-License-Identifier: AGPL-3.0-or-later
-->
# Kudos accounting, projection, and concurrency
Today, a kudos-changing business transaction records signed, typed events instead of treating the current balance
columns as the complete history of what happened. Currency movements go to the append-only `kudos_ledger`;
display totals and non-currency statistics go to `kudos_stat_events`. In `ledger` mode, a database-serialized
projector folds those events into the familiar user, worker, team, and statistics columns. Spend admission uses
single-payer reservations so it remains safe while the visible balance is catching up.
In brief:
- The balance and aggregate columns (`users.kudos`, `workers.kudos`, team totals, stats) still exist and are still
what ordinary read sites consume. They are now projections of the event history, and in `ledger` mode they can lag
by a projector interval.
- Currency movements are permanent rows in `kudos_ledger`; display totals and counters are permanent rows in
`kudos_stat_events`. Rows from one business event share one `event_id`.
- Spend admission never trusts the raw balance column. It takes a payer lock and checks `available_kudos`,
which subtracts active reservations and queued debits and deliberately ignores queued credits.
- One database-serialized projector folds unapplied events into the materialized columns. `applied = false` is
the work queue; there is no watermark that can skip a row.
- `shadow` mode is a temporary mode that changes no balance the old code would not have changed. `ledger` mode is an
explicit, online-reversible operator cutover.
To dive into code, start with `horde/classes/base/kudos.py` (the models and the emission primitives every
producer uses) and `horde/database/kudos_ledger.py` (the projector). The
[code map](../reference/kudos_accounting.md#code-map) covers the rest.
Previously, request activation, settlement, transfers, uptime rewards, trust promotion, awards, and administrative
adjustments changed the materialized rows directly inside their request transactions. The current columns were both
the mutation mechanism and the only practical accounting record. That made frequently used user rows part of many
otherwise unrelated transaction lock graphs, made multi-account operations sensitive to lock order, and left no
detailed audit trail. The new mechanism separates the durable fact that a movement occurred from the eventually
updated read models. A deliberately temporary `shadow` mode records the same events while retaining the old inline
projection, permitting comparison and an online, reversible cutover.
This page explains why that shape exists and the limits of the guarantees. The exact mutation rules and the
consumer inventory are in the [kudos accounting reference](../reference/kudos_accounting.md). The operator sequence
for cutover, rollback, and repair is in [kudos ledger operations](../how-to/kudos_ledger_operations.md). For the
user-facing purpose of kudos, see [What are kudos?](../haidra-assets/docs/kudos.md).
## The problem: shared mutable rows
A kudos settlement sounds like simple addition and subtraction, but one completed job updates a connected set of state:
- the requester's spendable balance, usage totals, records, and last-active time;
- the worker owner's spendable or evaluation balance;
- the worker's display kudos, contribution total, fulfilment count, and action buckets;
- the worker's team aggregates, using the team membership at settlement time; and
- the waiting prompt, processing generation, reservation, and transfer metadata that make the business event valid.
The old implementation performed much of that work inline. Concurrent activation and settlement transactions could
reach the same user and job rows through different paths. A transfer necessarily involved two users. Even when every
individual update was correct, inconsistent acquisition order could form a cycle: transaction A held a prompt or
source user while waiting for another user, while transaction B held that user and waited for A's row. Retrying a
deadlock victim prevents some user-visible failures, but it does not remove the contention or provide an audit trail.
The same design also coupled correctness to timing. Read-then-insert statistics could race on the first dimension
row; a crash between related updates was difficult to explain after the fact; a retry could repeat money without a
stable business key; and correcting a balance meant overwriting the only visible value rather than adding an
auditable compensating fact.
### Pre-ledger lock contention
Lock-chain and `pg_stat_statements` captures taken on the production primary on 2026-07-20 measured the cost. The
anonymous requester row (`users.id = 0`) carried `Lock:tuple` queues 15 waiters deep even during quiet periods, with a
mean lock acquisition around one second, maxima in minutes, and single-row balance updates observed blocked for hours
during incidents; the `FOR NO KEY UPDATE` acquisition sat among the top `pg_stat_statements` offenders. Removing the
users-row serialization moved the queue one level down rather than dissolving it: with balances appended instead of
updated inline, load testing found the anonymous user's `user_stats` row queued 10 to 12 deep behind multi-second
idle-in-transaction holders. Folding the statistics counters into the same event stream, so request transactions
append and write no shared aggregate row, is what removes the remaining queue.
The initiative therefore solves four related problems:
1. shrink and standardize the lock graph for kudos mutations;
2. make currency movement durable, typed, correlated, and replayable;
3. preserve spend safety while projection is asynchronous; and
4. make cutover, rollback, diagnosis, and repair explicit operations rather than improvised database edits.
## The mental model: facts, holds, and projections
There are three different kinds of state. Treating them as interchangeable is the easiest way to introduce an
accounting bug.
```mermaid
flowchart LR
B[Business event<br/>settlement, transfer, award] --> G[kudos_event<br/>one correlation id]
G --> L[(kudos_ledger<br/>currency facts)]
G --> S[(kudos_stat_events<br/>display and counter facts)]
A[Spend admission] --> R[(kudos_reservations<br/>temporary holds)]
R -. reservation id .-> L
L --> P[Serialized projector]
S --> P
P --> U[(users.kudos<br/>users.evaluating_kudos)]
P --> W[(worker and team<br/>display aggregates)]
P --> C[(stats and records)]
P --> X[consume or release holds]
U --> D[Display and ordinary reads]
U --> Q[Queue-priority snapshot]
U --> V[available/effective<br/>balance helpers]
R --> V
L --> V
```
**Facts** answer "what movement was accepted?" A `KudosLedger` row is one signed two-decimal currency delta against
one user's spendable or evaluation balance. A `KudosStatEvent` row is one non-currency or display delta. Rows from the
same business event share an `event_id`, but each row remains independently claimable and auditable.
**Holds** answer "what has already been promised?" A reservation belongs to exactly one payer and has a stable
`business_id`. It prevents two concurrent admissions from both relying on the same not-yet-projected balance. It is
not currency and never credits a recipient.
**Projections** answer "what should existing readers see cheaply?" `users.kudos`, `workers.kudos`, the team totals,
and the existing statistics tables remain denormalized read models. In ledger mode they can lag accepted events by a
projector interval. They are not discarded because queueing, API responses, leaderboards, and existing integrations
need inexpensive reads.
This separation also explains why worker "kudos" do not belong in the currency ledger. `workers.kudos` is a display
total attributed to a worker; the actual currency credit belongs to the worker owner in `users.kudos`. Likewise,
contributions and fulfilments are measured in things or counts. Mixing them into one table would make conservation,
reconciliation, and future reporting ambiguous.
## One business event, several postings
The `kudos_event` context assigns a shared event UUID and optional job metadata to related postings. For example, a
normal generation settlement can produce all of the following without directly editing the corresponding read-model
rows in ledger mode:
```mermaid
flowchart TD
E[Generation settlement<br />event]
E --> RD[Requester currency debit]
E --> OC[Owner currency credit]
E --> OE[Owner escrow credit<br/>while untrusted]
E --> WK[Worker display-kudos delta]
E --> UF[User usage and contribution records]
E --> WF[Worker contributions and fulfilment]
E --> TM[Team aggregates<br/>team id stamped now]
E --> LA[Last-active event]
```
The exact set varies with trust, cancellation, fake generations, shared keys, and job type. The important invariant is
that the business transaction commits its job state and emitted facts together. The event UUID supplies correlation;
it does not by itself make every producer retry idempotent. Transfers accept a caller idempotency key and validate a
replay. Other producers still depend on their existing job-state transaction to prevent duplicate settlement.
## Projection: durable work queue
In ledger mode, rows start with `applied = false`. The periodic projector:
1. tries to acquire a PostgreSQL transaction advisory lock, independent of the process-level quorum;
2. claims bounded, ID-ordered currency and statistics batches with `FOR UPDATE SKIP LOCKED`;
3. groups deltas by target and visits materialized targets in stable primary-key order;
4. updates balances, counters, records, worker totals, and team totals;
5. consumes or releases the matching reservations;
6. marks the exact claimed row IDs applied; and
7. commits all those changes in the same database transaction.
If the process dies before commit, neither the projection nor the applied flags commit. A later cycle sees the rows
again. If a lower numeric ID commits after a later row was already processed, it remains `applied = false` and is
claimed by a later cycle; there is no high-water mark that can skip it. "Exactly once" here means exactly one
committed fold of an accepted row, not exactly-once creation of arbitrary business events.
The applier heartbeat is deliberately not a correctness watermark. It only makes a stopped or delayed projector
observable. Applied currency and statistic events are retained permanently; the pruning hook is a compatibility
no-op.
## Deadlock mitigation
The ledger design is better at preventing the deadlock class that motivated its creation because ordinary producers
append new rows instead of acquiring write locks on hot user, worker, team, stats, and record rows. Multi-account
transfers serialize on the payer only and never lock a recipient for admission. Projection is moved to one
database-serialized writer, which accumulates a batch and applies each target class in a stable order.
```mermaid
flowchart LR
subgraph Before[Previous inline mutation]
A1[Activation] --> U1[(requester user row)]
S1[Settlement] --> U1
S1 --> U2[(worker-owner user row)]
T1[Transfer] --> U1
T1 --> U2
S1 --> J1[(job and aggregate rows)]
A1 --> J1
end
subgraph After[Ledger mode]
A2[Activation] --> N1[(new event rows)]
S2[Settlement] --> N2[(new event rows)]
T2[Transfer] --> PL[payer advisory lock]
PL --> N3[(new event rows)]
AP[one DB-serialized projector] --> U3[(ordered read-model updates)]
N1 --> AP
N2 --> AP
N3 --> AP
end
```
The claim is intentionally narrower than "deadlocks are impossible." Activation and settlement still change job
state, relationships, shared-key budgets, and other non-ledger rows. Database maintenance or unrelated code can also
lock a projection target. The bounded PostgreSQL deadlock retry on waiting-prompt activation remains for those cases.
The projector itself can be blocked, and its single-writer design trades write parallelism for a much simpler lock
graph. Queue age, heartbeat age, and database deadlock metrics must therefore be monitored after cutover.
The accounting-specific lock order is:
```text
mode transition: applier advisory lock -> control-row exclusive lock -> final fold
projector: applier advisory lock -> event rows -> ordered projection targets -> reservations
producer: control-row key-share lock -> optional payer advisory lock -> appended rows
reconciliation: repeatable-read snapshot; repair emission also takes the reconciliation advisory lock
```
Every mutation transaction pins the observed mode by holding a key-share lock on the control row until commit. An
exclusive mode change therefore waits for old-mode writers to finish before the new ownership rule becomes visible.
The transition back to shadow takes the applier lock first, waits for those writers, and folds the final ledger tail in
the same transaction. This prevents an old ledger-mode posting from appearing after inline projection resumes.
## Reservations make eventual balances spend-safe
Asynchronous projection means `users.kudos` alone cannot answer whether another debit may be accepted. A user could
have 100 visible kudos, submit two simultaneous 80-kudos transfers, and appear solvent to both transactions if their
debits only exist as pending ledger rows.
Admission closes that window with a transaction-scoped advisory lock derived from the payer's ID. Under that lock,
`available_kudos` subtracts the account floor, active holds, and ordinary queued debits while deliberately ignoring
queued credits. The accepted operation creates or reactivates a uniquely named reservation in the same transaction
as its ledger posting. Projection consumes request holds as their debits fold and releases transfer holds only when
the complete event has folded, including a transfer split across batches.
This is conservative by design: a queued credit cannot fund a new spend until it is projected. Conservatism can delay
work during projector lag, but it cannot authorize an overspend on the strength of unmaterialized income.
`effective_kudos` serves a different purpose. It adds all committed, unprojected currency deltas to the materialized
balance and clamps the result to the account floor. It is appropriate for a "new balance" response or diagnosis, not
for spend authorization, because it includes queued credits and does not represent holds.
## Compatibility rules
The ledger changes mutation mechanics and leaves the established economic policy intact:
- A debit cannot take an account below `get_min_kudos()`. If projection forgives part of a debit, it emits an already
applied `FLOOR_ADJUSTMENT` for the created amount so replay remains linear.
- Untrusted worker-owner rewards go wholly or partly to `evaluating_kudos`, depending on reward type. The projector
detects the final threshold-crossing contribution, grants trust, and emits an escrow-debit/spendable-credit pair.
This avoids requiring a later request to trigger promotion.
- Worker and team aggregates retain historical attribution. The producer stamps the worker's current `team_id` on
the event, so moving the worker before projection cannot move old credit to a new team.
- Shared-key kudos remain an inline per-key quota. They are not user currency and are not projected by this ledger.
- Job cost fields such as `waiting_prompts.kudos` and `consumed_kudos` are prices and job-local accounting, not account
balances.
Some ordinary reads intentionally remain eventually consistent. Queue priority is copied from the materialized user
balance when a request or interrogation is activated; API and login display paths generally expose materialized
balances; and the image-worker upfront eligibility recheck still reads the materialized balance even though initial
admission is protected by a reservation. Those paths can be stale by the projector lag. They do not authorize a
second spend, but they can temporarily show an old value or make a conservative/optimistic scheduling decision. The
[consumer inventory](../reference/kudos_accounting.md#consumer-and-read-model-inventory) records these sites so a
future change can deliberately choose `available_kudos`, `effective_kudos`, or the projection rather than blindly
replacing every `.kudos` read.
## Shadow mode
Shadow mode is a migration mechanism, not a second architecture. Business methods always emit the new typed events.
A small compatibility projector applies the historical inline mutations and marks the emitted rows already applied.
This provides permanent audit evidence without replaying a movement that already changed its target. The asynchronous
projector's trust-promotion and escrow-drain duties run only in ledger mode; in shadow mode the compatibility projector
owns them, so shadow observation changes no balance the previous code would not have changed.
The mode branch is confined to `horde/database/kudos_legacy_projection.py`, the control helpers, and the
applied-flag choice inside the two emission primitives. Removing the cutover period is therefore a small deletion:
remove the compatibility calls/module and the shadow mode transition, while leaving producers, event schemas,
reservations, and the ledger projector intact.
Shadow history is a forward audit beginning at deployment; it is not a reconstruction of all kudos ever created.
The existing materialized balances form the opening position. A transaction-consistent snapshot records that opening
position together with the applied-ledger totals visible at the same time.
## Recovery
The permanent event archive and balance snapshots make recovery demonstrable:
```mermaid
stateDiagram-v2
[*] --> Shadow: deploy schema and writers
Shadow --> Prove: observe, snapshot, reconcile
Prove --> Ledger: explicit mode change
Ledger --> Ledger: projector restart and drain
Ledger --> Repair: snapshot shows drift
Repair --> Ledger: compensating postings, drain, reconcile
Ledger --> Shadow: lock writers and drain final tail atomically
Shadow --> Ledger: prove again before recutover
```
A snapshot uses repeatable-read isolation and records each user's materialized spendable/escrow values plus the
applied ledger totals at that point. Reconciliation computes the expected current values from that baseline and later
applied events. Read-only reconciliation reports drift. Repair mode emits deterministic `RECONCILIATION` postings;
it never rewrites a balance, deletes a posting, or changes an old `applied` flag. Re-running the same repair cannot
duplicate it.
If the projector stops, accepted rows remain durable and unapplied: restore the projector and drain. If projection is
wrong, preserve evidence, reconcile from a known snapshot, review all drift, emit compensation, drain, and reconcile
again. If ledger ownership itself must be rolled back, switch to shadow through the control helper so the final tail
is folded atomically. Rolling directly back to code that does not understand reservations and shadow audit events is
unsafe.
This recovery model is why the archive is not pruned and why an operator must never "fix" an incident by toggling
`applied`, deleting events, or assigning balances directly. The operational commands and rehearsal checklist are in
[kudos ledger operations](../how-to/kudos_ledger_operations.md).
## Costs and boundaries
The architecture deliberately accepts several costs:
- Normal balance and aggregate reads are only eventually consistent in ledger mode.
- One projector limits write throughput; bounded batches, a small interval, and a bounded per-tick catch-up loop are
the scaling controls. Sharding it would require a new ordering and reservation design, not merely more threads.
- The permanent archives grow without automatic pruning and need capacity planning.
- PostgreSQL provides the production concurrency guarantees, and the unit suite runs against PostgreSQL. The SQLite
code branches serve the legacy `USE_SQLITE` runtime mode only; they short-circuit advisory locks and prove nothing
about concurrent-projector behavior.
- Projection exactly-once does not automatically make every producer idempotent. New externally retryable mutations
need a stable idempotency key and parameter-conflict behavior.
- Reconciliation covers user currency and evaluation escrow. Derived statistics are auditable and replayable from
`kudos_stat_events`, but the current snapshot/repair command does not reconcile every worker, team, or counter row.
These are preferable to an implicit, distributed lock graph, but they are still operational obligations. A safe
cutover requires shadow evidence, PostgreSQL concurrency tests, a current snapshot, a clean reconciliation, healthy
queue lag, and a rehearsed return to shadow before ledger mode becomes authoritative.
docs/haidra-assets
Submodule
+1
Submodule docs/haidra-assets added at 9408caf056.
docs/how-to/kudos_ledger_operations.md
+55
View File
@@ -0,0 +1,55 @@
<!--
SPDX-FileCopyrightText: 2026 Tazlin
SPDX-License-Identifier: AGPL-3.0-or-later
-->
# Kudos ledger operations
This is the operator procedure for the architecture described in [Kudos accounting, projection, and
concurrency](../explanation/kudos_accounting.md). Exact schemas, mutation rules, and health fields are in the
[kudos accounting reference](../reference/kudos_accounting.md).
The kudos ledger has two online modes:
- `shadow`: legacy inline balances and counters are authoritative; matching ledger rows are retained as already-applied audit history.
- `ledger`: request transactions append postings and the database-serialized projector materializes them asynchronously.
New installations and the migration SQL default to `shadow`; moving to `ledger` is always an explicit operator action. Both currency postings and non-currency statistic events are permanent archives. The projector is serialized by a PostgreSQL transaction advisory lock, independent of Redis quorum selection, and claims bounded batches with `FOR UPDATE SKIP LOCKED`.
## Code and schema boundaries
The new accounting models use SQLAlchemy 2.x typed mappings (`Mapped` and `mapped_column`) and database-portable `Uuid`, `Enum`, and JSON types. The SQLAlchemy mypy plugin checks the model attributes and constructors. Fixed units, projector record discriminators, aggregate names, and audit-detail keys are `StrEnum` values rather than string literals.
The currency ledger has one required `users.id` foreign key with `ON DELETE RESTRICT`; users are soft-deleted/wiped, and authoritative currency history must not become orphaned. `kudos_reservations.user_id` and `kudos_balance_snapshots.user_id` are ownership foreign keys with `ON DELETE CASCADE`: those operational rows only have meaning while the user exists. IDs in `kudos_stat_events` are intentionally immutable audit references rather than ownership foreign keys. Workers and teams can be hard-deleted, but counter history must survive; adding cascading foreign keys would destroy that history and restrictive foreign keys would break supported deletion. This exception is documented in the mapped model.
PostgreSQL advisory locks and repeatable-read setup are confined to `horde.database.kudos_db` and are expressed through SQLAlchemy functions/connection options, not textual execution. Counter upserts are confined to `horde.database.kudos_counters`. Shadow-mode inline projection is likewise confined to `horde.database.kudos_legacy_projection`; business methods always emit the new events. A final cutover removes that compatibility module and its direct calls without rewriting the accounting flow.
## Pre-cutover proof
1. Deploy the new code and schema to the full fleet in `shadow` mode. Do not mix it with code that does not write the audit rows.
2. Run through a representative peak-load window. Exercise transfers, upfront image/text/interrogation admission, cancellations, trust promotion, monthly awards, and admin adjustments.
3. Inspect `uv run python tools/kudos_ledger_admin.py status`. Investigate any non-zero old queue, applier heartbeat gap, or `oldest_pending_seconds` above 30 seconds.
4. Capture a transaction-consistent baseline with `uv run python tools/kudos_ledger_admin.py snapshot`.
5. Run `uv run python tools/kudos_ledger_admin.py reconcile <snapshot-id>`. It must report no unexplained drift.
6. Switch with `uv run python tools/kudos_ledger_admin.py mode ledger`. Control-row locks wait for every transaction that observed shadow mode before changing ownership; no service freeze is required.
Monitor pending row count, oldest pending age, heartbeat age, database deadlocks, reservation age/count, transfer rejection rate, and balance-floor adjustments throughout rollout. The `/api/v2/status/heartbeat` response exposes queue health and reports `DEGRADED` once the oldest pending event exceeds 30 seconds.
## Online rollback
Keep ledger mode active and pre-drain with `uv run python tools/kudos_ledger_admin.py drain` until `pending_rows` is near zero. Reconcile against the current baseline, then run `uv run python tools/kudos_ledger_admin.py mode shadow`.
The transition takes the database applier lock followed by an exclusive control-row lock, waits for every mutation that observed ledger mode to commit, and folds the final tail in the same transaction before changing ownership. This lock order is shared with the projector, avoiding an applier/control deadlock. Active upfront reservations can span the transition because shadow-mode debits consume the same holds inline. Never roll directly back to code that does not understand reservations and shadow audit rows.
## Recovery and repair
`reconcile <snapshot-id>` is read-only and compares the materialized balances with the snapshot plus all subsequently applied currency postings. Minimum-balance forgiveness is recorded as an explicit `FLOOR_ADJUSTMENT`, so replay remains exact across separate batches. `reconcile <snapshot-id> --apply` never overwrites a balance or old history: it serializes repair runs and emits one deterministic `RECONCILIATION` posting per affected user. Re-running it before or after projection cannot duplicate a repair.
If the projector stops, leave writers in ledger mode, restore the projector, and drain; unapplied rows are durable and the database advisory lock prevents two replicas from applying them. If projection is corrupt, take a fresh snapshot for evidence, reconcile against the last known-good baseline, review the complete drift list, apply compensating postings, drain, and reconcile again. Do not edit `applied`, delete postings, or directly overwrite balances.
For database disaster recovery, restore PostgreSQL to the selected PITR/WAL point, retain the permanent ledger/stat archives and balance snapshots, start in shadow mode, reconcile, then repeat the cutover proof. Ledger pruning is disabled.
## Automated drill coverage
`tests/unit/test_kudos_safety.py` demonstrates concurrent-projector exclusion, reservation overspend prevention, transfer idempotency, final-event trust promotion, an atomic ledger-to-shadow tail drain, snapshot drift detection, idempotent compensating repair, and replay across floor adjustments. `tests/unit/test_wp_activate_deadlock.py` preserves bounded PostgreSQL deadlock retry behavior. Run those tests against PostgreSQL before every cutover or recovery exercise.
docs/reference/kudos_accounting.md
+404
View File
@@ -0,0 +1,404 @@
<!--
SPDX-FileCopyrightText: 2026 Tazlin
SPDX-License-Identifier: AGPL-3.0-or-later
-->
# Kudos accounting reference
This page is the maintainer contract for kudos mutation and consumption. It inventories the authoritative events,
materialized targets, producers, consumers, locks, and tests. For design rationale, read [Kudos accounting,
projection, and concurrency](../explanation/kudos_accounting.md). For deployment and incident procedures, follow
[Kudos ledger operations](../how-to/kudos_ledger_operations.md).
## Source of truth by mode
| Concern | `shadow` mode | `ledger` mode |
| --- | --- | --- |
| Accepted currency movement | Append `kudos_ledger` row with `applied = true` | Append `kudos_ledger` row with `applied = false` |
| Accepted display/counter movement | Append `kudos_stat_events` row with `applied = true` | Append `kudos_stat_events` row with `applied = false` |
| Materialized mutation owner | `kudos_legacy_projection` applies the historical inline change | `apply_pending_kudos` folds events asynchronously |
| Spend authorization | Payer lock plus `available_kudos` and reservations | Payer lock plus `available_kudos` and reservations |
| Ordinary balance reads | Materialized column, current after the transaction | Materialized column, eventually consistent |
| Audit/replay role | Permanent forward audit; opening balance predates it | Authoritative post-cutover movement archive |
| Default | Yes, for new/migrated installations | No; operator must explicitly cut over |
The existing balance at first deployment is the opening position. The ledger contains deltas from deployment onward,
not a fabricated history of older movements.
## Vocabulary
| Term | Exact meaning |
| --- | --- |
| Currency | A delta to a user's spendable `users.kudos` or evaluation `users.evaluating_kudos` balance. |
| Display kudos | A worker/team attribution total. It describes earned work but is not spendable account currency. |
| Counter | A value denominated in kudos, things, or count and projected into stats, records, or aggregates. |
| Business event | One logical mutation such as a settlement or transfer, correlated by one `event_id`. |
| Posting | One signed row affecting exactly one target balance or statistic. Direction comes from the amount's sign. |
| Projection | A denormalized current value maintained from postings for inexpensive reads. |
| Hold/reservation | A temporary claim against one payer's available kudos. It is neither a debit nor a credit. |
| Applied | The posting was included in a committed projection transaction, or was projected inline in shadow mode. |
| Available balance | Conservative spend capacity after floor, active reservations, and unreserved queued debits. |
| Effective balance | Materialized balance plus all committed unapplied currency deltas, clamped to the account floor. |
## Data model
### Accounting tables
| Model/table | Purpose | Ownership and lifetime | Important constraints |
| --- | --- | --- | --- |
| `KudosLedger` / `kudos_ledger` | Permanent currency postings | Required `users.id` foreign key with `ON DELETE RESTRICT`; authoritative history must not be orphaned | One user target; spendable vs escrow selected by `escrow`; `NUMERIC(20,2)` amount; NaN rejected; partial unapplied index |
| `KudosStatEvent` / `kudos_stat_events` | Permanent display and counter postings | User, worker, and team IDs are immutable audit references, intentionally not ownership foreign keys because workers/teams may be hard-deleted | Exactly one of `user_id` and `worker_id`; typed unit; NaN rejected; partial unapplied index |
| `KudosReservation` / `kudos_reservations` | Payer holds for upfront work and transfers | User ownership foreign key with `ON DELETE CASCADE` | Unique `business_id`; positive original amount; non-negative remaining amount; active-user partial index |
| `KudosBalanceSnapshot` / `kudos_balance_snapshots` | User-currency reconciliation baseline | User ownership foreign key with `ON DELETE CASCADE` | One row per snapshot/user; records balance, escrow, and visible applied totals |
| `KudosLedgerControl` / `kudos_ledger_control` | Single-row mutation-mode control | Installation state | ID 1; `shadow` or `ledger`; non-null change time |
| `KudosLedgerApplierState` / `kudos_ledger_applier_state` | Projector heartbeat | Operational state only | ID 1; not a watermark and not used for exactly-once folding |
The mapped accounting models use SQLAlchemy 2 `Mapped`/`mapped_column` attributes. Runtime dialect operations are
confined to typed helpers: advisory locks and isolation in `horde/database/kudos_db.py`, counter upserts in
`horde/database/kudos_counters.py`, and legacy mode behavior in `horde/database/kudos_legacy_projection.py`. Raw DDL
belongs in the versioned migration file, not request code. The SQLAlchemy mypy plugin checks the accounting model
surface in CI.
### `KudosLedger` fields
| Field | Contract |
| --- | --- |
| `id` | Database identity used only for bounded claim order; it is not a transaction watermark. |
| `created` | Audit timestamp and queue-age source. |
| `event_id` | UUID shared by related postings. A deterministic UUID is produced when an idempotency key is supplied. |
| `entry_type` | Typed business classification from `KudosEntryType`; never encodes debit/credit direction. |
| `user_id` | Required currency owner. |
| `escrow` | `false` targets spendable balance; `true` targets evaluation escrow. |
| `amount` | Signed `NUMERIC(20,2)` delta. Positive credits and negative debits are separate postings. |
| `applied` | Per-row work-queue state changed in the same transaction as its projection. |
| `job_id`, `wp_type` | Optional correlation dimensions inherited from the active `kudos_event`. |
| `detail` | Typed-key audit metadata. The applier only interprets documented keys. |
### `KudosStatEvent` fields
`event_id`, `entry_type`, `job_id`, `wp_type`, `amount`, `detail`, and `applied` have the same correlation/audit
meaning as their currency counterparts. The remaining fields define a counter projection:
| Field | Contract |
| --- | --- |
| `user_id` | User counter target; mutually exclusive with `worker_id`. |
| `worker_id` | Worker counter target; mutually exclusive with `user_id`. |
| `worker_user_id` | Worker owner captured for audit; not itself a projection target. |
| `team_id` | Team attribution captured when the event is emitted. |
| `unit` | `kudos`, `things`, or `count`; consumers must not combine unlike units. |
| `stat_action` | Action/bucket dimension such as `generated`, `usage`, `contributions`, or `fulfilments`. |
| `record` | Typed projector discriminator or user-record dimension. |
### Stable enums and audit keys
| Enum | Values/role |
| --- | --- |
| `KudosLedgerMode` | `shadow`, `ledger` |
| `KudosUnit` | `kudos`, `things`, `count` |
| `KudosStatRecord` | `user_kudos`, `worker_kudos`, `last_active` |
| `KudosAggregate` | `contributions`, `fulfilments` |
| `KudosAuditDetail` | `reason`, `reservation_id`, `snapshot_id`, `touch_last_active` |
Do not add an untyped string discriminator when one of these axes describes it. Add or extend the enum and teach the
projector and tests about the new value together.
## Entry-type catalogue
An entry type answers “why did this posting exist?” The sign and target answer “what did it do?”
| `KudosEntryType` | Expected use |
| --- | --- |
| `GENERATION` | Image/text/interrogation settlement: requester debit, worker-owner spendable/escrow credit, and worker display event as applicable |
| `UPTIME_REWARD` | Periodic worker display credit and owner spendable/escrow credit |
| `EVALUATION_PROMOTION` | Paired escrow debit and spendable credit after trust promotion |
| `TRANSFER` | Paired source debit and destination credit for a user gift |
| `ADMIN_ADJUSTMENT` | Signed administrator adjustment |
| `AWARD` | Monthly, rating/aesthetic, and other application award credits |
| `STYLE_REWARD` | Style-owner currency reward; style object's own counters remain separate |
| `STAT_RECORD` | User record counts or thing totals |
| `STAT_CONTRIBUTION` | Worker/team contribution and fulfilment aggregates |
| `STAT_ACTIVITY` | Asynchronous user `last_active` touch |
| `FLOOR_ADJUSTMENT` | Explicit currency created when an account-floor rule forgives a debit |
| `RECONCILIATION` | Deterministic compensating currency posting emitted by an approved repair |
## Mutation contract
All new or modified kudos code must obey these rules.
1. **Use the public producer surface.** User currency enters through `User.modify_kudos` or
`User.modify_evaluating_kudos`; derived events enter through `emit_kudos_stat_event`. Do not assign a materialized
accounting column in a business path. Narrow exceptions are listed under [excluded kudos-like state](#excluded-kudos-like-state).
2. **Group a logical event.** Wrap a settlement, transfer, activation tax, or other multi-posting mutation in
`kudos_event`. Include `job_id` and `wp_type` where available.
3. **Make external retries explicit.** If an API mutation can be retried independently of locked business state,
accept a stable idempotency key, derive the event ID from it, and reject reuse with different parameters. A random
event UUID provides correlation only.
4. **Commit business state and postings together.** Do not commit part of an event, perform network work, and append
the rest later. Emission helpers flush by default so the caller owns the transaction boundary.
5. **One currency row, one user balance.** Worker/team IDs and counter units never belong in `KudosLedger`. Emit
separate debit and credit rows; do not encode a transfer as an opaque net amount.
6. **Reserve before authorizing eventual spend.** Under `acquire_payer_lock`, call `reserve_kudos` with a stable
`business_id`. Attach that reservation ID to matching debit metadata. Never use `effective_kudos` to authorize a
spend.
7. **Do not fund from pending credits.** `available_kudos` deliberately ignores them. Changing that rule requires a
proof covering credit rollback, event ordering, and projector failure.
8. **Preserve mode ownership.** Business code emits events in both modes and delegates temporary inline behavior to
`kudos_legacy_projection`. Do not add mode branches to endpoints or settlement methods.
9. **Record non-linear compatibility behavior.** If a rule floors, caps, forgives, or otherwise changes the requested
signed delta, emit the difference as its own typed posting so replay matches the projection.
10. **Capture historical dimensions at emission.** Team membership and similar attribution must be stamped on the
event. The projector must not infer past ownership from mutable current relationships.
11. **Use decimal accounting values.** Convert through `Decimal(str(value))` and round/quantize at the existing
two-decimal boundary. Never fold currency through binary-float accumulation.
12. **Keep the archive immutable.** Do not update old amounts, retarget postings, delete applied history, or toggle
`applied` as a repair. Emit a compensating posting.
13. **Add producer, fold, retry, and recovery tests.** At minimum pin the emitted rows and materialized result. A new
spend path also needs concurrent overspend coverage; a new retryable path needs replay/conflict coverage.
Direct materialized assignments are allowed only for initial data/bootstrap before an accounting movement exists,
or inside the shadow/projector/reconciliation implementation described here. A test fixture that needs an opening
balance should assign it directly and commit rather than creating misleading ledger history.
## Producer inventory
This is the current inventory of paths that create user currency or kudos-derived projection events.
| Business event | Entry types and targets | Primary code |
| --- | --- | --- |
| Waiting-prompt activation tax | Requester `GENERATION` debit; user usage/record/activity stats; upfront reservation consumption | `WaitingPrompt._activate`, `WaitingPrompt.record_usage`, `User.record_usage` |
| Image/text generation settlement or cancellation settlement | Requester `GENERATION` debit; owner spendable/escrow credit; worker kudos/contribution/fulfilment; user records; team aggregates | `ProcessingGeneration.record`, `WorkerTemplate.record_contribution`, `User.record_usage`, `User.record_contributions` |
| Interrogation form settlement/cancellation | Same currency pattern for requester/worker owner plus interrogation worker counters | `InterrogationForms.record`, `InterrogationWorker.record_contribution` |
| Worker uptime interval | Owner `UPTIME_REWARD` spendable/escrow credit and worker display/stat credit | `WorkerTemplate.record_uptime`, `User.record_uptime` |
| Trust threshold crossing | `EVALUATION_PROMOTION` escrow debit and spendable credit | Projector promotion/drain helpers; shadow compatibility projector |
| User transfer | Paired `TRANSFER` source debit/destination credit, transfer log, payer reservation | `transfer_kudos` and username/API-key wrappers |
| Monthly/recurring grant | `AWARD` user credit | `User.modify_monthly_kudos`, `User.receive_monthly_kudos` |
| Rating/aesthetic reward | `AWARD` user credit | Stable API rating/aesthetic endpoints |
| KoboldAI or other application award | `AWARD` user credit | Kobold and base API award endpoints |
| Style-owner reward | `STYLE_REWARD` user credit and user style record | `User.record_style` |
| Administrator adjustment | `ADMIN_ADJUSTMENT` signed user delta | User administration API through `User.modify_kudos` |
| Minimum-balance forgiveness | Already-applied `FLOOR_ADJUSTMENT` user credit | User projection helper in shadow mode; `_apply_user_deltas` in ledger mode |
| Reconciliation repair | Deterministic `RECONCILIATION` user spendable/escrow delta | `reconcile_balances(..., apply_repairs=True)` |
`User.modify_kudos` also emits the matching per-action user-kudos statistics event. `WorkerTemplate.modify_kudos`
emits worker display/stat events, not currency. Producer wrappers should retain that distinction.
Two interrogation contracts in that table correct behaviour the inline implementation had:
- An interrogation uptime crossing grants an untrusted owner exactly one reward, routed wholly to the spendable
balance by the alchemist bypass and minting no escrow, where the inline path credited both the balance and escrow
for the same crossing (pinned by
`tests/unit/test_kudos_invariants.py::test_interrogation_uptime_grants_untrusted_owner_exactly_one_reward`).
- Cancelling an interrogation form a worker is still processing settles it like a submission, crediting the worker
and owner and debiting the requester the form kudos plus burn, where the inline path moved the form to `CANCELLED`
ahead of the settle check and so settled nothing (pinned by
`tests/unit/test_kudos_invariants.py::test_cancelling_in_flight_interrogation_form_settles_like_a_submit`).
## Projection target inventory
| Materialized target | Event source | Projector rule |
| --- | --- | --- |
| `users.kudos` | Non-escrow currency postings | Sum per user, apply in user-ID order, clamp to `get_min_kudos`, record floor difference |
| `users.evaluating_kudos` | Escrow currency postings | Sum per user; promotion is evaluated after the fold |
| `users.last_active` | `STAT_ACTIVITY`/touch detail | Keep the latest event timestamp |
| `user_stats.value` | `user_kudos` stat record | Atomic insert-or-increment by `(user_id, action)` |
| `user_records.value` | `STAT_RECORD` | Atomic insert-or-increment by `(user_id, record_type, record)` |
| `workers.kudos` | `worker_kudos` stat record | Sum per worker |
| `worker_stats.value` | `worker_kudos` stat record | Atomic insert-or-increment by `(worker_id, action)` |
| `workers.contributions` | `STAT_CONTRIBUTION` / `contributions` | Sum thing deltas per worker |
| `workers.fulfilments` | `STAT_CONTRIBUTION` / `fulfilments` | Sum count deltas per worker |
| `teams.kudos` | Team-stamped worker kudos events | Sum per captured team ID |
| `teams.contributions` | Team-stamped contribution events | Sum per captured team ID |
| `teams.fulfilments` | Team-stamped fulfilment events | Sum per captured team ID |
| Reservation remainder/release time | Currency debit metadata or completed transfer event | Consume request holds by debit; release transfer hold only after every event posting is applied |
| Trusted user role and worker pause state | Projected escrow crossing threshold | Promote eligible mature user, unpause workers, then emit the escrow drain pair |
Currency rows and statistic rows are each claimed in a bounded batch. A single event may cross a batch boundary;
therefore any event-wide side effect must query for remaining unapplied rows rather than assuming a batch contains the
whole event.
## Consumer and read-model inventory
Choosing the correct read is part of the mutation contract. A blanket replacement of every `.kudos` access is wrong
because account currency, queue priority, worker attribution, job price, and shared-key quota have different meaning.
### User currency consumers
| Consumer | Current read | Consistency and rule |
| --- | --- | --- |
| Upfront image/text request admission | `reserve_kudos` -> `available_kudos` | Spend-safe and conservative; includes floor, holds, and queued debits; ignores queued credits |
| Interrogation admission and interrogation-worker check | `reserve_kudos` / `available_kudos` | Spend-safe; retryable form reactivation reuses its business ID |
| User transfer | `reserve_kudos` -> `available_kudos` | Payer-serialized; recipient is not locked; optional API idempotency key protects replay |
| Admin adjustment “new balance” response | `effective_kudos` | Includes the just-committed/unapplied delta; not an authorization value |
| Queue priority for image/text | `user.kudos` copied to `waiting_prompt.extra_priority` at activation | Snapshot of a potentially lagging projection; queue ordering thereafter uses `extra_priority` |
| Queue priority for interrogation | `user.kudos` copied to `interrogations.extra_priority` at construction | Snapshot of a potentially lagging projection |
| Stable worker's secondary upfront eligibility check | materialized `waiting_prompt.user.kudos` minus floor | Eventual legacy recheck; initial admission hold prevents overspend, but lag may transiently alter scheduling eligibility |
| User details, login/welcome, status and ordinary API display | materialized `user.kudos`/`evaluating_kudos` | Eventually consistent in ledger mode |
| Award endpoints returning `new_kudos` | generally materialized `user.kudos` | May return the pre-projection value unless the endpoint explicitly uses `effective_kudos` |
| User listing/sorting and inactive-account heuristics | materialized `User.kudos` | Eventually consistent; intended for administrative/read-model behavior, not spend admission |
| Trust promotion | materialized `evaluating_kudos` inside the serialized projector | Evaluated after fold; final qualifying event promotes without another request |
| Snapshot/reconciliation | materialized user columns plus applied ledger totals | Repeatable-read baseline; repair only through compensating postings |
The two legacy materialized reads in worker scheduling are documented rather than hidden. If they are changed, the
maintainer must decide the intended policy: `available_kudos` is conservative spend capacity; `effective_kudos`
includes queued income; the materialized value is a cheap, lagging priority/display projection. They are not
interchangeable.
### Derived-stat consumers
| Projection | Consumers |
| --- | --- |
| `workers.kudos`, `worker_stats` | Worker detail/API reward totals and per-action breakdowns |
| `workers.contributions`, `workers.fulfilments` | Worker details, top-worker queries, global totals, performance/leaderboard views |
| `teams.kudos`, `teams.contributions`, `teams.fulfilments` | Team detail/API aggregates and rankings |
| `user_stats` | User per-action kudos breakdown returned by user detail paths |
| `user_records` | User usage/contribution/request/fulfilment records, contributor queries, statistics endpoints |
| `users.last_active` | Account activity and stale-account lifecycle logic |
| Applier heartbeat and queue/reservation ages | Metrics, `/api/v2/status/heartbeat`, operator status command, cutover gates |
All of these can lag in ledger mode. Consumers that need transaction-local proof of an accepted event should inspect
the event/result they just created, not force the projector or assume a refreshed aggregate.
### Excluded kudos-like state
The following kudos-adjacent state is not a user currency projection and must not be routed through
`KudosLedger` without a separate design decision:
| State | Meaning and mutation rule |
| --- | --- |
| `UserSharedKey.kudos` and `utilized` | Per-key quota/budget, including `-1` for unlimited; consumed inline with the request, because the clamp at zero is an enforcement read in the request path and a separate endpoint sets the budget absolutely |
| `WaitingPrompt.kudos`, `consumed_kudos`, generation/form kudos | Estimated price, accumulated job cost, or reward returned for one job; kept inline because they are per-prompt rows rather than shared aggregates and the same event already emits a currency posting |
| `WorkerTemplate.kudos` | Worker-attributed display total; projected from `KudosStatEvent` |
| `Team.kudos` | Team-attributed display total; projected from worker stat events stamped with `team_id` |
| `Style.kudos`, `Style.contributions`, `Style.fulfilments` | Style-level popularity/contribution counters; still maintained by the style subsystem |
| `KudosTransferLog` | Auxiliary transfer history and policy/audit record; does not replace the paired currency postings |
| Test-user bootstrap balance | Absolute fixture/local-login opening state; direct assignment avoids inventing an accounting movement |
| `workers.last_reward_uptime` | An absolute cursor set to the worker's current uptime at each reward crossing, which a signed-delta fold cannot express |
| `workers.aborted_jobs` | Resets to zero on hourly rollover and on tripping the drop threshold, so its value depends on event ordering and timing rather than a sum, and the threshold decision is read in the same path that increments it |
## Reservation lifecycle
| Operation | Stable business ID | Created/reactivated | Consumed/released |
| --- | --- | --- | --- |
| Image/text upfront admission | `upfront:<waiting-prompt-id>` | Before activation is accepted | Debit projection consumes it; prompt deletion/cancel paths and the expired-prompt cleanup release unused remainder |
| Interrogation upfront admission | `interrogation:<form-id>` | When a worker claims a waiting form | Debit projection consumes it; cancellation/failure paths and the expired-interrogation cleanup release it; a retry reactivates the row |
| Transfer | `transfer:<source-user-id>:<event-id>` | Before paired postings commit | Released only after no unapplied posting remains for the event |
`business_id` uniqueness supplies retry stability. Reactivation may not change payer. Reservation mutation uses row
locks only after payer-scoped admission serialization; release functions are idempotent and return zero when no active
hold remains. `available_kudos` reads the balance, active holds, and queued debits in one statement so all terms come
from a single database snapshot; a fold or release committing mid-read cannot hide a debit from every term.
## Concurrency and lock reference
| Primitive | Scope | Acquired by | Purpose |
| --- | --- | --- | --- |
| Applier advisory transaction lock | Installation | Projector and mode transition | At most one database projector; serialize final drain with projection |
| Payer advisory transaction lock | User ID | Reservations/transfers/admission | Prevent concurrent promises from overspending one payer without locking recipients |
| Reconciliation advisory transaction lock | Installation | Repair mode | Prevent concurrent compensation emitters |
| Control-row key-share lock | Mutation transaction | `get_kudos_ledger_mode` | Pin mode until the writer commits |
| Control-row exclusive lock | Installation | `set_kudos_ledger_mode` | Wait for every old-mode writer before ownership changes |
| Event-row `FOR UPDATE SKIP LOCKED` | Bounded batch | Projector | Claim exact unapplied work without a watermark |
| Reservation-row `FOR UPDATE` | Business ID | Consume/release | Serialize hold depletion/release |
| Repeatable-read transaction | Snapshot/reconciliation command | Reconciliation helpers | Observe balances and applied totals from one consistent database snapshot |
Required accounting lock order is applier lock before exclusive control-row lock. Projection targets are visited in
stable ID order. Code that needs another accounting lock must document where it fits before it is merged. The legacy
`USE_SQLITE` runtime mode short-circuits PostgreSQL advisory locks and therefore cannot validate these concurrency
properties.
## Failure and recovery invariants
- A projector cycle commits both target updates and exact applied flags, or neither.
- A row is selected by `applied = false`, never by `id > watermark`.
- A stopped projector is a lag incident, not lost work; restore it and drain in ledger mode.
- A transfer hold remains active until both sides of the event are materialized, even when a batch splits the event.
- A trusted user's positive escrow with no in-flight promotion pair is self-healed by a later projector cycle.
- Minimum-balance flooring creates an explicit `FLOOR_ADJUSTMENT` so snapshot replay equals projection.
- A snapshot records both opening columns and the applied totals visible at that instant.
- Reconciliation excludes prior `RECONCILIATION` entries from its movement baseline and emits one deterministic repair
event per snapshot/user.
- Ledger-to-shadow transition owns the applier lock, waits for ledger-mode writers, drains the final tail, and changes
mode in one transaction.
- Applied currency and statistic history is permanent. `prune_applied_kudos_ledger` is intentionally a no-op.
Never recover by deleting ledger/stat rows, editing old amounts or targets, changing `applied`, or directly assigning a
production balance. Preserve evidence and emit compensation. See the [operations guide](../how-to/kudos_ledger_operations.md)
for the executable procedure.
## Observability contract
`kudos_applier_health` reports:
| Field | Meaning |
| --- | --- |
| `pending_rows` | Combined unapplied currency and statistic event count |
| `oldest_pending_seconds` | Age of the oldest unapplied event across both queues |
| `heartbeat_seconds` | Time since the projector last completed a cycle; may grow even when no rows are pending |
| `active_reservations` | Count of holds with positive remaining amount and no release time |
| `oldest_reservation_seconds` | Age of the oldest active hold |
The background task runs every three seconds and keeps folding within a tick while full batches drain, up to a bounded
number of catch-up cycles, so a backlog clears at many batches per tick while each fold stays a small transaction. The
heartbeat endpoint reports `DEGRADED` when the oldest pending event exceeds 30 seconds. Operators should alert on queue age and reservation age, not only row count: a steady queue can be
healthy under load, while one old row or hold can indicate a poisoned path.
## Code map
| File | Responsibility |
| --- | --- |
| `horde/classes/base/kudos.py` | Typed models, event context, mode control, currency/stat emission primitives |
| `horde/database/kudos_ledger.py` | Bounded claim/fold transaction, trust promotion, floor recording, health |
| `horde/database/kudos_reservations.py` | Available/effective balance calculations and hold lifecycle |
| `horde/database/kudos_reconciliation.py` | Repeatable-read snapshots, drift calculation, compensating repair emission |
| `horde/database/kudos_db.py` | PostgreSQL advisory-lock and isolation primitives |
| `horde/database/kudos_counters.py` | Dialect-specific typed atomic counter upsert |
| `horde/database/kudos_legacy_projection.py` | Temporary shadow-mode inline projection; the intended cutover deletion boundary |
| `horde/classes/base/user.py` | User currency, usage, contribution, uptime, style, and trust producer surface |
| `horde/classes/base/worker.py` | Worker display/stat producer surface and uptime/contribution orchestration |
| `horde/classes/base/processing_generation.py` | Generation settlement event boundary and reservation cleanup |
| `horde/classes/base/waiting_prompt.py` | Activation tax, requester usage debit, queue-priority snapshot |
| `horde/classes/stable/interrogation.py` | Interrogation reservation, settlement event, and cleanup |
| `horde/database/functions.py` | Transfer admission, paired postings, idempotency validation |
| `horde/database/threads.py` | Periodic projector invocation and health metric recording |
| `horde/enums.py` | Stable accounting discriminators and metadata keys |
| `sql_statements/5.1.0.txt` | Idempotent production schema migration and counter uniqueness preparation |
| `tools/kudos_ledger_admin.py` | Status, drain, snapshot, reconcile/repair, and mode commands |
| `docs/how-to/kudos_ledger_operations.md` | Cutover, rollback, disaster-recovery, and rehearsal how-to |
## Regression-test map
| Contract | Primary coverage |
| --- | --- |
| Emission shape, event grouping, currency/stat separation | `tests/unit/test_kudos_ledger.py`, `test_kudos_balance.py`, `test_kudos_safety.py` |
| Existing generation, cancellation, uptime, monthly, interrogation semantics | `test_kudos_settlement.py`, `test_kudos_image_cancel.py`, `test_kudos_uptime.py`, `test_kudos_monthly.py`, `test_kudos_interrogation.py` |
| Counter and team fold equivalence | `test_kudos_counters.py`, `test_kudos_counter_fold.py` |
| Concurrent projector exclusion and spend reservation | `test_kudos_safety.py` on PostgreSQL-capable paths |
| Transfer replay and parameter conflict | `test_kudos_safety.py`, integration kudos endpoint tests |
| Trust promotion on final qualifying posting | `test_kudos_safety.py`, `test_kudos_ledger.py` |
| Atomic ledger-to-shadow final drain | `test_kudos_safety.py` |
| Snapshot drift, idempotent compensation, floor replay | `test_kudos_safety.py` |
| Remaining activation deadlock retry/failed-session telemetry | `test_wp_activate_deadlock.py` |
| Image/text/interrogation upfront admission behavior | Integration request-activation tests and `test_kudos_interrogation.py` |
| Migration idempotency against mapped schema | `test_kudos_safety.py` |
The unit suite provisions PostgreSQL through testcontainers, so the advisory-lock and concurrent-projector tests
exercise the real primitives. The legacy `USE_SQLITE` runtime mode short-circuits those locks and is not covered by
the suite. Before cutover or a recovery rehearsal, run the safety and integration suites as required by the
operations guide.
## Review checklist for a new kudos change
- Is the value currency, a display attribution, a counter, a price, or a quota?
- What is the business event boundary, and which postings share its `event_id`?
- Is the operation externally retryable? If so, what stable key and parameter-conflict rule make it idempotent?
- Can it authorize spend? If so, where is the payer lock and stable reservation?
- Which projection owns the read, what lag can its consumer tolerate, and should the consumer use materialized,
available, or effective balance?
- Does the producer capture mutable attribution such as team membership at event time?
- Does the change introduce another lock? Where does it fit in the documented order?
- Can a batch split the event, and does any event-wide side effect handle that case?
- Is every non-linear adjustment represented by a posting?
- Can the change be rolled back through the supported mode transition without running old code against new holds?
- Do tests cover emission, fold, concurrency/retry behavior, and recovery—not merely the final number?
- Have this reference, the explanation, and the operations guide been updated when the contract changed?