mirror of
https://github.com/Haidra-Org/AI-Horde.git
synced 2026-11-06 05:17:31 +00:00
tazlin
1a5d2ee30a
`horde/oai` is a separate ASGI process that serves `/v1/chat/completions` and reaches the Horde over HTTP and Redis; it does not import the Flask application (ADR 0023).
- `FrontSettings` reads every `HORDE_OAI_*` variable through pydantic-settings; an invalid value stops the start and states the variable
- `horde/oai/schema.find_refused_parameter` returns the first unsupported key, role, content part or `horde` option for a 400 that states it, and `drop_default_no_ops` removes the `DEFAULT_NO_OP_PARAMETERS` keys holding their default, since stock clients send them unprompted; `ChatCompletionRequest` validates the rest against the shared `ChatHordeOptions`
- `HordeClient` rotates calls across the configured Horde API roots, skips and backs off a root that refuses or drops a connection, and sends cancels through a separate smaller pool, so slow cancels do not starve submits
- `CompletionSignalListener` holds one pattern subscription per process; after `HORDE_OAI_REDIS_LIVENESS_SECONDS` without a message it sends a PING with a token and subscribes again when the PONG does not arrive in time, because a black-holed connection does not deliver or raise anything
- `ChatRequestWatcher` reads the status on each signal, on a jittered fallback poll that stretches with the reported wait, and once more at the deadline; fail-fast is decided only from a status read in the current wake, and a failed deciding read is retried after `signal_recheck_seconds`, so a transient error does not cancel a request whose worker may have started; a request seen before that answers 404 waits up to the same interval for its faulted signal
- `create_app` serves `POST /v1/chat/completions`, `GET /v1/models`, `GET /v1/models/{id}` and `GET /healthz`; a completion request is read under a byte ceiling and read timeout (413 `RequestTooLarge`, 408 `RequestBodyTimeout`), screened, admitted through a per-source then a global in-flight cap, submitted to `/v2/generate/chat/async`, and answered as `chat.completion` or as server-sent events
- A bearer token is accepted only when every character is printable non-space ASCII, so a token that cannot be sent as a native header is 401 `InvalidAuthorizationHeader`
- `horde/oai/sse.py` buffers a stream: keepalive comments with the queue position while the request waits, then the role, content, finish and optional usage chunks and `[DONE]`; `horde/oai/mapping.py` builds the native body, the completion with `x_horde` details and estimated usage when the worker did not report any, and the OpenAI error envelope
- Once the Horde holds a request, or may hold one after a lost submit answer, every error carries `x-should-retry: false`, so a client SDK does not create a second charged job
- Cancels on disconnect, deadline, fail-fast and shutdown are bounded, shielded from the client's disconnect and sent at most once per request
- `python -m horde.oai` runs the app under uvicorn, trusting `X-Forwarded-For` from `HORDE_OAI_FORWARDED_ALLOW_IPS` and draining open requests for `HORDE_OAI_SHUTDOWN_DRAIN_SECONDS` on stop; `docker-compose.yaml` adds the `aihorde_oai` service
- Runtime dependencies fastapi, uvicorn, httpx, sse-starlette, pydantic-settings and sniffio, dev dependencies respx and openai; sniffio is declared because httpcore imports it on every request and a failed import is not cached
- `tests/unit/oai` drives the app against a respx-mocked Horde and fakeredis: settings, poll cadence, the completion and model routes, error envelopes, the request lifecycle and cancels, screening, admission, the Horde client and the listener's liveness ping
92 lines
2.6 KiB
YAML
92 lines
2.6 KiB
YAML
# SPDX-FileCopyrightText: 2024 Tazlin <tazlin.on.github@gmail.com>
|
|
# SPDX-FileCopyrightText: 2024 ceruleandeep
|
|
#
|
|
# SPDX-License-Identifier: AGPL-3.0-or-later
|
|
|
|
services:
|
|
aihorde:
|
|
build:
|
|
context: .
|
|
dockerfile: Dockerfile
|
|
args:
|
|
AI_HORDE_DEPENDENCY_GROUPS: ${AI_HORDE_DEPENDENCY_GROUPS:-}
|
|
image: aihorde:latest
|
|
container_name: aihorde
|
|
ports:
|
|
- "7001:7001"
|
|
# The port number written in front of the colon (:) is the port number to be exposed to the outside, so if you change it, you can access it with localhost:{changePort}.
|
|
environment:
|
|
# Flask obtains its environment variables from the .env file.
|
|
# If you set a profile, the .env_{PROFILE} file is read instead.
|
|
- PROFILE=docker
|
|
volumes:
|
|
# .env_{PROFILE} is copied into the image when it is built.
|
|
# So that you can change the environment variables without rebuilding the image, mount the .env file.
|
|
- .env_docker:/app/.env_docker
|
|
# Likewise, you can mount the horde directory to change the source code without rebuilding the image.
|
|
- ./horde:/app/horde
|
|
networks:
|
|
- aihorde_network
|
|
depends_on:
|
|
postgres:
|
|
condition: service_healthy
|
|
redis:
|
|
condition: service_started
|
|
|
|
# The OpenAI-compatible chat front (horde/oai). It runs from the same image as the horde and reaches it over the
|
|
# compose network. See docs/how-to/run_openai_chat_front.md.
|
|
aihorde_oai:
|
|
image: aihorde:latest
|
|
container_name: aihorde_oai
|
|
command: ["python", "-m", "horde.oai"]
|
|
ports:
|
|
- "7002:7002"
|
|
environment:
|
|
- HORDE_OAI_HORDE_URLS=http://aihorde:7001/api
|
|
- HORDE_OAI_REDIS_URL=redis://redis:6379/7
|
|
volumes:
|
|
- ./horde:/app/horde
|
|
networks:
|
|
- aihorde_network
|
|
depends_on:
|
|
aihorde:
|
|
condition: service_started
|
|
redis:
|
|
condition: service_started
|
|
|
|
postgres:
|
|
image: ghcr.io/haidra-org/ai-horde-postgres:latest
|
|
container_name: postgres
|
|
restart: always
|
|
environment:
|
|
POSTGRES_PASSWORD: changeme
|
|
volumes:
|
|
# PostgreSQL 18+ keeps major-version-specific directories below this
|
|
# parent so an image update cannot reinterpret an older data directory.
|
|
- postgres_data:/var/lib/postgresql
|
|
ports:
|
|
- "5432:5432"
|
|
networks:
|
|
- aihorde_network
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U postgres"]
|
|
interval: 2s
|
|
timeout: 5s
|
|
retries: 30
|
|
|
|
redis:
|
|
image: redis:7
|
|
container_name: redis
|
|
restart: always
|
|
ports:
|
|
- "6379:6379"
|
|
networks:
|
|
- aihorde_network
|
|
|
|
networks:
|
|
aihorde_network:
|
|
driver: bridge
|
|
|
|
volumes:
|
|
postgres_data:
|