--- title: "Your first chat completion through the Horde" summary: "Use the OpenAI Python SDK against the Horde's chat front: pick a served model, send and stream a completion, run one tool round and parse a json_schema reply." topics: [chat, requests] order: 10 --- # Your first chat completion through the Horde Topics: [chat](../topics.md#chat), [requests](../topics.md#requests) This tutorial takes the official OpenAI Python SDK from installation to a structured reply generated by a Horde worker. Each step ends with output to compare against before moving on. It needs Python, network access to a Horde chat front, and optionally a Horde API key. At the end, [Where to go next](#where-to-go-next) lists the pages that cover each feature in depth. The front's base URL is a deployment value, written here as `OAI_BASE_URL`. For the public horde it is `https://oai.aihorde.net/v1`. ## 1. Install the SDK ```bash python -m pip install "openai>=1.109,<2" ``` The Horde's conformance tests run against this range. Check the installation: ```bash python -c "import openai; print(openai.__version__)" ``` The output is a version such as `1.109.1`. ## 2. Set the base URL and a key The front takes a Horde API key as the bearer token. Without a registered key, use the anonymous key `0000000000`; anonymous requests are served after requests from users with kudos, so they wait longer when the queue is busy. ```bash export OAI_BASE_URL="https://oai.aihorde.net/v1" export HORDE_API_KEY="" ``` Start Python and build the client: ```python import json import os from openai import OpenAI client = OpenAI(base_url=os.environ["OAI_BASE_URL"], api_key=os.environ["HORDE_API_KEY"]) print(client.base_url) ``` The output is the base URL you exported, with a trailing slash. ## 3. Pick a model with workers online `GET /v1/models` lists only models that at least one online chat worker serves. Each entry includes the Horde's details beside the OpenAI fields. The SDK exposes them through `model_extra`. ```python for card in client.models.list(): details = card.model_extra print( card.id, "workers:", details["x_horde"]["workers"], "max_completion_tokens:", details["top_provider"]["max_completion_tokens"], "supported:", details["supported_parameters"], ) ``` Each line looks like this: ```text koboldcpp/Meta-Llama-3.1-8B-Instruct workers: 2 max_completion_tokens: 512 supported: ['max_tokens', 'temperature', 'top_p', 'stop', 'seed', 'tools', 'tool_choice', 'parallel_tool_calls', 'response_format'] ``` Choose a model whose `supported_parameters` include `tools` and `response_format` for the later steps. These are the combined declarations of its online workers. They do not guarantee that one worker supports every combination, JSON Schema or the guarantees required by a strict request. See [Conformance](../how-to/send_chat_completions.md#conformance) if a later step reports missing declarations. Keep the reply length within the card's `max_completion_tokens`: ```python MODEL = "" MAX_TOKENS = 200 ``` An empty list means no chat worker is online. The list changes as workers come and go, so a model can disappear between steps. List again and pick another. ## 4. Send a completion ```python completion = client.chat.completions.create( model=MODEL, messages=[ {"role": "system", "content": "You are a terse assistant."}, {"role": "user", "content": "Name one colour."}, ], max_tokens=MAX_TOKENS, ) print(completion.choices[0].message.content) print(completion.choices[0].finish_reason) print(completion.model) print(completion.usage) print(completion.model_extra["x_horde"]) ``` The output has the reply, `stop` (or `length` when the reply reached `MAX_TOKENS`), the model that generated it, and the token counts: ```text Blue. stop koboldcpp/Meta-Llama-3.1-8B-Instruct CompletionUsage(completion_tokens=3, prompt_tokens=21, total_tokens=24, ...) {'request_id': '...', 'worker_id': '...', 'worker_name': '...', 'kudos': ..., 'charge_basis': 'output_estimate', 'estimated_usage': False, ...} ``` `x_horde` is the Horde's record of the request: which worker served it and the kudos it cost. `estimated_usage: True` means the worker did not report token counts and `usage` is an estimate. An `x_horde["notice"]` appears when the worker returned no visible text. The request waits in the Horde's queue until a worker picks it up, so a reply can take from a few seconds to minutes. ## 5. Stream the completion ```python stream = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "Name one colour."}], max_tokens=MAX_TOKENS, stream=True, stream_options={"include_usage": True}, ) for chunk in stream: if chunk.choices: choice = chunk.choices[0] print("delta:", choice.delta.role, repr(choice.delta.content), "finish:", choice.finish_reason) if choice.finish_reason: print("x_horde:", chunk.model_extra["x_horde"]["request_id"]) else: print("usage:", chunk.usage) ``` For a buffered reply without reasoning, the output can look like this: ```text delta: assistant None finish: None delta: None 'Blue.' finish: None delta: None None finish: stop x_horde: ... usage: CompletionUsage(completion_tokens=3, prompt_tokens=21, total_tokens=24, ...) ``` A buffered reply arrives in one content chunk. When a worker posts live progress, the front can send several content chunks even for a short reply. Posts usually arrive about once a second and are paced into smaller chunks. The finish and usage chunks still appear at the end. While the request waits, the SDK reads and discards keepalive comments. ## 6. Complete a tool round A tool round has four parts: the request offers a tool, the model returns a call, your code runs the function, and a second request sends the result back for the model to use in its reply. Define the tool and the local function: ```python WEATHER_TOOL = { "type": "function", "function": { "name": "get_weather", "description": "Read the current weather of a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}, }, } def get_weather(city): return f"18C and clear in {city}" ``` Ask a question that needs it: ```python messages = [{"role": "user", "content": "What is the weather in Paris? Use the tool."}] calling = client.chat.completions.create(model=MODEL, messages=messages, tools=[WEATHER_TOOL], max_tokens=MAX_TOKENS) choice = calling.choices[0] print(choice.finish_reason, choice.message.content) for tool_call in choice.message.tool_calls or []: print(tool_call.id, tool_call.function.name, tool_call.function.arguments) ``` The output is the finish reason `tool_calls`, a null content, and the call: ```text tool_calls None call_... get_weather {"city":"Paris"} ``` A model can answer in prose instead of calling the tool, with `finish_reason` `stop`. To require the call, send `tool_choice={"type": "function", "function": {"name": "get_weather"}}` with the same request, using a model whose card lists `tool_choice`. This requires the worker's `tool_choice_enforced` guarantee under default strict conformance. Continue below only if the reply includes a tool call. If it returned text instead, retry the tool request with the forced choice above on a supporting model. The assistant message is appended as returned by the SDK, followed by one `tool` message per call with the call's `id`: ```python messages.append(choice.message) for tool_call in choice.message.tool_calls: arguments = json.loads(tool_call.function.arguments) messages.append({"role": "tool", "tool_call_id": tool_call.id, "content": get_weather(arguments["city"])}) answering = client.chat.completions.create(model=MODEL, messages=messages, tools=[WEATHER_TOOL], max_tokens=MAX_TOKENS) print(answering.choices[0].finish_reason) print(answering.choices[0].message.content) ``` The output is `stop` and a sentence that uses the result, such as `It is 18C and clear in Paris.` A `tool` message includes `role`, `tool_call_id` and `content`. Any other key on it, such as `name`, is dropped and not forwarded, as OpenAI ignores it. ## 7. Ask for a JSON reply and parse it A `json_schema` response format asks the worker to constrain the reply to a schema. The `schema` key is required. ```python COLOUR_FORMAT = { "type": "json_schema", "json_schema": { "name": "colour", "strict": True, "schema": { "type": "object", "properties": {"colour": {"type": "string"}}, "required": ["colour"], "additionalProperties": False, }, }, } structured = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "Name one colour."}], response_format=COLOUR_FORMAT, max_tokens=MAX_TOKENS, ) document = json.loads(structured.choices[0].message.content) print(document) ``` The output is a dictionary with exactly the schema's key, such as `{'colour': 'blue'}`. ## Where to go next - [Send chat completions through the OpenAI-compatible front](../how-to/send_chat_completions.md) covers every request key the front accepts or refuses, the `horde` extension object, images, errors and retries, and limits. - [Add chat completions to a web frontend](../how-to/add_chat_to_a_frontend.md) covers calling the front from a browser, parsing the stream by hand and running the tool loop in a client. - [Chat front reference](../reference/chat_front.md) holds the request keys, message roles, response and stream shapes, and the model list fields. - The [chat requests explanation](../explanation/chat_requests.md) covers how a chat request travels from the front to a worker and back.