tazlin
b0e6f94c3b
- `text_wp_speed_gate_clause` offers a chat request above the popping worker's lease capacity when no capable chat worker is online, through a correlated NOT EXISTS over the workers table. A capable worker is online, accepts chat jobs for one of the request's models, is not in maintenance or paused, declares chat limits that admit the request, declares every chat feature the request requires and has a capacity of at least its `max_length` (`capable_chat_worker_clauses`, `text_worker_lease_capacity_covers_clause` in `horde/classes/kobold/worker.py`) - `TextArrayContainsAll` in `horde/classes/base/sql_constructs.py` compiles the feature test per dialect: `@>` over coalesced `text[]` on PostgreSQL, and a `json_each` anti-join on SQLite that reads SQL NULL and JSON `null` as an empty array. A faster worker without a required feature therefore does not keep a request from a slower worker that has it - `TextWorker.can_generate_with_eligibility_inputs` returns `lease_speed` only while a capable worker is online. It reads the fact through `TextWaitingPrompt.has_capable_chat_worker`, which keeps the result on the request object for the request's current lengths. Availability counts the slower workers, so such a request is possible and is not refused as `no_eligible_worker` - `CHAT_WORKER_ONLINE_WINDOW` is defined in `horde/classes/kobold/worker.py`, so the capable worker test and the chat model queries share one online window - an over-capacity chat assignment stores `over_capacity_at_assignment` and is counted by the `horde.text.gate_shadow.*` instruments as a legacy one is. Those instruments gain a `horde.chat` attribute - the front's finished-request line ends with `max_tokens` and `max_tokens_source`, and `horde.oai.requests.errors` gains `horde.max_tokens_bucket`, the upper bound of the limit's `BUCKETS_MAX_TOKENS` bucket or `over`, so refusals can be read by the requested limit - ADR 0041 records the preference. ADR 0038, ADR 0040, the chat request and chat front references and the chat worker how-to describe it - unit tests cover the pop clause and the eligibility check with and without a capable worker, faster workers that cannot take the request, the unmeasured speed, availability, the required features, the containment on both dialects, the chat shadow records and the front label and log fields. Integration tests cover a slow sole worker receiving the request and a slow worker passed over for a capable one
AI-Horde: Community-Powered AI Generation
The AI Horde is a free community service that lets anyone create AI-generated images and text. In the spirit of projects like Folding@home (sharing compute for medical research) or SETI@home (sharing compute for the search for alien signals), AI Horde lets volunteers share their computer power through workers to help others create AI art and writing.
When you make a request - like asking for "a painting of a sunset over mountains" - the AI Horde system finds available volunteer computers that can handle your job. It's similar to how ride-sharing apps connect passengers with nearby drivers, but instead of rides, you're getting AI-generated content.
The system uses "kudos" points to keep things fair. Workers earn kudos when their computers help process requests, which they can use to get priority for their own requests or leave unspent to help others. Importantly, kudos can never be bought or sold - this is strictly against the Terms of Service. If you would like to learn more about kudos, see our detailed explanation.
What makes AI Horde special is that it's completely free and community-run, with a strong commitment to staying that way. The kudos system is specifically designed to ensure that access to these resources remains equitable. While users with more kudos get faster service, anyone can use it for free, even anonymously, and kudos never expire.
The AI-Horde hopes to ensure that everyone gets a chance to use these exciting AI technologies, regardless of their financial means or technical resources. You can read more about why we do this in the motivations document.
Sponsors
Table of Contents
- AI-Horde: Community-Powered AI Generation - Table of Contents
- Sponsors
- Technical Introduction
- Getting Started with AI Horde
- Contribute your GPU (Become a worker)
- Integrate with the AI-Horde
- Community
Technical Introduction
The AI Horde is an enterprise-level ML-Ops crowd-sourced distributed inference cluster for AI Models. This middleware supports both image and text generation, making it highly versatile. It is designed to be highly scalable, allowing for seamless drop-in/drop-out of compute resources.
Key Features
- Scalability: The system can scale effortlessly to accommodate varying workloads, ensuring efficient use of available resources.
- Flexibility: Supports both image and text generation, catering to a wide range of AI applications.
- Community-Powered: Relies on spare/idle resources provided by the community, making advanced AI accessible to everyone.
Public Version
The public version allows individuals without powerful GPUs to use advanced AI models like Stable Diffusion or Large Language Models (LLMs) such as Pygmalion/Llama. This is achieved by leveraging the spare computing power provided by the community, running software we call 'workers'. Additionally, it supports clients, such as third party websites, games and apps, to utilize AI-generated content through a REST API. You can see the public instance's performance and application statistics on our grafana instance.
Private Deployment
The AI Horde middleware can also be deployed privately within any enterprise environment. It can be installed within hours and can scale your ML solution within days of deployment, providing a robust and flexible solution for enterprise-level AI needs.
Technologies Used
- Python: 3.9 or later
- PostgreSQL
- Redis: For caching
- Docker support for full containerization
How It Works
For a high-level overview of how the AI Horde operates, including diagrams of the request/job lifecycle, see the request/job lifecycle explanation.
Maintainers and contributors can find repository-specific architecture documentation in the local documentation index, including the kudos accounting explanation, reference, and operational procedure.
There are also individual readmes for each specific mode supported by the AI Horde.
For common questions, check the FAQ. If you'd like more details on features of the horde or the reasoning behind it, see the documentation which goes greater into depth.
Getting Started with AI Horde
You can use any service powered by the AI Horde either with a registered account or anonymously. You can find a partial list of services powered by the AI-Horde on our main website.
OAuth2 Registered Account (Recommended)
Note: The only information we store from your account is your unique ID. We do not use your id for any other purpose. See our privacy policy for more details.
- Visit AI Horde Registration
- Log in using one of the supported OAuth2 services
- Choose your username and receive your API key
- Benefits:
- Start with higher priority in generation queues
- Can change username or reset API key if needed
- Maintain and track your kudos balance
- Can recover account access through OAuth2 service
- Minimum kudos balance of 25
Pseudonymous Account
- Visit AI Horde Registration
- This method is the default if you do not log in with an OAuth2 service (google, github, discord, etc)
- Important: If you lose your API key, the account cannot be recovered
- Cannot change username or reset API key
- Still earns and maintains kudos
- Still better priority than anonymous users
Anonymous Usage
- Use API key '0000000000'
- Lowest priority in generation queues
- No kudos tracking
- No need to register
- Service may be restricted for anonymous users during high load
Why Register?
The main benefit of registration is participating in the kudos system. Kudos determine your priority in the queue - the more kudos you have, the faster your requests are processed. You can earn kudos by:
- Running a worker to help process other users' requests
- Receiving kudos as a thank-you for donations (though kudos cannot be directly bought or sold)
- Being online and available as a worker
- Participating in the kudos economy on the official discord by logging in with the bot. (Use the command
/loginin any channel, click the blue button and enter your API key at the popup).
Remember: AI Horde is committed to remaining free and accessible. While the kudos system provides priority benefits, it's designed to encourage community contribution rather than commercialization. All users, even anonymous ones, can access the service's core features. Read about this on the official developer's blog.
Contribute your GPU (Become a worker)
Be sure to register first.
Image
The officially supported way to add your GPU for image generation is by running the worker software horde-worker-reGen.
Text
There are multiple ways to contribute your GPU for text generation:
- AI-Horde-Worker can connect to inference backends and 'bridge' you to the AI-Horde
- KoboldAI
- KoboldCPP
- Aphrodite Engine
- ... and many others. Join the discussion in the #horde channel of the KoboldAI Discord Server to learn more.
Integrate with the AI-Horde
If you want to build an integration to the AI Horde (Bot, application, scripts etc), please consult our Integration Readme
Community
If you have any questions or feedback, we have a vibrant community on discord