OctoFlux: one gateway for many AI providers

OctoFlux is a product we build and run ourselves. It is an OpenAI-compatible LLM gateway: a service that sits between an application and the OpenAI-compatible AI providers it uses, and decides where each request should go. This article explains the problem it addresses and how it works.

The problem with talking to providers directly

An application that calls a language model usually starts by calling one provider with one API key. That is fine until the application has to be dependable. Providers apply rate limits. Keys run out of quota. A service that worked this morning returns errors this afternoon. Different providers offer different models, and the best choice for a request can change.

Handling all of that inside the application means writing retry logic, tracking which keys are healthy, and adding special cases for every provider. That code has little to do with what the application is for, and it has to be written again in every project.

What OctoFlux does

OctoFlux moves that work out of the application and into a gateway. The application sends its requests to OctoFlux using the same API format it would use for OpenAI. OctoFlux then handles provider selection, API-key rotation, local rate limiting, health tracking, retries and failover.

Because the API is OpenAI-compatible, an OpenAI-compatible SDK can be pointed at OctoFlux by changing its base URL and API key. The design goal is to keep the API the application sees stable while making the providers behind it interchangeable. OctoFlux works with Groq, OpenRouter, NVIDIA NIM and other providers that expose OpenAI-compatible APIs.

How requests are routed

Routing in OctoFlux is driven by configuration and by what the gateway knows about the current state of each provider:

• Providers and models have priorities, so preferred options are tried first
• Routing is health-aware, so providers and keys that are failing are avoided
• API keys are rotated, and rate limits are enforced locally for each provider and key
• Model aliases let an application ask for a name of your choosing
• Asking for the model "auto" lets the gateway choose where the request goes

Routing is also capability-aware, so chat requests and embedding requests are sent to models that can serve them.

What happens when something fails

Failover is where a gateway earns its place, and it is also where a careless one causes harm. OctoFlux is deliberate about which failures are worth retrying. It retries rate limits, timeouts, connection failures and server errors, rotating to another key or provider as it does so.

It does not retry invalid requests, and it does not retry requests that fail because they exceed a model's context length. Sending those again would only repeat the same failure somewhere else. Retry attempts are bounded, so a request cannot loop indefinitely. For streamed responses, failover happens before the first byte of the response is sent, so a response is never switched to another provider partway through. If a stream fails after it has started, the connection ends.

Seeing what is going on

A gateway that hides problems is not much better than no gateway. OctoFlux tracks health for each provider and for each key, including cooldown periods, and probes providers in the background. It writes structured JSON logs with request IDs, exposes a metrics endpoint, and keeps usage counters.

Those counters cover requests, tokens, latency, fallbacks, timeouts and rate limits. An admin API and a web dashboard give access to the same information, and providers and keys can be tested by hand.

Adding and changing providers

Providers are described in configuration, not in code. A new provider can be added without writing Python, with secrets supplied through environment variables, limits set for each provider and each key, and custom authentication headers where a provider needs them. Configuration can be reloaded while the gateway is running.

Where to find it

OctoFlux is built with Python and FastAPI, and its code is public on GitHub at github.com/octoprograms/Octo-Flux. The repository includes the documentation and an architecture note that sets out the design decisions, the trade-offs and what the project deliberately does not try to do. You can see it running at octoflux.vercel.app.

If your application depends on language models and you would like help making that dependable, get in touch with us.