Self-hosted LLM gateway: one key, many models

Why a gateway at all

By mid-2026 our projects were calling models from half a dozen providers: free tiers, a couple of low-cost subscriptions and pay-as-you-go accounts. Each had its own key, its own API quirks and its own way of failing. Every project held its own copies of those keys, and changing a model meant changing code in every project that used it.

A gateway puts one service in the middle. Projects talk to it with one API and one key each. The gateway holds the provider keys, chooses the model, retries elsewhere when a provider fails, and records what every call cost.

How we run it

• Pinned version, config in git. The configuration is deployed by a script that validates it, takes a backup, pushes it, restarts the service and probes it. In September someone hand-edited the live copy three times and the repository fell 73 lines behind, so the next deploy would have quietly reverted all of it. Edits now go through git only.
• Seats named for the job. Projects ask for something like llm-reason or agent-coder, not a vendor's model name. By late September we had 106 seats across five providers, from free tiers to one low-cost paid plan.
• A "council" for second opinions. A set of seats, each on a different model, that can be asked the same question. That only helps if the seats really are different models: when we counted, 42 seats came from only 13 model families, so we moved to one council seat per family.
• Scoped keys. Each project gets a virtual key that may call only the seats it needs. The master key is never handed to a project, and the spend and admin endpoints refuse anything else. The scope is a real gate: one project's key had been limited to a single seat by mistake, and every other seat was refused until we fixed it.
• A daily steward. Since October a scheduled job checks every seat each morning, scouts for replacement models, and applies seat changes that do not raise cost: at most ten a day, each read back and reversible on its own. Anything that changes model family, raises cost or cannot be confirmed goes to a person. Plain code makes those decisions. A language model writes only the summary, because the one we tried was not good enough at the judgement.

What works

1. One key per project, limited to what it needs. Leaking a project key exposes a few seats, not every provider account.
2. Model changes without code changes. When a free model disappears or a better one appears, the seat is repointed and every project follows.
3. Fallback chains. When a provider is down or rate-limited, the request moves to the next model in the chain and the project never sees the failure. Since September, our job-named seats try their main model, then two paid low-cost seats, before any free tier.
4. Expensive models locked out. In August one seat accounted for about 100% of our spend. We repointed the expensive seats to cheaper models, the dearest route fell from $1.25 to $0.15 per million tokens, and we removed the "pass-through" that let any key call any model by name. It had been used zero times. A call naming a paid model directly is now refused.

What bit us

Fallbacks hide dead models. This is the main lesson, and it applies to any gateway with automatic fallback. In July, 10 of the 12 council seats pointed at free models the provider had retired. Every request "succeeded", because each fell back to the same two working models. A 12-seat council was really a 2-seat council in 12 hats. The only sign was in the response: the model field showed a raw model ID instead of the seat name. Later, our health check logged six dead seats every day for nine days before anyone re-seated them. In September a coding request "succeeded" only after three silent fallbacks, answered by the last model in the chain.

What we do now: the morning check calls every seat directly, so a dead main model shows up within a day instead of hiding behind its fallbacks. On ordinary traffic, the model field is the tell, and a fallback that fires is a fault to look at, not a success.

Free model lists change weekly. In July the provider's free tier was down to 14 models, and 10 of the ones in our config were gone. In one week in August a whole model family vanished from all four free providers at once, another provider dropped all its free models of one family, and a third moved its large models behind a subscription. Our working rule is that any free model older than about six months is probably dead.

Cheap models can fake tool calls. A low-cost model told us, four times out of four, that it had created a task it never created, once with an invented ID. To the gateway that was a successful, billed call. Only checking our own database caught it. When we tested again in September on our paid low-cost seats, three models got a full tool loop right five or six times out of six with no fabrications. Test tool use with a complete loop, because every model passes the first turn.

Spend tracking was wrong, then budgets were too right. For months our spend caps did nothing, because reasoning tokens had no price in the configuration: the gateway's figure was about 190 times lower than the provider's real bill ($0.0073 against $1.5363). Once that was fixed, the caps worked too well. A budget on our development key tripped and stopped calls, and a $1-a-day cap across the whole gateway once stopped every AI call on our network over notional money that was never actually spent. A budget trip looks exactly like a model failure from the outside.

Reasoning models need room. Given a small output limit, reasoning models spend it all thinking and return an empty reply. Ten of 24 council seats did this at 160 tokens, and one used 918 of 942 tokens on reasoning. It reads as an outage.

Upgrades are one-way. Our upgrade to 1.98.0 in August ran 151 database migrations. Putting the old image back would not have been a rollback, so we took a database dump first. Back up before every gateway upgrade.

The daily model check

The model lists behind a gateway never stand still. Providers add new models, change which ones are free and withdraw old ones, week after week, usually without telling anyone who calls them. A gateway configured once and left alone slowly fills with seats that point at nothing, and the fallbacks hide it.

So checking the models is part of our daily workflow, not a clean-up we do when something breaks. A scheduled job, our model steward, runs every day. It calls every model on the gateway, and every fallback model behind it, to confirm that each one still exists and still answers. A model that has gone, or no longer replies, is replaced with one that does a similar job: a reasoning model with another reasoning model, a coding model with another coding model. Like-for-like swaps that do not raise cost are applied and read back; anything else waits for a person.

If you run a gateway with more than a handful of models, plan for a rolling check like this from the start. Without it, you find out a model has gone only when a fallback fires, if you notice at all.

What it costs

LiteLLM itself is free and open source, and it runs on our own hardware, so the gateway adds no bill of its own. The cost is attention: a daily health check, a monthly look at which free models have gone, and a rule that every config change goes through git. We cover what the models themselves cost in our market note.

The verdict

7 / 10. For a small company with several projects calling several model providers, a self-hosted gateway is worth it: scoped keys, job-named seats and one place to change a model pay for themselves the first time a free model disappears. It loses points for how quietly it fails. Fallbacks and budgets both turn a fault into something that looks like normal behaviour. If you run one, check which model actually answered, and treat any fallback as a fault until you know why it fired.