Lance Keay

PostsReviewsModel gateways

LiteLLM as a home-lab model gateway

Three weeks with one endpoint in front of four model providers: what it fixed, what it cost me, and where it got in the way.

Lance Keay27 Oct 202614 min readTags: review, homelab
Verdict
KeepWorth running as soon as you call more than one provider. If you only ever call one, the provider's own SDK is less to maintain.
Cost
Free to self-hostPaid enterprise tier for SSO and some admin features. My cost was a small VM and a Postgres database.
Licence
MIT coreEnterprise features sit in a separately licensed directory. Check which ones you rely on.
What I tested
Proxy in Docker4 vCPU / 16 GB home server, Postgres for keys and spend, four providers, three internal apps, 21 days.

Sample review. Written to show how a review page reads on this site. The structure is real; treat the findings as illustrative until Lance publishes his own testing notes.

Why I wanted a gateway

By the end of the summer I had three small internal tools talking to four model providers, each with its own SDK, its own key in its own .env file, and its own idea of what a rate-limit error looks like. Switching one tool to a cheaper model meant a code change and a redeploy. Finding out what last month cost meant logging into four dashboards.

What I wanted was boring: one OpenAI-compatible endpoint inside the network, one key per app, a spend figure per key, and a way to fail over when a provider has a bad afternoon.

The setup

LiteLLM's proxy runs as a single container with a YAML file describing the models it can route to. Postgres is optional, but you need it for virtual keys and spend tracking, so I ran it from day one.

model_list:
  - model_name: fast
    litellm_params:
      model: provider-a/small-model
      api_key: os.environ/PROVIDER_A_KEY
  - model_name: fast
    litellm_params:
      model: provider-b/small-model
      api_key: os.environ/PROVIDER_B_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2

Two entries with the same model_name become a pool. The apps ask for fast and stop caring which provider answers. That alone was worth the afternoon it took.

What worked

  • One endpoint, one client. All three apps now use the OpenAI client pointed at the proxy. Moving a tool to another model is a YAML change and a restart.
  • Per-app keys with budgets. Each app gets a virtual key with a monthly cap. When the summariser went into a retry loop, it hit its cap instead of my card.
  • Fallbacks I didn't have to write. When one provider returned errors for an evening, requests went to the other pool member and nobody noticed.

What got in the way

The configuration surface is large, and the docs move quickly. Twice I followed an example that used a setting name from an older release. Pin the image tag, read the changelog before upgrading, and keep the config in git.

The admin UI is useful for checking keys and spend, but I would not hand it to someone non-technical. Some of the features I reached for first, such as SSO, are in the enterprise tier.

The gateway becomes the single point every model call goes through. Treat it like a database: monitored, backed up, upgraded on purpose.

Comparison

LiteLLM next to the other ways of doing this

Feature columns reflect each project's documentation as read for this sample; check before relying on them.

Option Runs where Licence OpenAI-style API Fallbacks & retries Per-key budgets Spend logs Effort to run Best for
LiteLLM proxyReviewed here Your server MIT core + enterprise Yes Yes Yes Yes, with Postgres Medium: container, DB, config Several apps, several providers
Portkey AI GatewayOpen-source gateway Your server or theirs MIT (gateway) Yes Yes Hosted platform Hosted platform Low to run, more for full features Routing rules without a database
OpenRouterHosted router Their cloud Proprietary service Yes Yes Credit limits Yes None Trying many models quickly
Provider SDKs directlyWhat I had before In each app Varies Per provider You write them Provider dashboards Per provider Low at first, grows per app One app, one provider
Reverse proxy + scriptsNginx or Caddy Your server Open source Passes through Upstream retries only No Access logs only Low, but all glue is yours Hiding keys from clients

Verdict

Keep. It removed more code than it added, and the per-key budgets paid for the setup time the first time something looped. The cost is another stateful service to look after.

If you are deciding whether to bring someone in to set this up for a team, the gateway is the easy part. The work is agreeing which apps get which models, at what budget, and who gets paged when it falls over. That is the kind of work I take on.