Ollama on one homelab GPU: three weeks as my local model server

SampleThis review is sample content written to show the layout. Benchmark figures in the table are illustrative, not measurements.

Why I tried it

I wanted a model server in the house that my editor, a small document-search tool and one long-running agent could all talk to, without sending code or client documents to a hosted API. The box already existed: a tower in the cupboard with a single 24 GB card that had been running a media server and not much else.

Ollama is the default answer people give for this, so I gave it three weeks of normal use before trying the alternatives. The question was not "does it run" (it does, in about five minutes) but whether it stays out of the way once other things depend on it.

What I tested

• Daily coding help from the editor, pointed at the OpenAI-compatible endpoint.
• Document Q&A over a folder of PDFs, through a small retrieval script.
• One agent running scheduled jobs overnight: summarising logs and opening tickets.
• Concurrency: what happens when all three hit the GPU at once, which is the moment a home server turns into a shared server.

Models were a mid-sized coding model and a general instruct model, both quantised to fit the card with room for context. I kept the same models across all three servers in the comparison so the differences come from the server, not the weights.

Setup

Ollama runs in Docker with the GPU passed through. Two environment variables did most of the work: one to keep models loaded between requests, and one to allow parallel requests per model.

services:
  ollama:
    image: ollama/ollama
    ports: ["127.0.0.1:11434:11434"]
    volumes: ["ollama:/root/.ollama"]
    environment:
      OLLAMA_KEEP_ALIVE: "24h"
      OLLAMA_NUM_PARALLEL: "4"
    deploy:
      resources:
        reservations:
          devices: [{ driver: nvidia, count: 1, capabilities: [gpu] }]
volumes:
  ollama: {}

The port is bound to localhost and published through the reverse proxy with authentication in front. Ollama has no user accounts of its own, so anything that can reach the port can use, pull and delete models. Treat it like a database, not a website.

Compared with llama.cpp server and vLLM

Same machine, same models where the formats allow, same three workloads. Hover or tab into the table to follow a row and a column at once.

The single-user numbers are close enough not to matter. The difference shows up as soon as more than two things ask at once: vLLM batches requests on the GPU and keeps its throughput climbing, while Ollama and llama.cpp mostly take turns.

Where it falls short

• No accounts or keys. Authentication is your reverse proxy's job. Fine at home, a real gap the moment a second team uses it.
• Concurrency flattens out. Parallel requests help, but under load the overnight agent made my editor feel slow the next morning.
• Quiet memory decisions. When two models do not fit, Ollama unloads one without telling the caller. The request just takes longer.
• Defaults hide context length. Check the context window it actually loaded rather than trusting the model card.

Verdict

Ollama is the easiest way I know to put a model behind an API on your own hardware, and for one or two people that ease is the whole point. I still run it on my laptop and recommend it to anyone starting out.

For the shared box I moved the serving to vLLM and kept one model loaded. It took an evening, it needed more care, and it handled the overnight agent and the editor together without the morning slowdown.

More reviews · What the hosted alternatives cost this month