Ollama, three months in: local models without the yak-shaving
I ran Ollama as the model server for my homelab from August to October, then put it beside llama.cpp's server, vLLM and LM Studio to see where each one fits. Short version: it is the easiest way to run local models on one machine, and the wrong tool once several people share a GPU.
Sample post. This review shows the article layout for lancekeay.com. The licences and features described are accurate as far as I know at the time of writing, but they change quickly. Check each project's own docs before you rely on them.
What I tested
I wanted to find out whether Ollama could be the one model server behind everything else in the homelab: a chat front-end, a coding assistant in the editor, and a couple of small scripts that summarise logs overnight. I didn't benchmark it in a lab. I used it for real work and kept notes whenever something got in the way.
- Machines: a desktop with a single 24 GB consumer GPU, and a fanless mini PC with no discrete GPU, running small models on the CPU.
- Models: a mix of 7–14B instruction-tuned models at 4-bit quantisation for daily use, and a 30B-class model when it would fit.
- Clients: Open WebUI, an editor extension pointed at the OpenAI-compatible endpoint, and plain
curlfrom cron jobs. - Deployment: the official install script first, then the Docker image with the GPU passed through, which is what I'd recommend.
What works
Getting started really is quick. One command installs a service, ollama pull fetches a model, and a REST API is listening on port 11434. Because it speaks the OpenAI-style chat completions format as well as its own API, nearly every tool I tried connected without a plugin.
docker run -d --gpus=all -v ollama:/root/.ollama \
-p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull <model>
curl http://localhost:11434/v1/chat/completions -d '{...}'Model management is the other big win. Switching models is a matter of naming a different one in the request. Ollama loads it, unloads idle models after a timeout, and keeps the rest of the machine usable. A Modelfile lets you pin a system prompt and parameters under a name of your own, which made the overnight scripts reproducible.
The best thing I can say about it is that after the first week I stopped thinking about it.
Where it falls short
The trouble starts when more than one person, or more than one busy script, hits the server at once. Ollama can handle parallel requests, but it isn't built for high-throughput batching the way vLLM is. When a teammate and I both used the chat front-end during a working session, responses queued up noticeably.
- Defaults hide things. The default context length is conservative, and long prompts get cut short without much warning. Set it explicitly per model.
- Observability is thin. I wanted a Prometheus endpoint and request-level metrics. In the end I put a reverse proxy in front and logged there.
- No auth. The API has no authentication of its own. Keep it bound to localhost or a private network, and put a proxy with auth in front of anything shared.
Compared with the alternatives
Highlighted row = this reviewSwipe the table sideways to see every column. The tool name stays pinned.
| Tool | Verdict | Licence | Cost | Setup effort | OpenAI-style API | Many users at once | Runs without a GPU | Best for |
|---|---|---|---|---|---|---|---|---|
| Ollama | Use it | MIT | Free | Low: one command | Yes | Limited | Yes, slowly | A personal or single-machine model server that many tools can share |
| llama.cpp server | Use it | MIT | Free | Medium: build or fetch binaries, manage model files yourself | Yes | Limited | Yes | Full control over flags and quantisation; tiny footprints |
| vLLM | Use with care | Apache-2.0 | Free | High: GPU drivers, Python env or container, one model per server | Yes | Yes, built for it | Not really | Serving a team or an app from a dedicated GPU box |
| LM Studio | Use with care | Proprietary | Free download | Low: desktop app | Yes | No | Yes | Trying models on a laptop with a GUI; not a server |
These are qualitative ratings from my own use, not benchmarks. "Limited" means it handles a few parallel requests but doesn't batch them efficiently. Licences are taken from each project's repository at the time of writing.
Verdict
Use it if you want one machine to serve local models to your own tools. Nothing else I tried gets from a clean install to a working, OpenAI-compatible endpoint with so little friction, and model switching is genuinely pleasant.
Plan your exit before you share it. When I was serving a small team, I'd put vLLM on a dedicated GPU box behind an authenticating proxy and keep Ollama on my own desktop for experiments. The two get along fine, because client code that targets the OpenAI-style API doesn't care which one answers.