Ollama, llama.cpp, vLLM and LM Studio, each run for real work on the same homelab box with one 24 GB graphics card. Which one to install depends on how many people will use it.
Sample review: the speeds, power figures and scores below are placeholders that show the layout, not measurements.
Verdict
Ollama for a shared team box, llama.cpp when you want every token a second, vLLM once several people hit it at the same time.
Cost
All four free to use; power at about 0.31 kWh per working day
Licence
MIT (Ollama, llama.cpp), Apache-2.0 (vLLM), proprietary freeware (LM Studio)
What was tested
Code search, commit messages, document Q&A and a small tool-calling agent, over three weeks
Test box
One 24 GB GPU, 64 GB RAM, Debian 12, models at 7B to 14B in 4-bit and 8-bit
The setup
The question I wanted answered was narrow: for a small team that wants private drafts, code search and an agent or two without sending data to a hosted API, which runner should go on the one GPU box in the cupboard?
All four ran on the same machine, one at a time, with the same models where the formats allowed it. Each served an OpenAI-compatible endpoint, so the same client scripts drove every test.
# the client never changed, only the base URL
export OPENAI_BASE_URL=http://gpu-box.lan:8080/v1
python bench.py --suite code-search --runs 20
What was tested
Code search: questions about a 140,000-line TypeScript repository, answered with retrieved files in the prompt.
Commit messages: a diff in, one message out, checked by hand against what I would have written.
Document Q&A: runbooks and vendor PDFs, with answers checked against the source page.
A small agent: four tools (shell, HTTP, file read, search) and a loop of up to twelve calls.
Each tool also had to survive the dull parts: a reboot, a model update, two people using it at once, and a week without anyone looking at it.
The comparison table
One row per runner. Speeds are for an 8B model in 4-bit on one request, then the total across eight requests at once. Scroll the table sideways on a small screen; the first column stays put.
Runner
Licence
Install
API
Model formats
Tokens/s, 1 user
Tokens/s, 8 users
First token
Multi-user
Verdict
Ollama
MIT
One script or package
Own API and OpenAI-compatible
GGUF
92
210
180 ms
Queues requests
The default for a team box
llama.cpp
MIT
Build or release binary
OpenAI-compatible (llama-server)
GGUF
104
240
150 ms
Parallel slots
Fastest single user; most flags to learn
vLLM
Apache-2.0
Python package or container
OpenAI-compatible
Hugging Face weights, several quantised formats
88
540
240 ms
Batches well
Best once people share it
LM Studio
Proprietary freeware
Desktop installer
OpenAI-compatible local server
GGUF, MLX on Apple silicon
90
120
190 ms
Single desk
Good on a laptop, wrong for a server
Figures for layout only, not measurements. Higher tokens per second is better; a lower first-token time is better.
Where each one fits
Ollama: the one to install first
It took the least of my time. Pulling a model is one command, updates did not break the client scripts, and it unloads idle models so the card is free for other work. It queues rather than batches, so the eighth person waits.
llama.cpp: when speed on one request matters
It is the engine under several of the others, and running it directly gives you every switch: context size, offload layers, parallel slots. That control is the cost too; the right flags took an afternoon.
vLLM: when the box is shared
On one request it was no faster than the others. With eight at once it pulled well ahead, because it batches requests on the card. It wants more memory headroom and more care at setup.
LM Studio: a desk tool
The nicest way to try models on a laptop. The licence and the desktop-first design make it the wrong choice for a server others depend on.
What I would change
Three weeks is enough to find setup pain and daily annoyances, not enough to see how each project handles a year of updates. I will rerun the table when the next major version of each lands, and note here what changed.
Need someone to set up private models or an agent pipeline for your team?
