Why I ran this test
I wanted one machine at home to answer requests from my editor, a few scripts and a chat page that three friends use. All three servers speak the same OpenAI-style API, so the client side did not change; only the server did.
Each server ran for a working week as the only back end. I kept notes on what broke, what I had to look up, and how the card behaved when more than one person was typing.
The test box
- One 24 GB graphics card, 64 GB of RAM, a SATA SSD for model files.
- Debian stable, Docker for anything that shipped an image.
- An 8B instruction model as the daily driver, and a 32B model, quantised, for the hard questions.
Ollama
Ollama is the shortest path from nothing to an answer. One install, one ollama run, and the model downloads and loads. It swaps models on request and frees the card after an idle timeout, which suits a machine that does other work.
The limit shows with several users: requests share the card and everyone slows down together. For one person that never mattered.
llama.cpp server
Ollama is built on llama.cpp, and running llama-server directly gives you every flag Ollama hides: context size, batch sizes, parallel slots, which layers go to the GPU. It is the smallest and most predictable of the three, and the one I trust to run for months untouched.
The cost is attention. You pick the GGUF file, you set the slots ahead of time, and switching models means restarting the process.
vLLM
vLLM is a serving engine first. Continuous batching and paged attention keep many requests moving at once, which is exactly what the shared chat needed. It also reserves most of the card at start-up by design, so nothing else on the box gets the GPU.
Install is the hard part: the Python package, the CUDA version and the driver have to agree. The Docker image solves most of that.
Side by side
The same box, the same 8B model in each server’s preferred format, the same four-user script.
| Ollama | llama.cpp server | vLLM | |
|---|---|---|---|
| Getting it running | |||
| Licence | MIT | MIT | Apache 2.0 |
| Install | One installer or a Docker image | Build from source or a release binary | Python package or a Docker image; needs a matching CUDA stack |
| First model answering | One command: ollama run | Download a GGUF file, then start llama-server with flags | Start the server with a Hugging Face model id |
| Model format | GGUF, pulled from its own library or imported | GGUF | Hugging Face weights (safetensors), plus quantised formats such as AWQ and GPTQ |
| Running it day to day | |||
| Switching models | Loads on request, unloads after an idle timeout | One model per process; restart to switch | One model per server; run a second server for a second model |
| OpenAI-compatible API | Yes | Yes | Yes |
| Four users at once | Works; requests share the card and slow down together | Works with --parallel slots set ahead of time | Built for it: continuous batching keeps all four moving |
| Sample throughput, 4 users, 8B model | ~38 tokens/s each | ~41 tokens/s each | ~70 tokens/s each |
| Memory when idle | Frees the card after the timeout | Holds the model until stopped | Reserves most of the card up front by design |
| Upgrades | Installer or image; painless in my runs | Rebuild or new binary; flags change now and then | Pin versions; CUDA and driver updates need care |
| Verdict | |||
| Best for | One person and quick trials | People who want every flag and the smallest footprint | Teams sharing one GPU |
Throughput rows are sample values for this layout, not measurements. Licences checked against each project’s repository.
What I would pick
For a laptop or a desktop that one person uses: Ollama, and do not think about it again. For a box that sits in a cupboard and serves one model to a few scripts: llama.cpp server with fixed flags. For anything that several people use at the same time: vLLM in Docker, with the versions pinned.
What I did not test
Multi-GPU setups, CPU-only serving, and image or audio models. Each deserves its own post.