vLLM on a single 24 GB card: worth the setup?
I moved the house LLM server from Ollama to vLLM for a month, with llama.cpp server running alongside as a control. Here is what changed, and what it cost me in evenings.
Fastest of the three once it runs. Overkill unless more than one person uses the box.
- Cost
- Free; runs on hardware you own
- Licence
- Apache 2.0
- Setup time
- An evening, plus a driver fight
- Compared with
- llama.cpp server, Ollama
- Use it if
- Several users or agents share one GPU
- Skip it if
- It is just you, or you swap models daily
Why I tried it
The box under the stairs serves a chat front end for the household, a couple of coding agents, and a nightly job that summarises my RSS backlog. With Ollama, everything was fine until two of those overlapped. Then the agents queued behind each other and the chat window sat there thinking.
vLLM's pitch is batching: it serves many requests at once from one copy of the model, and keeps the GPU busy while it does. That is exactly the problem I had, so it seemed worth an honest month.
What I tested
- The same model, Llama 3.1 8B Instruct, in each server's preferred format: AWQ for vLLM, Q4_K_M GGUF for the other two.
- One user at a time, then eight concurrent clients sending a fixed set of 400 prompts.
- Tokens generated per second, time to first token, and memory held when idle.
- The dull things: install, upgrades, restarts after a power cut, swapping models.
Every run was repeated three times and I report the middle one. These are measurements from one machine, not a benchmark suite. Your card, driver and model will give you different numbers; the shape should hold.
Tokens generated per second, higher is better
Same model and prompts on one RTX 3090. Both panels share a scale, so bar lengths compare across them. Sample figures for this design.
One user
Eight users at once, combined
For one person, the difference is small enough that you would not notice it in a chat window. With eight clients it is the whole story: vLLM keeps each user at roughly the speed of a single-user session, while the others share out a much smaller total.
| Server | Licence | Model format | 1 user, tok/s | 8 users, tok/s | First token, 8 users | VRAM when idle | Swapping models | Best for |
|---|---|---|---|---|---|---|---|---|
| vLLM | Apache 2.0 | AWQ / safetensors | 128 | 820 | 0.4 s | 21.6 GB | Restart the server | Several users or agents on one card |
| llama.cpp server | MIT | GGUF | 112 | 290 | 1.9 s | 6.1 GB | Restart, or run several | Tight memory, mixed CPU and GPU |
| Ollama | MIT | GGUF | 104 | 210 | 3.2 s | 0 GB (unloads) | On request, automatic | One person trying many models |
Scroll the table sideways for all columns.
What it cost
The software is free. The cost is memory and time. vLLM reserves most of the card up front for its cache, which is why the idle figure looks alarming: nothing else gets to use that GPU while it runs. I gave up running a small image model alongside it.
Setup took an evening, most of it matching CUDA, driver and Python wheel versions inside the container. Upgrades since then have been one image tag change each.
What annoyed me
- Changing models means restarting the server. Ollama spoiled me here.
- Startup takes a minute or two while it profiles memory and compiles. Fine for a server, irritating while experimenting.
- Error messages assume you already know what a KV cache block is.
Verdict
Fastest of the three once it runs. Overkill unless more than one person uses the box.
I have kept vLLM on the main card, because this house does have more than one user and two agents. If it were just me, I would go back to Ollama tomorrow and not miss anything.
This post is sample content written for the site design. The setup and reasoning are realistic; the measured figures are illustrative, not real benchmark results. Licences are as published by each project.