Lance Keay

Posts / Review

Serving a local model to a coding agent: Ollama, vLLM and llama.cpp compared

Lance Keay 11 min read reviewshomelab

Sample post: the structure is real; the measurements are illustrative placeholders, not published results.

I wanted a coding agent that keeps working when the internet doesn't, and that doesn't send a client's code anywhere. That means an open-weights model on my own GPU, behind an endpoint the agent already knows how to talk to. There are three obvious ways to do it. I ran all three for a week each.

What I tested

  • Hardware: one desktop, a single 24 GB consumer GPU, 64 GB RAM, Linux.
  • Model: the same ~30B-parameter open-weights coding model on each server, 4-bit quantised where the server needed it.
  • Client: a terminal coding agent pointed at each server's OpenAI-compatible endpoint, with tool calling switched on.
  • Work: real tickets on a TypeScript API and a Go CLI: bug fixes, small features, test writing. Not synthetic benchmarks.
  • Not tested: multi-GPU setups, Apple silicon, or Windows hosts.

What a coding agent actually needs

Chat front-ends forgive a lot. Agents don't. The two things that decided this review were whether tool calls came back as valid JSON every time, and how long the agent sat waiting before the first token of each step. Raw tokens per second matters less than you'd think: an agent spends most of its time reading files and running tests, not generating.

The third thing is concurrency. The moment you run a second agent, or a teammate points theirs at your box, a server that queues requests one by one starts to feel slow in a way no single-user benchmark shows.

Ollama

Install, pull, run. Ollama is the easiest of the three by a distance, and its model library means you rarely touch a config file. It was the only one I had working inside ten minutes.

The cost of that ease is control. Context length, batch size and how the GPU memory is split are all decided for you unless you go looking, and the defaults favour short chats. Once I raised the context window to what an agent needs, speed dropped noticeably and I had to restart the server to change it.

vLLM

vLLM is built for serving many requests at once, and it shows. With two agents running in parallel it was the only server where neither one felt slower. Its tool-call parsing was also the most reliable in my runs, once I'd set the right parser for the model.

It is also the fussiest. It wants to claim most of the GPU's memory up front, it is happiest with specific quantisation formats, and the first afternoon went on matching flags to the model. If the box is also your desktop, expect to close things before starting it.

vllm serve ./models/coder-30b-awq \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice --tool-call-parser hermes

llama.cpp server

The llama.cpp server sits in between: one binary, GGUF models, and every knob exposed. It has the best behaviour when memory is tight, because you can offload some layers to the CPU and keep going. It was the one I trusted on a machine I also use for other things.

Its weak spot in my week was tool calling: correct most of the time, but occasionally it returned a call the agent couldn't parse, and the agent had to retry. That is survivable, but it adds up over a long session.

Side by side

Same model, same GPU, same agent. Throughput is generation speed for one agent with a 32k context. Sample figures for layout only.

Server Licence Setup Throughput (tok/s) First token Tool calls parsed Two agents at once Best for
Ollama MIT Minutes
38
 0.9 s 96% Queues; second agent waits Trying local models, single user
vLLM Pick Apache-2.0 An afternoon
52
 0.4 s 99% Both stay quick A dedicated box serving a team
llama.cpp server MIT An hour
44
 0.6 s 91% Works with parallel slots Tight memory, shared desktop

Swipe the table sideways to see every column →

What I'm running now

vLLM, on a machine that does nothing else. For my own work that was the right trade: an afternoon of setup for an agent that never makes me wait and never chokes on its own tool calls.

If you only want to see whether a local model is good enough for your work, start with Ollama. You'll have an answer the same evening, and if the answer is yes, moving to vLLM later is a config change in your agent, not a rewrite.

The model matters more than the server. If your agent struggles locally, try a different model before you try a different server.
← All posts Questions or corrections? Email me