← Field notes

How parallel agent requests can slow down an entire Ollama session

Two coding-agent sessions sharing one Ollama slot waited a median 46 seconds for a response to start, against one second with a slot each. Over half the prompt tokens the server processed were repeats. Why hybrid models lose their place, and which settings change it.

  • L4 · Ollama and llama.cpp · Qwen3.8-27B
  • Ollama
  • llama.cpp
  • prompt cache
  • coding agents

Ask a coding agent to fix a failing test. It might read a file, run the test, edit a function, then run the test again. Each time a tool returns a result, the agent sends another request to the model. From your terminal it looks like one conversation. From the server’s side it’s a series of increasingly long prompts, most of which it has already seen.

Normally, that repetition is cheap. The server keeps the work it did on the previous turn and picks up where the new text begins. Now open a second terminal and give the same agent a task in another project. Both sessions send the same tool descriptions and much of the same setup text, but each has its own instructions and history. What happens to the first session’s work while the server answers the second?

We tried this at Tensward with two unmodified Qwen Code sessions running against Ollama. When both sessions took turns on one slot (the place where the server holds an active request’s model state), the median request waited about 46 seconds for the response to start. When we gave each session a slot of its own, it waited about one second.

Two sessions sharing a slot have to wait for each other. We expected some queueing. What surprised us in the logs was how much work the server did while they waited. With a shared slot, 54% of the prompt tokens it processed were repeats. With a slot each, that fell to 0.4%.

Why couldn’t Ollama reuse the first 8,921 tokens?

Imagine pausing the server just after it reads token 8,921 of session A. Up to this point, sessions A and B are identical. The model is in exactly the state B needs. But the server keeps reading A, and B arrives only after A’s prompt is finished. Can the server get back to that moment?

In a model that uses full causal attention throughout, the server keeps a pair of vectors for each token in each attention layer: a key and a value. Together these entries form the KV cache. Reading token 8,922 adds another entry. The entries for tokens 1 through 8,921 stay as they were, because earlier tokens cannot attend to later ones. As long as those entries are still in memory, the server can keep that prefix, discard A’s suffix, and start processing B where the prompts diverge.

Qwen3.8-27B is a hybrid. Its full-attention layers have that same KV cache. Its recurrent layers also carry a running state that each token updates in place. When a layer processes token 8,922, its new state replaces the one it had at 8,921. Trimming the attention layers’ KV cache gets those layers back to the right point. The recurrent layers still hold the result of reading the rest of A.

That’s why the server takes a checkpoint, also called a snapshot here. Both words mean a saved copy of the state needed to resume at a particular token position. If the server had saved one at 8,921, B could start with its first new token. A checkpoint at 8,000 would also help: the server could restore it and process the remaining 921 shared tokens to rebuild the state it needed. The server saves checkpoints at selected positions; each one took about 150 MB in this test.

Two sessions of one agent: their prompts are identical for 8,921 tokens, and the first snapshot the server holds is at token 10,151, so none of the shared part can be reused.
The sessions share the shaded prefix. The dot marks the earliest saved state, already past the point where the prompts diverge.

Now follow that one slot through three requests. A goes first. When A finishes, the slot holds A’s state. B goes next. A’s earliest checkpoint, at token 10,151, already includes 1,230 tokens specific to A. B has to start from token one. As the server processes B, it replaces A’s state in the slot with B’s. Then A comes back with a tool result. A needs exactly the state it left behind one request ago. The slot now holds B’s state, and the server never saved a copy of A’s.

Why the slowdown lasted across turns

llama.cpp, the engine inside the Ollama version we tested, can save the outgoing session’s state in host RAM before handing its slot to another session. That would let A pick up where it left off when its tool result arrives. A whole conversation state was about 1 GB in our test. The engine checks how much of the old prompt the incoming request shares before deciding whether to make that copy.

Our requests shared 60–66% of their tokens. That cleared the engine’s 50% threshold for reusing the slot directly, even though the matching text contained no usable checkpoint. On the switch from A to B, that decision cost A its saved place. On the switch back, it cost B its place. As long as the sessions kept taking turns this way, each one displaced the state the other would need next. Both agents kept their conversation histories and could finish their work. The server kept paying to rebuild those histories’ model state before it could answer the next request.

Measuring the cost with the queue still there

Giving each agent a slot changes both cache reuse and queueing. To see what the cache decision alone could change, we kept both agents on one slot in standalone llama.cpp. That build exposes a flag, -sps 0, that sends each request through the RAM cache. We ran the same coding task twice per setting.

Two sessions sharing one slot Default With -sps 0
Total prompt tokens processed about 128,800 (128,776 and 128,865) 43,300 and 43,700
Median wait for the first byte 48.5 s and 47.1 s 17.4 s and 16.8 s
Peak host memory 2.4–2.9 GiB 7.2 GiB

The second agent still needed its first prompt processed in full. After that, each agent could resume from its own saved conversation. Saving and loading state cost about 0.8 seconds per request. From the timestamps, we estimate that roughly 14 of the remaining 17 seconds were spent waiting for the other session to finish its response.

On Ollama 0.40.16, we could give each session a slot, but the runner didn’t expose the similarity flag. We used OLLAMA_NUM_PARALLEL=2 and OLLAMA_CONTEXT_LENGTH=16384 on our 24 GB L4. That split the one-slot run’s 32,768-token budget between the sessions. The largest prompt was 14,956 tokens, so it fit. The 46-to-1-second result includes both the change in cache behavior and the ability to serve requests concurrently.

What happens when eight requests start together?

Giving the two coding agents separate slots solved their switching problem. A different experiment showed how slots can also miss a useful checkpoint while processing their own prompts. This time we started eight short conversations together. They all used the same system prompt, so the end of that prompt was a natural place for each slot to save state for future requests.

Eight slots during one burst: three keep the snapshot at the end of the shared part and five miss it; the five re-read about three times as many tokens on later requests
Each row is one slot. Each dot on the right is a later request in this eight-slot Ollama run.

Read across a row. On the left is where that slot saved state during its first request. On the right is how many tokens it had to process again on later requests. The three slots that saved state at the end of the shared prompt averaged 125 tokens. The other five averaged 370.

In this setup, the server processed up to 512 prompt tokens per batch, split among active slots. Some slots ran out of room in the batch just before the end of the shared prompt and saved an earlier checkpoint. When the next batch crossed the desired boundary, the default spacing rule skipped the new checkpoint because the previous one was fewer than 8,192 tokens back.

Allowing closer checkpoints with --checkpoint-min-step 0 let all eight slots save at the boundary. Across four paired Ollama runs, throughput rose 10.9% and median time to first token fell 17.5%. These were short tool-calling conversations. We did not measure this setting’s cost over a long agent session.

Which setting changes which behavior?

We could see both decisions in a smaller test with two assistants that shared tool definitions but had different instructions. After a switch, the server needed to recover the returning assistant’s conversation. On subsequent requests, it also needed a checkpoint at the end of that assistant’s system prompt. On llama.cpp, changing the spacing alone reduced median prompt processing on follow-ups from 506 tokens to 126.5, while switches still took 426.5 tokens. Changing both settings brought switches down to 124.5 tokens, roughly the amount of new input.

For the coding-agent case, the first things we’d check are how many sessions share each slot and how many prompt tokens the server actually processes after a switch. Those numbers let you tell whether a user waiting on the next tool call is also paying to process the same instructions again.

Test conditions and prior work

We used Qwen Code 0.27.4, Qwen3.8-27B Q4_K_M on one NVIDIA L4, Ollama 0.40.16 and standalone llama.cpp b12441. The coding-agent comparison covered one small fix-the-tests task, with one run per Ollama configuration and two per llama.cpp configuration. First byte measures when an individual response started, including time in the queue. One coding session alone, including our test with two sub-agents, showed little repeated processing. In one llama.cpp run with the shortcut disabled, a background helper reached our eight-minute limit after the coding work and tests were done.

A llama.cpp fork had already changed the similarity decision to consider the newest usable checkpoint. An Ollama discussion describes hybrid-model clients displacing each other’s cached state. We encountered this behavior while investigating the cost of saving model state and measured what it meant for two real coding sessions.

Find yours on your own machine.

The open-source CLI measures how your model is served and offers one change at a time.