Measuring LLM Prefix Caching: The Cache Hit Rate Metric

Prefix caching is one of the biggest cost levers in LLM serving. vLLM, SGLang, TGI, and most hosted providers all do some version of it: during prefill they compute a key-value (KV) cache, and if a later request shows up with the same prompt prefix, they reuse that cache instead of recomputing it. Done well, a lot of expensive prefill compute turns into a cheap cache lookup.

Whether it helps depends on how much of your traffic re-sends the same prefix, and most benchmarking runs don't tell you. This is part of my LLM benchmarking guide. Here I want to focus on how to actually measure cache effectiveness: the metric, why it matters for agentic workloads, and the cost angle.

What prefix caching reuses

During prefill, the server computes the KV cache for the input prompt. If a later request sends a prefix the server has already seen, it can skip recomputing that part and just serve it from cache. The classic example is a multi-turn conversation: each turn re-sends the entire prior history, and ideally everything except the newest user message comes back from cache.

This is also why cache reuse only shows up in multi-turn or shared-prefix workloads. A single isolated request has nothing to reuse. Turn 0 is always a cold start. So if you want to measure caching, you have to re-send history, which means multi-turn requests.

Why this matters most for agentic workloads

A chat conversation re-sends some history every turn, but an agentic coding loop does this on fast-forward, and at a scale where caching stops being optional and starts dominating both latency and cost.

Here's how a coding agent actually runs (Claude Code, Cursor, Cline, that sort of thing). It loops: read the task, decide on an action, call a tool to read a file or run a command, get the result back, decide the next action, call another tool. Each one of those iterations is a new API request, and every request re-sends the entire accumulated context. The system prompt, the original task, all the prior reasoning, every previous tool call and its result. The only genuinely new content is the latest tool result and the model's next decision. Everything before that is a prefix the server has already computed.

• 
  Latency. Without caching, each step's time-to-first-token includes recomputing prefill for the whole context. As the conversation grows, every step gets a little slower, so the agent loop itself drags as the session wears on. With good caching, only the new tool result triggers prefill, and TTFT stays roughly flat across the session instead of climbing.


• 
  Cost. Prefill compute scales with context length, so re-paying for the full context on every tool call gets expensive fast. Agentic sessions are notoriously long, often tens of thousands of tokens and well past 100k. With caching you pay for each token's prefill once instead of on every subsequent step.

So an agentic session is really just a long multi-turn conversation where history gets re-sent every turn, which is exactly the case cache_hit_rate was built for. If you're picking a serving setup for agentic workloads, cache hit rate under a realistic multi-turn load is one of the most telling numbers you can collect.

One nuance worth knowing: some providers also offer explicit prompt caching, where the client marks cache breakpoints (Anthropic's cache_control is the example). That's a different mechanism from the automatic prefix caching most OpenAI-compatible endpoints do, and llmperf-rs only measures the automatic kind. For a standard tool-call loop against a vLLM-style endpoint, automatic prefix caching is what applies.

The cache hit rate metric

The metric I use measures cache reuse against the content that was previously sent, not the whole request. Caching only reuses what the server has already seen: the assistant's prior outputs and earlier user prompts that get echoed back in the next request. New tokens in the current turn can never be cached, because the server hasn't seen them before.

Implemented in llmperf-rs:

cache_hit_rate = sum(cached_tokens) / sum(total_tokens_of_non_final_turns)

• 
  Numerator: the sum of cached_tokens reported by the endpoint on each turn, read from prompt_tokens_details.cached_tokens in the streamed usage object. None means the endpoint didn't report the field.


• 
  Denominator: the sum of each turn's total tokens (input + output) for every turn except the last turn. The last turn's content is never re-sent in a later request, so it can never be served from cache and is excluded.

100% means every previously-sent token came back from cache. In a perfect cache, cached_tokens equals the prior-turn total on every warm turn.

Edge cases

A few that bite in practice:

• 
  Single-turn runs report None. With only one turn there's no history to re-send, so the denominator is zero. Cache hit rate is a multi-turn concept.


• 
  An all-None run reports None. If the endpoint never reports cached_tokens, you get None, not zero. That's deliberate: None means "not measurable", which is different from 0.0 (a cache that's just never hit).


• 
  A mismatched endpoint dilutes instead of nulling. If some turns report cached_tokens and others don't, the unobserved turns are left out of the numerator but their re-sent history still counts in the denominator. So a noisy endpoint just pulls the ratio down rather than wiping it out.


• 
  Cold start contributes nothing. Turn 0 reports cached_tokens = 0 or just omits it, so it doesn't move the numerator either way.

How to run it

You need multi-turn requests, which in llmperf-rs is --multi-turn N:

export OPENAI_API_BASE=http://localhost:8000/v1   # vLLM with prefix caching enabled

llmperf --model Qwen/Qwen3-4B-Instruct-2507 \

        --multi-turn 5 \

        --max-num-completed-requests 10

The summary then includes a cache_hit_rate field (alongside the TTFT/ITL/throughput metrics covered in the main guide):

Example value only, illustrative and not from a real run.

"cache_hit_rate": 0.91

That single summary number aggregates across the whole run. Per-turn cached_tokens and turn_index are also written to the individual-responses file if you want to see how the cache builds up over turns after the cold start.

A note on reasoning content

There's a subtlety if you're benchmarking reasoning models. The common guidance is to discard a model's reasoning_content from the message history you send back, to save tokens. llmperf-rs does the opposite for multi-turn runs: it echoes the previous turn's reasoning_content on the assistant message.

The reason is exactly this topic. Providers that support prefix caching over reasoning (Z.ai's "Preserved thinking" with clear_thinking: false, for example) can reuse the KV cache across turns only if the reasoning is re-sent. Dropping it to save on echoed-input tokens throws away the cache reuse, which usually costs more than it saves. Providers that don't understand reasoning_content just ignore the field, so it's safe to send.

So if you're measuring cache hit rate on a reasoning model, make sure you're re-sending the reasoning. Otherwise you're measuring a workload that disables its own cache.

Caveats

• 
  Token-count accuracy matters less here than elsewhere, because the ratio is between two token sums rather than a token count against a wall-clock time. Chat-template token variance affects numerator and denominator similarly, so it mostly cancels out.


• 
  Cache behavior depends on server config, not just the model. vLLM's prefix caching can be on or off; KV-cache size and eviction policy affect whether a warm turn actually hits cache. A low cache hit rate under load can point at preemption rather than a broken cache.


• 
  This measures endpoint-level prefix caching, not GPU-level cache statistics. For kernel-level breakdowns you'd want a tool like aiperf or trtllm-bench. See my notes on llmperf alternatives.

Wrapping up

If you've enabled prefix caching, cache hit rate is how you confirm it's earning its keep. The key thing to get right is the denominator: measure cache reuse against the history you re-sent, not against the whole request, or you'll understate a cache that's working fine. And remember it's strictly a multi-turn metric. A single-turn benchmark tells you nothing about caching.

The full version with the exact math and more detail is on my blog.