Prompt Caching: Paying Once for a Prefix

Every request to a language model API starts with prefill: the model reads the whole prompt and computes the keys and values of every token (The KV Cache). An agent with a 20,000-token system prompt and tool list pays for that prefill on every call, though the first 20,000 tokens are the same each time. Prompt caching keeps the result of that work and charges less to reuse it. It is the KV cache, kept between requests, and the prefix sharing of SGLang and RadixAttention, turned into a price.

Table of contents

What is actually cached

The keys and values a token produces at every layer depend on that token and on every token before it, and on nothing after it: attention is causal (The Transformer’s Three Attention Patterns). So the KV cache of a prompt’s first $n$ tokens is a pure function of those $n$ tokens. Two requests that start with the same $n$ tokens can share it; change one token at position $i$, and every position from $i$ on has different keys and values, while everything before $i$ is still valid.

That gives the rule every provider’s caching follows: only an exact prefix can hit. Anthropic’s documentation puts it as “100% identical prompt segments” up to the cached point; OpenAI’s matches the longest cached prefix at its block boundaries. A cache is not a lookup of similar prompts. Any of these misses everything after it: one changed character near the start, a timestamp in the system prompt, a reordered tool list.

How the APIs expose it

As of October 2026, from the providers’ documentation:

• Anthropic caches when asked, in two ways: a single top-level cache_control field, which puts the breakpoint on the last cacheable block and moves it forward as a conversation grows; or cache_control markers on individual blocks (up to 4 per request), each caching everything before and including it, in a fixed order: tools, then the system prompt, then messages. Entries live 5 minutes by default, or 1 hour at a higher write price, and each hit refreshes the lifetime. Writing costs 1.25 times the base input price (2 times for 1 hour); reading costs 0.1 times, less on some models. Prompts below a minimum length, 512 to 4,096 tokens depending on the model, aren’t cached at all, with no error.
• OpenAI caches automatically, from 1,024 tokens on its newest models, which also accept explicit breakpoints. On those, a write costs 1.25 times the input price, a read 0.1 times, and entries live at least 30 minutes after their last use; older models charge no write premium and keep entries for minutes in memory, with extended retention of up to 24 hours.

The details differ and change, but the shape is the same everywhere: pay a little extra to store a prefix, pay a tenth or less to reuse it, and lose it if it isn’t reused soon.

The break-even, worked

# When prompt caching pays, from the published multipliers (Anthropic's
# prompt-caching docs, October 2026): a cache write costs 1.25x the base
# input price with a 5-minute lifetime, 2x with a 1-hour lifetime; a cache
# read costs 0.1x (0.05x for Claude Opus 5.5, 0.025x for Claude Fable 5.1).
# A prefix of P tokens sent N times within the lifetime:
#   uncached: N * P;   cached: write * P + (N - 1) * read * P.
def cost(n, write, read=0.1):
    return write + (n - 1) * read                    # in units of P tokens at the base price

for name, write in (("5-minute", 1.25), ("1-hour", 2.0)):
    n = next(n for n in range(1, 100) if cost(n, write) < n)
    print(f"{name} cache (write {write}x, read 0.1x): cheaper from use {n} on; "
          f"at 10 uses it costs {cost(10, write) / 10:.0%} of uncached, at 100 uses {cost(100, write) / 100:.0%}")

# A 20,000-token system prompt for Claude Opus 5.5 (listed: $4 per million
# input tokens, $5 for 5-minute cache writes, $0.20 for cache reads), 1,000
# requests, each within 5 minutes of the previous one, so the cache stays
# warm and is written once.
P, price, N = 20_000, 4 / 1e6, 1_000
print(f"\n20k-token prefix, {N:,} requests: uncached ${N * P * price:,.2f}, "
      f"cached ${cost(N, 1.25, read=0.05) * P * price:,.2f}")

# What the provider holds for it: the prefix's KV cache, per cached copy.
for model, kv in (("Qwen2.5-1.5B", 28 * 1024), ("Qwen2.5-32B", 256 * 1024), ("DeepSeek-R1 (MLA)", 61 * 576 * 2)):
    print(f"KV cache for that 20k-token prefix, {model:18s}: {P * kv / 2**30:5.2f} GiB")

Full script: breakeven.py

5-minute cache (write 1.25x, read 0.1x): cheaper from use 2 on; at 10 uses it costs 22% of uncached, at 100 uses 11%
1-hour cache (write 2.0x, read 0.1x): cheaper from use 3 on; at 10 uses it costs 29% of uncached, at 100 uses 12%

20k-token prefix, 1,000 requests: uncached $80.00, cached $4.10
KV cache for that 20k-token prefix, Qwen2.5-1.5B      :  0.53 GiB
KV cache for that 20k-token prefix, Qwen2.5-32B       :  4.88 GiB
KV cache for that 20k-token prefix, DeepSeek-R1 (MLA) :  1.31 GiB

With a 5-minute cache, the second use already pays for the write; by the tenth, the prefix costs 22% of what it would uncached, and the cost approaches the read price, a tenth, as reuse grows. A 1-hour cache needs a third use, and is worth it for traffic with gaps longer than 5 minutes but under an hour. For an agent sending a 20,000-token prefix a thousand times while it works, the bill for that prefix drops from $80 to about $4 at Claude Opus 5.5’s listed prices, where a read costs a twentieth (about $8 at a tenth).

What it saves in time

The price is half the story. A cached prefix also skips its prefill, the part of a request’s latency before the first token. Measured with Qwen2.5-1.5B-Instruct on this laptop, an Apple M4 with 16 GB, where the whole measurement takes about 3 minutes. The prefix text is blog_train.txt, the blog text that the data script of Sequences Before Attention writes, run on the posts as they stood on 2026-10-04. The timing function of ttft.py synchronizes the GPU before and after the timed call, through sync, which picks the CUDA or Apple-GPU call for the machine:

@torch.no_grad()
def first_token(ids, past=None):
    sync(); t0 = time.perf_counter()
    out = m(ids, past_key_values=past, use_cache=True, logits_to_keep=1)   # logits for the last position only, as a server does
    out.logits[0, -1].argmax().item()                                 # forces the result
    sync()
    return time.perf_counter() - t0, out

print(f"{'prefix':>7s} {'cold':>9s} {'cached':>9s} {'saved':>6s} {'prefix KV held':>15s}")
for n in (1024, 2048, 4096, 8192):
    prefix = tok(text[:n * 6], return_tensors="pt").input_ids[:, :n].to(DEVICE)
    full = torch.cat([prefix, question], 1)
    first_token(full)                                                 # warm-up
    cold = np.median([first_token(full)[0] for _ in range(5)])
    _, pre = first_token(prefix)                                      # compute the prefix's cache once
    cached = np.median([first_token(question, copy.deepcopy(pre.past_key_values))[0] for _ in range(5)])
    print(f"{n:7d} {cold * 1e3:7.0f} ms {cached * 1e3:7.0f} ms {1 - cached / cold:6.0%} {n * kv_per_token / 2**20:11.0f} MiB")

Full script: ttft.py

The script loads the model through lm.py and first prints the KV cache’s size per token, from the model’s config.

KV cache per token: 28 KiB (2 x 28 layers x 2 KV heads x 128 x 2 bytes)
 prefix      cold    cached  saved  prefix KV held
   1024     918 ms      52 ms    94%          28 MiB
   2048    1843 ms      60 ms    97%          56 MiB
   4096    3854 ms      77 ms    98%         112 MiB
   8192    8819 ms     112 ms    99%         224 MiB

Without the cache, the time to the first token grows faster than the prefix: eight times the tokens took about ten times as long. Most of prefill is linear in the prompt (every token passes through every weight), but attention’s share grows with the square of the length, so it is a larger part of the work at 8,192 tokens than at 1,024. With the cached prefix, only the question’s tokens are processed, attending to it, and the first token came 94% sooner at 1,024 tokens and 99% sooner at 8,192: 0.11 seconds instead of 8.8. The cached time still grows a little with the prefix, since those few tokens attend to all of it. These are forward-pass times on one laptop; an API’s time to first token also includes the network and queueing, so the saving a provider reports is smaller than these percentages.

The last column is the price on the provider’s side: 28 KiB of keys and values per token for this model, so 224 MiB held in memory for an 8,192-token prefix, for as long as the entry lives. That is what the cache-write premium and the short lifetimes pay for.

What the provider holds

The last lines of the break-even output are why cache lifetimes are minutes, not days. A 20,000-token prefix is 0.5 GiB of KV cache on a 1.5B model and almost 5 GiB on a 32B one with grouped-query attention, in GPU memory or a nearby tier, per distinct prefix. Architecture changes the bill: DeepSeek-R1, a 671B model with multi-head latent attention (Anatomy of a Frontier Model), holds less per token than the 32B model does (68.6 KiB against 256 KiB, counting its compressed latent in 16 bits; Anatomy of a Frontier Model sizes the newer V3.2’s at fewer bytes, stored in 8-bit). A provider keeps an entry only while the reuse it expects pays for the memory, and drops it when it goes cold.

Ordering a prompt for the cache

Everything follows from the exact-prefix rule. Put what changes least first and what changes most last:

1. tool definitions, which rarely change;
2. the system prompt, without timestamps, user names or anything else per request;
3. long reference documents shared across requests;
4. the conversation so far, appended to, never rewritten;
5. the new user message.

A cache breakpoint goes after the last stable part. In a multi-turn conversation it moves forward each turn (Anthropic’s automatic mode does this itself), so each request reads the previous turn’s prefix from the cache and writes a slightly longer one. Anthropic looks for an earlier cached prefix only within 20 blocks of a breakpoint, so an agent turn that adds more than that (many tool calls and results) needs a second breakpoint to keep hitting.

How agents break their own cache

Agents (From Function Calling to the Agent Loop) are where caching saves the most, because they resend a long and growing context on every step, and where it breaks most easily:

• Editing the past. Summarizing or clearing old tool results to save context (Context Engineering) changes tokens in the middle of the prompt, and everything after them misses. Compaction trades a one-time cache miss for a shorter prompt from then on; doing it every turn means always missing.
• Changing tools. Adding or removing a tool rewrites the first part of the prompt, and invalidates everything.
• Dynamic content up front. The current date, a request ID or a random example order in the system prompt makes every request a miss.
• Changing settings. Some request parameters are part of the cached prefix: on Anthropic, the thinking configuration and tool_choice; on OpenAI, the tools, the reasoning effort and the output format. Changing them, or adding or removing images, misses from that point on.

Where this leads

• Context Engineering decides what goes into the context in the first place.
• Paged Attention and SGLang and RadixAttention come back to prefix caching inside a serving engine.

Takeaways

• Prompt caching keeps the KV cache of a prompt’s prefix between requests. Attention is causal, so a prefix’s keys and values depend only on the prefix, and only an exact prefix can hit.
• Providers charge more to write an entry (1.25 times the input price for 5 minutes on Anthropic) and a tenth to read it, so a cached prefix pays for itself from its second use (its third, for a 1-hour entry).
• Besides the price, a hit skips the prefix’s prefill: on this laptop the first token of an 8,192-token prompt came in 0.11 seconds instead of 8.8.
• The provider holds the cache in memory, 0.5 GiB for a 20,000-token prefix on a 1.5B model and nearly 5 GiB on a 32B one, which is why entries live minutes.
• Order prompts from least to most changing: tools, system prompt, reference documents, conversation, new message. Timestamps up front, reordered tools, or edits to earlier turns break the cache from that point on.

References

Start with Anthropic’s and OpenAI’s prompt-caching documentation, which change often and give the current prices and limits; the SGLang paper shows the same prefix sharing inside a server.

• Anthropic, Prompt caching (documentation, accessed October 2026): breakpoints, lifetimes, the write and read multipliers and the minimum lengths.
• OpenAI, Prompt caching (documentation, accessed October 2026): automatic caching, its prices and retention.
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS, 2024): RadixAttention, prefix sharing inside a server.
• Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference (MLSys, 2024): reusing attention states across prompts.