Tag: kv-cache
All the articles with the tag "kv-cache".
KV Cache Compression: Eviction and Quantization
Updated:After sharing key-value heads, two levers are left for the KV cache: store fewer tokens, or store each in fewer bits. Both are lossy in different ways. The budget worked for Llama-2-13B, eviction from sliding windows to attention sinks, heavy hitters and SnapKV, and per-token versus per-channel quantization as in KIVI; then both measured on a real model's loss on held-out text, against the full cache.
The KV Cache: How Transformers Avoid Recomputing the Past
Updated:Generating text recomputes every earlier token's keys and values unless they are kept. The KV cache keeps them, and it is exact: in Qwen2.5-1.5B-Instruct the next token matched at 32 of 32 steps. It turned a step at 4,096 tokens from 4.7 s into 65 ms on a laptop. Then what it costs: 28 KiB per token for Qwen, 800 KiB for Llama 2 13B, why decoding is limited by memory reads, how batch size trades latency for throughput, and how GQA, MLA and cross-layer sharing shrink the cache.
Paged Attention: Managing the KV Cache Like Virtual Memory
Updated:A server can't know how long an answer will be, so a KV cache stored in one contiguous piece must be reserved for the worst case. Measured on requests from the two datasets the vLLM paper used, ShareGPT and Alpaca, that leaves 3 to 63% of the reservation holding real tokens; 16-token blocks with a block table, as an operating system pages memory, hold 90 to 98%. A paged cache built for a real model and checked token for token, attention over scattered blocks as a GPU kernel, the block size, sharing with copy-on-write, prefix caching by block hashes, and what the paper's throughput numbers do and don't say.
Prompt Caching: Paying Once for a Prefix
Updated:API prompt caching is the KV cache and prefix sharing sold as a price. What is actually cached, why only an exact prefix can hit, breakpoints and lifetimes, the write and read prices and the break-even worked from published multipliers, the time a cached prefix saves measured on a 1.5B model, the gigabytes a provider holds for one, how to order a prompt, and how agents break their own cache.