Tag: streaming-llm

All the articles with the tag "streaming-llm".

• KV Cache Compression: Eviction and Quantization

  Updated:19 Oct, 2026

  After sharing key-value heads, two levers are left for the KV cache: store fewer tokens, or store each in fewer bits. Both are lossy in different ways. The budget worked for Llama-2-13B, eviction from sliding windows to attention sinks, heavy hitters and SnapKV, and per-token versus per-channel quantization as in KIVI; then both measured on a real model's loss on held-out text, against the full cache.