Tag: inference
All the articles with the tag "inference".
Speculative Decoding, Derived: Draft Models, N-grams, Medusa and EAGLE
Updated:Decoding is memory-bound, so checking k guessed tokens costs about as much as generating one. A rejection-sampling rule makes the output exactly the target model's distribution whatever the guesser, proved and checked by sampling. The expected tokens per step derived and checked, the speedup with the draft's cost, and an interactive simulator; draft models, prompt lookup, Medusa heads and EAGLE; and acceptance measured on real models for two kinds of task.
KV Cache Compression: Eviction and Quantization
Updated:After sharing key-value heads, two levers are left for the KV cache: store fewer tokens, or store each in fewer bits. Both are lossy in different ways. The budget worked for Llama-2-13B, eviction from sliding windows to attention sinks, heavy hitters and SnapKV, and per-token versus per-channel quantization as in KIVI; then both measured on a real model's loss on held-out text, against the full cache.
Quantization from Scratch: Scale, Zero-Point, GPTQ and AWQ
Updated:Quantization maps floats onto a small integer grid with a scale and a zero-point; everything else is about where the rounding error lands. A worked 4-bit example, per-tensor vs per-channel vs group-wise scales, and GPTQ and AWQ implemented from their papers and run on two real layers of Qwen2.5-0.5B with their real inputs: what each buys at 8, 4 and 3 bits, why GPTQ needs enough calibration data, and why AWQ helps only where activations have outliers.
Reasoning Models and Test-Time Compute
Updated:A reasoning model spends generated tokens as compute: trained with verifiable rewards, it learns to think before answering, and gets better the longer it thinks. What test-time compute means, how RL produces long chains and distillation passes them to small models, thinking budgets, and what 1,000, 10,000 and 50,000 thinking tokens cost in KV cache and time on two real models. Then a 1.5B model on checkable word problems, two seeds: answering directly, reasoning step by step, and voting over up to 16 chains, with the tokens each one costs, then on harder eight-step problems, where voting stops helping and a verifier could still gain.