Tag: quantization
All the articles with the tag "quantization".
KV Cache Compression: Eviction and Quantization
Updated:After sharing key-value heads, two levers are left for the KV cache: store fewer tokens, or store each in fewer bits. Both are lossy in different ways. The budget worked for Llama-2-13B, eviction from sliding windows to attention sinks, heavy hitters and SnapKV, and per-token versus per-channel quantization as in KIVI; then both measured on a real model's loss on held-out text, against the full cache.
Quantization from Scratch: Scale, Zero-Point, GPTQ and AWQ
Updated:Quantization maps floats onto a small integer grid with a scale and a zero-point; everything else is about where the rounding error lands. A worked 4-bit example, per-tensor vs per-channel vs group-wise scales, and GPTQ and AWQ implemented from their papers and run on two real layers of Qwen2.5-0.5B with their real inputs: what each buys at 8, 4 and 3 bits, why GPTQ needs enough calibration data, and why AWQ helps only where activations have outliers.
Running LLMs on a Laptop: GGUF and llama.cpp
Updated:Local inference is the quantization and memory-bandwidth story at consumer scale: GGUF packages a quantized model in one memory-mappable file, and llama.cpp turns bytes per second into tokens per second. Inside a real GGUF file, what K-quant names like Q4_K_M mean, and tokens per second measured on this laptop's GPU and CPU against the bandwidth prediction, with what each quantization level costs in quality.
TensorRT-LLM: Compiling Inference Engines Ahead of Time
Updated:TensorRT-LLM began by compiling each model into an engine: operations fused into fewer kernels, and every kernel timed and chosen for the GPU and the shapes it would serve. Both steps measured on a laptop GPU: fusion removed 26% of the operations per token and 7% of prefill time; the best version of one kernel depended on the shape, and the worst cost twice as much. Then why NVIDIA moved the project onto a PyTorch runtime, and what it keeps of the compiled design.