Tag: efficiency
All the articles with the tag "efficiency".
Attention Alternatives: Sparse, Linear, and Recurrent
Updated:Softmax attention costs quadratic compute at prefill and a cache that grows with every token. There are three ways out: attend to fewer tokens, drop the softmax, or replace attention with a recurrence. Each is derived here and checked in code: linear attention matches the quadratic form to 1e-16 and runs 75x faster than it at 16K tokens on a laptop GPU, a 64-wide linear-attention state reads back 100% of 64 stored pairs but 35% of 512, and the configs of thirteen released models show what shipped.
Quantization from Scratch: Scale, Zero-Point, GPTQ and AWQ
Updated:Quantization maps floats onto a small integer grid with a scale and a zero-point; everything else is about where the rounding error lands. A worked 4-bit example, per-tensor vs per-channel vs group-wise scales, and GPTQ and AWQ implemented from their papers and run on two real layers of Qwen2.5-0.5B with their real inputs: what each buys at 8, 4 and 3 bits, why GPTQ needs enough calibration data, and why AWQ helps only where activations have outliers.