Tag: attention
All the articles with the tag "attention".
Where Am I? Position in Attention, from Sinusoids to RoPE
Updated:Attention on its own can't tell word order apart, proved in two lines. How position gets in: added sinusoids, learned embeddings, ALiBi's distance penalty, and RoPE's rotation, derived so the query-key score depends only on relative distance. Then a tiny Transformer trained with each, tested on sequences four times longer than it trained on, on one task that needs exact relative position and one that needs to look back over any distance, and what the results say about long-context extension.
The Transformer's Three Attention Patterns: Self, Masked, and Cross
Updated:Self-attention, masked attention and cross-attention are one computation, softmax(QKᵀ/√d)V, with Q, K and V drawn from different places. Derived from a weighted average, worked through on small matrices, then checked on real models: Qwen2.5-1.5B's mask hides the future exactly, its heads learn to look one token back or at the first token, a translation model's cross-attention lines German words up with English ones, and the cost grows as the square of the length.