Tag: alibi
All the articles with the tag "alibi".
Where Am I? Position in Attention, from Sinusoids to RoPE
Updated:Attention on its own can't tell word order apart, proved in two lines. How position gets in: added sinusoids, learned embeddings, ALiBi's distance penalty, and RoPE's rotation, derived so the query-key score depends only on relative distance. Then a tiny Transformer trained with each, tested on sequences four times longer than it trained on, on one task that needs exact relative position and one that needs to look back over any distance, and what the results say about long-context extension.