Tag: cross-attention
All the articles with the tag "cross-attention".
The Transformer's Three Attention Patterns: Self, Masked, and Cross
Updated:Self-attention, masked attention and cross-attention are one computation, softmax(QKᵀ/√d)V, with Q, K and V drawn from different places. Derived from a weighted average, worked through on small matrices, then checked on real models: Qwen2.5-1.5B's mask hides the future exactly, its heads learn to look one token back or at the first token, a translation model's cross-attention lines German words up with English ones, and the cost grows as the square of the length.