Tag: transformers
All the articles with the tag "transformers".
The Transformer's Three Attention Patterns: Self, Masked, and Cross
Updated:Self-attention, masked attention and cross-attention are one computation, softmax(QKᵀ/√d)V, with Q, K and V drawn from different places. Derived from a weighted average, worked through on small matrices, then checked on real models: Qwen2.5-1.5B's mask hides the future exactly, its heads learn to look one token back or at the first token, a translation model's cross-attention lines German words up with English ones, and the cost grows as the square of the length.
Byte-Pair Encoding: How Text Becomes Tokens, One Merge at a Time
Updated:A byte-level BPE tokenizer built from scratch and traced merge by merge: UTF-8 bytes, GPT-2's pre-tokenizer, training by count and encoding by rank, special tokens, the file format, a trainer made fast without changing a merge, 2.2 GB of stories, and what a tokenizer charges Vietnamese.