Tag: tokenizer

All the articles with the tag "tokenizer".

• Byte-Pair Encoding: How Text Becomes Tokens, One Merge at a Time

  Updated:16 Oct, 2026

  A byte-level BPE tokenizer built from scratch and traced merge by merge: UTF-8 bytes, GPT-2's pre-tokenizer, training by count and encoding by rank, special tokens, the file format, a trainer made fast without changing a merge, 2.2 GB of stories, and what a tokenizer charges Vietnamese.