Tag: bits-per-byte
All the articles with the tag "bits-per-byte".
What "Generative" Means: Autoregression, the Chain Rule, and Perplexity
Updated:A language model is one factorization, p(text) = the product of each token's probability given the ones before it, and training, sampling, caching and evaluation all follow from it. The chain rule worked on this blog, teacher forcing vs generation, what a small model writes when greedy and when sampling, perplexity derived from cross-entropy, why perplexity can't be compared across tokenizers and bits per byte can, measured with a character model and a BPE model on the same text.