Tag: temperature

All the articles with the tag "temperature".

• Knowledge Distillation: Learning from a Teacher's Distribution

  Updated:20 Oct, 2026

  A teacher's full next-token distribution carries far more information per example than the one token the text used. Soft labels and dark knowledge on a real teacher, temperature and the tau-squared factor derived and checked, forward vs reverse KL on a bimodal target, sequence-level and on-policy distillation, and distilled reasoning. Then Qwen2.5-0.5B trained on this blog from hard labels, from a 1.5B teacher's distribution, and from both, two seeds each, with what each learns and forgets.

• Decoding: From a Distribution to Text

  Updated:19 Oct, 2026

  A language model outputs a probability for every possible next token; decoding is the policy that turns that into text. Greedy and beam search, temperature derived, top-k, top-p and min-p on a real model's distributions, the order the steps run in, repetition penalties, and why the same prompt can give different text. Each one measured with Qwen2.5-1.5B on twenty paragraphs of this blog, against what a person actually wrote next.