Tag: information-theory

All the articles with the tag "information-theory".

• Cross-Entropy and the Softmax Gradient: One Loss for Every Classifier

  Updated:19 Oct, 2026

  From two routes to four: the softmax, the loss every language model is trained with and where it comes from in information theory, why its gradient is again just prediction minus target, how to compute it without overflow, label smoothing, a four-way router read through its confusion matrix, and what the same loss costs in memory at a 128,256-token vocabulary.