Tag: regularization

All the articles with the tag "regularization".

• Features and Overfitting: How a Model Memorizes, and How L1 and L2 Stop It

  Updated:24 Oct, 2026

  Give the router 379 features instead of one and it scores 100% on its training data and worse on new questions. Why that happens, why the weights run to infinity, how L2 and L1 penalties stop them (derived, with the soft-threshold that makes L1 exact zeros), why one well-chosen feature matches all 379, and how to tell a real difference from noise when the splits share data.

• Dropout and Normalization: Two Layers Every Diagram Has

  Updated:16 Oct, 2026

  Two layers that sit in almost every network diagram, and what each one actually does: dropout as a regularizer, derived and tested (on its own it makes the router overconfident; alongside weight decay, and in long training, it helps), then normalization, from BatchNorm and why language models can't use it, to LayerNorm and RMSNorm with worked vectors and hand-written backward passes, what normalizing does to the gradient and the learning rate, and what a norm layer costs.