Tag: gradient-descent
All the articles with the tag "gradient-descent".
Initialization and Optimizers: Where to Start, and How to Step
Updated:Why a deep network's signal shrinks or explodes layer by layer and how He initialization keeps it level, then the optimizers every language model is trained with: momentum and its square-root speed-up, Adam as a per-parameter rescaling with its bias correction derived, why training starts with a warmup, what AdamW's weight decay really changes, and what Adam costs in memory.
Backpropagation, by Hand: The Chain Rule, Run Backward
Updated:Why a hidden layer can learn what a straight line can't, and how backpropagation gets the gradient of every weight in one backward pass. Worked by hand on XOR with every number, then as matrix rules for a whole layer, checked against finite differences, and costed: where the '6 × parameters FLOPs per training token' rule comes from.
Learning Is Optimization: Linear and Logistic Regression, Trained by Gradient Descent
Updated:What it actually means for a model to learn: pick a loss, then walk downhill. Worked end to end on real data from this blog, with the loss choice, the closed-form fit, gradient descent and its learning-rate limit, and a first logistic-regression router whose gradient reappears in every later post.