Tag: momentum
All the articles with the tag "momentum".
Initialization and Optimizers: Where to Start, and How to Step
Updated:Why a deep network's signal shrinks or explodes layer by layer and how He initialization keeps it level, then the optimizers every language model is trained with: momentum and its square-root speed-up, Adam as a per-parameter rescaling with its bias correction derived, why training starts with a warmup, what AdamW's weight decay really changes, and what Adam costs in memory.