Posts
All the articles I've posted.
Features and Overfitting: How a Model Memorizes, and How L1 and L2 Stop It
Updated:Give the router 379 features instead of one and it scores 100% on its training data and worse on new questions. Why that happens, why the weights run to infinity, how L2 and L1 penalties stop them (derived, with the soft-threshold that makes L1 exact zeros), why one well-chosen feature matches all 379, and how to tell a real difference from noise when the splits share data.
How a GPU Runs Deep Learning: Memory, the Roofline, and a First Kernel
Updated:A GPU is thousands of simple cores fed by a memory system that is usually the real bottleneck. The H100's layout from its whitepaper, threads, warps and blocks, the roofline and why an operation's FLOPs per byte decides which ceiling it hits, then real kernels written and measured on a laptop GPU: a vector add, coalesced vs scattered reads, and a matrix multiply from naive to tiled, with the arithmetic of why tiles and matrix shapes matter.
Initialization and Optimizers: Where to Start, and How to Step
Updated:Why a deep network's signal shrinks or explodes layer by layer and how He initialization keeps it level, then the optimizers every language model is trained with: momentum and its square-root speed-up, Adam as a per-parameter rescaling with its bias correction derived, why training starts with a warmup, what AdamW's weight decay really changes, and what Adam costs in memory.
PyTorch from the Inside: Tensors, Autograd and Precision
Updated:A deep-learning framework is three things: a tensor library, an autograd tape, and a device model. Strides and views, a 55-line autograd that matches PyTorch to the last bit or two, what the recorded graph looks like, why GPU timings lie without a synchronize, FP32 vs FP16 vs BF16 bit by bit and why BF16 won, the router's training loop matched to the hand-written AdamW, the 16 bytes per parameter measured, and what torch.compile adds.