Tag: pytorch
All the articles with the tag "pytorch".
PyTorch from the Inside: Tensors, Autograd and Precision
Updated:A deep-learning framework is three things: a tensor library, an autograd tape, and a device model. Strides and views, a 55-line autograd that matches PyTorch to the last bit or two, what the recorded graph looks like, why GPU timings lie without a synchronize, FP32 vs FP16 vs BF16 bit by bit and why BF16 won, the router's training loop matched to the hand-written AdamW, the 16 bytes per parameter measured, and what torch.compile adds.