Tag: bfloat16

All the articles with the tag "bfloat16".

• PyTorch from the Inside: Tensors, Autograd and Precision

  Updated:24 Oct, 2026

  A deep-learning framework is three things: a tensor library, an autograd tape, and a device model. Strides and views, a 55-line autograd that matches PyTorch to the last bit or two, what the recorded graph looks like, why GPU timings lie without a synchronize, FP32 vs FP16 vs BF16 bit by bit and why BF16 won, the router's training loop matched to the hand-written AdamW, the 16 bytes per parameter measured, and what torch.compile adds.