Tag: memory-bandwidth
All the articles with the tag "memory-bandwidth".
How a GPU Runs Deep Learning: Memory, the Roofline, and a First Kernel
Updated:A GPU is thousands of simple cores fed by a memory system that is usually the real bottleneck. The H100's layout from its whitepaper, threads, warps and blocks, the roofline and why an operation's FLOPs per byte decides which ceiling it hits, then real kernels written and measured on a laptop GPU: a vector add, coalesced vs scattered reads, and a matrix multiply from naive to tiled, with the arithmetic of why tiles and matrix shapes matter.
FlashAttention: Exact Attention at the Speed of Memory
Updated:Standard attention is slow at long contexts not because of its arithmetic but because it writes an n-by-n matrix of scores to memory and reads it back. FlashAttention computes exactly the same result without ever storing that matrix, using tiling and a softmax computed one block at a time. The online softmax derived and checked to within 1e-15, the memory traffic counted, a fused kernel measured against the three-step version, causal masking, and what FlashAttention 2 and 3, and Flash-Decoding, changed.