Tag: hardware
All the articles with the tag "hardware".
Beyond the GPU: TPUs, LPUs and What Specialization Buys
Updated:A GPU keeps machinery for any parallel program; specialized chips win by removing it. A systolic array simulated cycle by cycle on a 3×3 matmul, and why it reuses each value read from memory hundreds of times; Google's TPU from the 2017 original to Ironwood, its interconnect and compiler; Groq's LPU, which keeps a model in on-chip SRAM and schedules every cycle at compile time; and the arithmetic of the trade: 609 chips to hold a 70B model in SRAM, against what eight HBM chips can do for one stream of tokens.