Tag: inference
All the articles with the tag "inference".
Serving Metrics and Prefill–Decode Disaggregation
Updated:Time to first token is a prefill problem and time per output token is a decode problem; serving both on one GPU makes them interfere, and splitting them trades that interference for a KV-cache transfer and a ratio to get right. The metrics and SLOs, goodput, chunked prefill, and disaggregation, then a step-by-step simulation of an 8B model on H100s under Poisson traffic, measuring every gap between tokens: why averages hide stalls, why chunked prefill sized to the target wins in an idealized model, and what disaggregation offers that the model leaves out.
Beyond the GPU: TPUs, LPUs and What Specialization Buys
Updated:A GPU keeps machinery for any parallel program; specialized chips win by removing it. A systolic array simulated cycle by cycle on a 3×3 matmul, and why it reuses each value read from memory hundreds of times; Google's TPU from the 2017 original to Ironwood, its interconnect and compiler; Groq's LPU, which keeps a model in on-chip SRAM and schedules every cycle at compile time; and the arithmetic of the trade: 609 chips to hold a 70B model in SRAM, against what eight HBM chips can do for one stream of tokens.
Running LLMs on a Laptop: GGUF and llama.cpp
Updated:Local inference is the quantization and memory-bandwidth story at consumer scale: GGUF packages a quantized model in one memory-mappable file, and llama.cpp turns bytes per second into tokens per second. Inside a real GGUF file, what K-quant names like Q4_K_M mean, and tokens per second measured on this laptop's GPU and CPU against the bandwidth prediction, with what each quantization level costs in quality.