Tag: goodput
All the articles with the tag "goodput".
Serving Metrics and Prefill–Decode Disaggregation
Updated:Time to first token is a prefill problem and time per output token is a decode problem; serving both on one GPU makes them interfere, and splitting them trades that interference for a KV-cache transfer and a ratio to get right. The metrics and SLOs, goodput, chunked prefill, and disaggregation, then a step-by-step simulation of an 8B model on H100s under Poisson traffic, measuring every gap between tokens: why averages hide stalls, why chunked prefill sized to the target wins in an idealized model, and what disaggregation offers that the model leaves out.