Serving Metrics and Prefill–Decode Disaggregation

The KV Cache split generation into two phases: prefill processes the whole prompt at once and is compute-bound; decode produces one token per step and is memory-bound. A serving system runs both, for many users at once, on the same hardware. This post is about measuring what users feel, and about the scheduling decisions that follow from the two phases being so different.

Table of contents

What to measure

A user of a streaming chat (Streaming Tokens to the User) feels two delays:

• Time to first token (TTFT): from sending the request to seeing the first token. It is queueing plus prefill, and grows with the prompt.
• Time per output token (TPOT): the average gap between output tokens over a request.
• Inter-token latency (ITL), or time between tokens: each individual gap. A gap is one decode step, and grows with how much else the GPU is doing in that step. A request with a 6 ms average can still contain one 500 ms stall, which TPOT hides and ITL doesn’t, and which the user sees as the text freezing.

A service promises limits on these, a service-level objective (SLO), usually on a high percentile: “99% of requests see their first token within 2 seconds and no gap longer than 50 ms”. High percentiles matter because a chat answer is hundreds of tokens: a stall that hits 1% of steps hits nearly every user. Throughput, tokens or requests per second, says how much the hardware does; it can rise while users suffer. Goodput (Zhong et al., 2024) is the highest request rate per GPU at which a target share of requests (say 90%) still meet the SLO, and is the number that decides how many GPUs a service needs. Near capacity, waiting time grows sharply, as in any queue: time to first token can be fine at moderate load and explode as the load approaches capacity.

Why the phases interfere

With continuous batching (vLLM Architecture), every step of the GPU serves all running requests: one decode token each, plus whatever prompts are waiting to be prefilled. A step’s time is set by the larger of reading the weights and doing the arithmetic (see the roofline of How a GPU Runs Deep Learning). Decode-only steps are cheap: the weights are read once for the whole batch. But a step that also prefills a 2,000-token prompt does 2,000 tokens of compute, and every user who was decoding waits for it: their next token is late by the whole prefill. Long prompts arriving at random make the gaps jumpy, and the high percentiles are what SLOs measure.

Two fixes:

• Chunked prefill (Agrawal et al., 2024, Sarathi-Serve): split each prompt into chunks and add one chunk to each step, so every step’s time is bounded by the chunk size, which is chosen to fit the per-token target (Sarathi-Serve calls it the token budget). Prefill takes more steps, which costs some TTFT; decode stalls disappear.
• Disaggregation (Zhong et al., 2024, DistServe; Patel et al., 2024, Splitwise): run prefill and decode on different GPUs. Decode GPUs never see a prefill, so TPOT is as smooth as it can be. The prompt’s KV cache has to move from the prefill GPU to the decode GPU, and the number of GPUs given to each phase has to match the traffic.

What the KV transfer costs

For the model below, the KV cache is 128 KiB per token: a 2,000-token prompt’s cache is 250 MiB, 5 ms over a 50 GB/s network link, while its prefill takes 65 ms. Both grow with the prompt, the prefill a little faster once attention’s quadratic cost counts, so there is no prompt length where the transfer starts to dominate. It is about 8% of the prefill’s time on an InfiniBand-class link, and much less over NVLink within a server. And it needn’t wait for the prefill to finish: Splitwise streams the cache layer by layer as it is computed, and DistServe places paired prefill and decode GPUs in the same server so the transfer stays on NVLink. The simulation below adds it after the prefill, without overlap, so it is pessimistic there. Small next to the prefill is not small next to a gap between tokens: a 16,000-token prompt’s cache takes 42 ms on the same link, and the request’s first decode step waits for all of it. In the simulation the transfer sets disaggregation’s worst gap, and the split of GPUs sets its goodput.

A simulation

A step-by-step simulation of serving an 8B model on H100s, with each step’s time from the roofline (weights and KV cache read at 3.35 TB/s, FLOPs at half of peak). Requests arrive at random, with prompts around 2,000 tokens (200 to 16,000) and 256 output tokens each. Every gap between a request’s tokens is recorded, and the SLO is on the worst one: first token within 2 seconds and every later token within 50 ms (loose) or 20 ms (tight). Chunked prefill is run with chunks of 512 and of 128 tokens. The step time, from serving_sim.py:

import numpy as np

# Serving one 8B model (Llama-3-8B shapes: 32 layers, 8 KV heads of 128,
# BF16) on H100s, simulated step by step. A step processes some prompt
# tokens (prefill) and one new token for every running sequence (decode);
# its time is the roofline of "How a GPU Runs Deep Learning": the larger of reading the weights
# and the KV cache it touches, and doing its FLOPs at half of peak.
N, BW, FLOPS = 8.0e9, 3.35e12, 0.5 * 989e12
KV_TOKEN = 2 * 32 * 8 * 128 * 2                         # bytes per token of KV cache (128 KiB)

def step_time(prefill_tokens, decode_seqs, decode_ctx):
    """Seconds for one step: prefill_tokens prompt tokens, decode_seqs
    sequences generating one token each with decode_ctx total context."""
    flops = 2 * N * (prefill_tokens + decode_seqs)
    bytes_ = 2 * N + KV_TOKEN * decode_ctx
    return max(bytes_ / BW, flops / FLOPS)

Full script: serving_sim.py

The rest of the script draws the arrivals and prompt lengths and schedules them with continuous batching on colocated GPUs and on separate prefill and decode GPUs. On a colocated GPU, each step takes a budget of prompt tokens: the whole first waiting prompt, or at most one chunk, mixed with one token for every running sequence:

            budget = chunk if chunk else (waiting[0][2] if waiting else 0)
            pre, done = 0, []
            for w in waiting:
                take = min(w[2], budget - pre)
                if take <= 0:
                    break
                w[2] -= take; pre += take
                if w[2] == 0:
                    done.append(w)
            ctx = sum(r[0] for r in running)
            t += step_time(pre, len(running), ctx)

Full script: serving_sim.py

On the disaggregated GPUs, each prompt is prefilled whole on the first free prefill GPU, and its first decode step waits for that prefill to end plus the KV cache’s transfer, KV_TOKEN * prompt / link. The script reports each design:

KV cache per token 128 KiB; a 2,000-token prompt's cache is 250 MiB, 5.2 ms over a 50 GB/s link; its prefill alone takes 65 ms; a step stays weight-bound up to 148 tokens

2 GPUs; share of requests whose first token comes within 2 s and every later token within 50 ms (loose) or 20 ms (tight)
req/s design                     TTFT p50    p99  TPOT p99 worst gap p99  loose  tight
    4 colocated, whole prefill     0.07s   0.47s    10.8ms    517.8ms    16%     5%
    4 colocated, chunks of 512     0.07s   0.48s     8.7ms     16.9ms   100%   100%
    4 colocated, chunks of 128     0.09s   0.59s     5.8ms      6.1ms   100%   100%
    4 disaggregated 1 + 1          0.09s   0.61s     6.6ms     44.8ms   100%    83%
    8 colocated, whole prefill     0.08s   0.52s    21.3ms    517.9ms     2%     0%
    8 colocated, chunks of 512     0.09s   0.54s    12.4ms     17.4ms   100%   100%
    8 colocated, chunks of 128     0.13s   1.04s     6.2ms      6.6ms   100%   100%
    8 disaggregated 1 + 1          0.22s   2.11s     7.2ms     43.8ms    99%    77%
   12 colocated, whole prefill     0.10s   0.60s    27.2ms    518.8ms     0%     0%
   12 colocated, chunks of 512     0.10s   0.65s    14.5ms     17.5ms   100%   100%
   12 colocated, chunks of 128     0.21s   1.45s     6.2ms      6.7ms   100%   100%
   12 disaggregated 1 + 1          6.25s  10.05s     7.3ms     43.3ms     6%     4%

4 GPUs
req/s design                     TTFT p50    p99  TPOT p99 worst gap p99  loose  tight
   24 colocated, chunks of 128     0.20s   1.31s     6.2ms      6.7ms   100%   100%
   24 disaggregated 2 + 2          5.85s  12.80s     7.9ms     45.7ms    12%     9%
   24 disaggregated 3 + 1          0.10s   0.51s    18.9ms     57.7ms    98%     7%

Read down each block as traffic grows.

• Interference is in the worst gaps, not the averages. With whole prefills, a 16,000-token prompt stalls every decoding user for about half a second: the 99th-percentile worst gap is 518 ms at every load, and even at 4 requests a second only 16% of requests never see a gap over 50 ms. The mean (TPOT) hides this: its 99th percentile is 11 ms.
• Chunked prefill, sized to the target, fixes it. With chunks of 512, 99% of requests’ worst gaps are under 18 ms; with chunks of 128, under 7 ms, and chunks of 128 meet even the tight target for every request at every load tried, including 24 requests a second on 4 GPUs. Smaller chunks cost time to first token (the 99th percentile rises from 0.65 to 1.45 seconds at 12 requests a second), still within target.
• Disaggregation, in this model, doesn’t beat tuned chunking. Its decode GPU never stalls on a prefill, but a request’s first decode step waits for the cache to arrive and for the next step. The transfer grows with the prompt, to 42 ms for 16,000 tokens, so long prompts’ transfers make up most of the 43 to 46 ms 99th-percentile worst gap here. And a fixed split of GPUs can run out of one phase: one prefill GPU tops out at about 11 requests a second, so at 12 the prompts queue for seconds; with 4 GPUs at 24 requests a second, 2 + 2 starves prefill and 3 + 1 crowds decode (19 ms per token).

Two cautions on reading this. The comparison isn’t symmetric: two colocated GPUs both prefill, so they have twice the prefill capacity of a 1 + 1 split, which, more than interference, is why the split collapses at 12 requests a second. And the model is idealized in ways that flatter chunking. Its roofline assumes compute and memory overlap perfectly, so in it a step is free up to the ridge point of 148 tokens, and a 128-token chunk rides along almost at no cost. Real steps pay per-step overheads, each chunk re-reads the prompt’s earlier KV cache, and attention’s quadratic FLOPs are left out. It also has no memory limits or preemption, no batching of prefills (harmless above the ridge point), and no prefix caching.

Disaggregation’s advantages in practice come from what the model leaves out: running prefill and decode with different parallelism and batch settings, and so tuning each phase’s latency separately (DistServe); different hardware for each phase, since decode needs memory bandwidth and prefill needs compute (Splitwise); and isolating the very long prompts that make any shared step slow. Mooncake, Kimi’s serving system, runs disaggregation in production around a distributed pool of cached KV in the cluster’s otherwise idle CPU memory and SSDs, so a prompt whose prefix was seen before skips its prefill, and it rejects requests early under overload, which is how a service protects its goodput when traffic outruns it.

Where this leads

• KV Cache Compression shrinks the KV cache itself: evicting tokens and quantizing entries.
• Paged Attention and vLLM Architecture show how a real engine stores the cache in blocks and runs the continuous batching and chunked prefill simulated here.
• Speculative Decoding, Derived makes each decode step produce more than one token.

Takeaways

• Users feel time to first token (queueing plus prefill) and the gaps between later tokens; TPOT averages the gaps and hides stalls, ITL doesn’t. SLOs bound high percentiles, and goodput is the request rate per GPU that still meets them.
• Prefill and decode on the same GPU interfere: in simulation, whole prefills of long prompts froze decoding users for about half a second, while the average time per token looked fine.
• Chunked prefill with a chunk sized to the per-token target removed the stalls at every load simulated, at some cost in time to first token.
• Disaggregation removes the interference at the price of a KV-cache transfer (about 8% of the prefill time here, but 42 ms for a 16,000-token prompt, which sets the worst gap between tokens) and a split of GPUs that must match the traffic; a wrong split loses badly. Its real advantages are per-phase tuning, per-phase hardware and isolating long prompts, which an idealized model doesn’t capture.

References

Start with DistServe for goodput and disaggregation, then Sarathi-Serve for chunked prefill; Mooncake shows disaggregation in production.

• Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI, 2024): chunked prefill and the token budget.
• Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (OSDI, 2024): goodput, disaggregation, placement search.
• Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting (ISCA, 2024): disaggregation with different hardware per phase, and streaming the KV cache layer by layer.
• Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving (FAST, 2025): disaggregation in production, for Kimi.