Tag: serving
All the articles with the tag "serving".
Serving Metrics and Prefill–Decode Disaggregation
Updated:Time to first token is a prefill problem and time per output token is a decode problem; serving both on one GPU makes them interfere, and splitting them trades that interference for a KV-cache transfer and a ratio to get right. The metrics and SLOs, goodput, chunked prefill, and disaggregation, then a step-by-step simulation of an 8B model on H100s under Poisson traffic, measuring every gap between tokens: why averages hide stalls, why chunked prefill sized to the target wins in an idealized model, and what disaggregation offers that the model leaves out.
Streaming Tokens to the User: SSE, WebSockets, and the Wait the Reader Feels
Updated:Streaming doesn't make a model generate any faster. It moves the wait the reader feels from the whole answer to the first token. A streaming server on Python's standard library, delivering a real model's tokens three ways: one response, server-sent events, and a WebSocket written by hand. Measured: time to first text, bytes on the wire, reading speed against generation speed, what happens when the user leaves, and how a buffering proxy quietly undoes all of it.
System Design for AI: A Real-Time Voice Agent, Millisecond by Millisecond
Updated:A voice agent has about one second between the user stopping and the agent starting, and five components share it. Requirements and a worked latency budget stage by stage, where end-of-turn detection turns out to be the biggest item; transport over WebRTC; streaming speech recognition and synthesis; the LLM's time to first token; cascaded versus speech-to-speech models; barge-in and tool calls mid-turn; and capacity for 1,000 simultaneous calls from a simulation of continuous batching, which needs far fewer GPUs than intuition says because speech is slow.