Earlier posts looked at single components: how a GPU serves a model (Serving Metrics and Prefill–Decode Disaggregation), how tokens reach a browser (Streaming Tokens to the User), which chip to use (Beyond the GPU). This one designs a whole system, the way a system-design interview or a real project would: requirements first, then estimates, then components, each chosen to fit the numbers. The system is a real-time voice agent: a caller speaks, the agent answers out loud, and the conversation should feel like talking to a person.
Five components on one path, each streaming into the next. Schematic.
- A voice agent has about one second between the caller stopping and the agent starting, shared by transport, turn detection, STT, the LLM and TTS.
- The biggest item isn’t a model: waiting to be sure the caller has finished. Semantic turn detection cuts the budget from 1,160 to 910 ms.
- Every stage streams, and audio travels over WebRTC on UDP in 20 ms Opus frames, because a late packet is worse than a lost one.
- Speech is slow, so the LLM is idle most of a call: for 1,000 calls the constraint is where 262 GB of conversation caches live between turns, not compute.
Table of contents
Open Table of contents
Requirements
Functional: hold a spoken conversation; understand the caller; answer from a language model that can call tools (look up an order, book a slot); let the caller interrupt.
Non-functional, and these drive the design:
- Latency. In conversation between people, the gap between one person stopping and the other starting is typically around 200 ms (Stivers et al., 2009, across ten languages). Machines don’t have to match that, but past about a second the pause feels like the line dropped. The target here: the agent’s first audio within about 1 s of the caller stopping, at the median, and not much more at the tail.
- Scale. 1,000 simultaneous calls at peak.
- Robustness. Mobile networks with jitter and loss; background noise; callers who pause mid-sentence.
- Observability and cost: every stage’s latency per turn, and a cost per call-minute.
The latency budget
A cascaded voice agent chains three models: speech-to-text (STT), a
language model, and text-to-speech (TTS). Every stage between the caller’s
last word and the agent’s first sound spends part of the second. Here are
the stages with figures for a well-built system calling a hosted LLM over
the network. They are assumptions, stated so they can be argued with, not
measurements; small items (the VAD’s own 30 ms analysis windows, audio
resampling, packetizing TTS output) are left out. From latency_budget.py:
stages = [ # (stage, milliseconds, why)
("audio frame + jitter buffer", 40, "20 ms Opus frames; a jitter buffer of about one frame"),
("network, user -> server", 30, "one way, same region, WebRTC over UDP"),
("end-of-turn detection", 400, "VAD waits for this much silence before deciding the user is done"),
("speech-to-text, final", 100, "a streaming recognizer has the words already; it finalizes"),
("LLM time to first token", 250, "prefill of a cached 2,000-token prompt plus queueing"),
("LLM, first clause", 150, "TTS needs a few words to start: ~8 tokens at ~19 ms each"),
("text-to-speech, first audio", 120, "streaming TTS, time to its first audio chunk"),
("network, server -> user", 30, "one way"),
("playback buffer", 40, "the client's own jitter buffer before audio plays"),
]
total = sum(ms for _, ms, _ in stages)
print(f"{'stage':30s} {'ms':>5s} {'cumulative':>11s}")
run = 0
for name, ms, _ in stages:
run += ms
print(f"{name:30s} {ms:5d} {run:11d}")
print(f"\ntotal {total} ms; end-of-turn detection alone is {400 / total:.0%} of it")
# Two levers, applied to the same budget.
for name, change in (("semantic turn detection (400 -> 150 ms of silence)", {"end-of-turn detection": 150}),
("speech-to-speech model (STT + LLM + TTS -> 350 ms)", None)):
if change:
t = sum(change.get(n, ms) for n, ms, _ in stages)
else:
t = sum(ms for n, ms, _ in stages if not n.startswith(("speech-to-text", "LLM", "text-to-speech"))) + 350
print(f" with {name}: {t} ms")Full script: latency_budget.py
stage ms cumulative
audio frame + jitter buffer 40 40
network, user -> server 30 70
end-of-turn detection 400 470
speech-to-text, final 100 570
LLM time to first token 250 820
LLM, first clause 150 970
text-to-speech, first audio 120 1090
network, server -> user 30 1120
playback buffer 40 1160
total 1160 ms; end-of-turn detection alone is 34% of it
with semantic turn detection (400 -> 150 ms of silence): 910 ms
with speech-to-speech model (STT + LLM + TTS -> 350 ms): 890 msThe total is over budget, and the largest single item isn’t a model at all. It’s end-of-turn detection: deciding that the caller has finished. A voice activity detector (VAD) knows when there is speech, but people pause mid-sentence, so the system has to wait for some silence before treating a pause as the end of a turn. Too short a wait and the agent interrupts people who were thinking; too long and every reply starts late. 400 ms of silence is a third of the budget.
A third of the second goes to waiting for silence.
Data: the stated assumptions aboveThat makes it the first lever. Semantic turn detection uses a small model that reads the transcript so far and judges whether the sentence is complete (“I’d like to book a table for” is not; “That’s all, thanks” is), so the silence threshold can be short when the words say the turn is over and long when they don’t. Cutting the average wait to 150 ms brings the budget to 910 ms.
A second lever overlaps stages instead of shrinking them. During the silence wait, the system can already finalize the transcript and start the LLM’s prefill on it, speculatively, and throw the work away if the caller keeps talking. Many streaming recognizers also do their own endpointing, so “wait for silence” and “finalize the transcript” are partly the same 400 ms rather than two items in series. Done well, much of the STT and TTFT rows disappear behind the turn detection.
The LLM rows also depend on where the model runs. 250 ms is a typical time to first token for a hosted API reached over the network; a self-hosted model with the conversation cached, as in the capacity section below, can start in tens of milliseconds, which takes another 200 ms off.
The rest of the budget follows from one design rule: every stage streams. STT transcribes while the caller is still speaking, so when the turn ends it only finalizes. The LLM’s first tokens go to TTS as soon as they form a clause, not when the reply is complete. TTS sends its first audio chunk while it’s still synthesizing the rest. A system that waited for each stage to finish would spend several seconds.
Transport: WebRTC
Phone-quality conversation needs audio delivered in tens of milliseconds, and it tolerates a lost packet far better than a late one. That makes TCP a poor fit for the audio’s last mile, the caller’s own network: TCP retransmits a lost packet and holds back everything behind it, which is right for a web page and wrong for a voice. (Between data centers, where loss is rare, audio is often carried over WebSockets on TCP without trouble.) WebRTC (RFC 8825) carries audio over UDP in encrypted real-time transport packets (SRTP), compressed with the Opus codec in 20 ms frames, and the receiver keeps a jitter buffer to smooth out uneven arrival: about one frame on a good connection, several frames on a congested mobile one, which comes straight out of the budget. Getting a UDP connection through home routers and corporate firewalls is its own problem. ICE (interactive connectivity establishment, RFC 8445) tries candidate paths in turn: direct, then via the public address a client appears from, discovered with STUN (RFC 8489), and finally through a TURN relay server (RFC 8656) when no direct path works. For phone calls, a telephony provider bridges the phone network (over SIP, with the narrowband G.711 codec on the phone side) to the same audio stream, and that bridge adds latency the budget above doesn’t include. Servers in several regions, close to the callers, keep the network legs at the 30 ms the budget assumes.
ICE tries the cheapest path first. Schematic.
The text side (transcripts, tool events, the agent’s state) can travel on a WebRTC data channel or a WebSocket (Streaming Tokens to the User); it’s small and latency tolerant by comparison.
The three models
Speech-to-text. A streaming recognizer emits partial transcripts every few hundred milliseconds and revises them as more audio arrives. Whisper (Radford et al., 2022) set the accuracy bar for open models but processes 30-second windows, so real-time systems either use streaming-native recognizers or run Whisper-style models on short overlapping chunks. What matters for the budget is the time from the end of speech to a final transcript, which a streaming recognizer keeps around 100 ms.
The language model. Two numbers from Serving Metrics and Prefill–Decode Disaggregation matter: time to first token (TTFT), which sits directly on the critical path, and the time per output token, which matters only until the first clause is ready for TTS. After that, the agent is limited by how fast it can speak: about 150 words a minute, roughly three or four tokens a second. A model that generates 50 tokens a second is already twelve times faster than it needs to be. Keeping TTFT low means a short uncached part of the prompt: the system prompt, tools and conversation so far are reused from the prefix cache, and only the newest turn is prefilled.
Text-to-speech. Streaming TTS starts producing audio from the first few words. Its time to first audio, 120 ms in the budget’s assumptions, is the last model cost on the path.
Cascaded or speech-to-speech
The alternative is one model that takes audio in and produces audio out, with no text in between. Moshi (Défossez et al., 2024) models the user’s and its own audio streams together and reported about 200 ms of latency in practice; OpenAI reported that GPT-4o could answer audio in as little as 232 ms, 320 ms on average. Removing the STT and TTS hand-offs saves their latency, and the model hears tone and emphasis that a transcript loses. Swapping the three stages for one 350 ms stage brings the same budget to 890 ms, if the system still waits the same 400 ms for silence. API-style speech-to-speech services wait too: OpenAI’s Realtime API, with server-side voice activity detection, waits by default for 500 ms of silence (its API reference, as of October 2026: “Defaults to 500ms”). Vendor figures like GPT-4o’s don’t include that wait, so they aren’t comparable to the budget’s total. Moshi goes further: it is full-duplex, listening and speaking at the same time with no separate turn detection, much as people do.
The cascade keeps advantages that matter in production: each component can be chosen, measured and replaced separately; the transcript is there for logging, tools, guardrails and compliance; and the language model can be any model, including one much stronger than the available speech-to-speech models. Many production systems are still cascades, with speech-to-speech used where latency and naturalness matter most.
Modularity and a transcript against latency and tone.
Barge-in and tool calls
Barge-in is the caller interrupting the agent. The VAD keeps listening while the agent speaks; when the caller starts talking, the system stops TTS playback immediately, cancels whatever the LLM and TTS were still producing, and records how much of the reply the caller actually heard, so the conversation history doesn’t claim the agent said things that were cut off. Not every sound is an interruption: “uh-huh”, a cough or a door shouldn’t stop the agent, so systems require a minimum stretch of speech, or actual words from STT, before cancelling. And echo cancellation has to stop the agent’s own voice, played through the caller’s speaker, from triggering the VAD; it usually runs on the caller’s device (the browser’s built-in canceller, or the phone’s own), which knows exactly what it is playing and when, while the server hears the echo only after codecs and network jitter have altered it.
Tool calls mid-turn (see From Function Calling to the Agent Loop) add their own latency to the critical path. A lookup that takes 800 ms breaks the budget on its own. The usual answers: start speaking a filler clause the LLM produces before the tool call (“Let me check that for you”), run slow tools in parallel with that speech, and prefer tools that answer in tens of milliseconds for anything conversational. And plan for the slow case: a timeout on every tool and on the LLM itself, with a fallback (a smaller, faster model, or a scripted “I’m still looking”) when it fires, because silence on a call reads as failure.
Capacity: 1,000 calls
How many GPUs does the LLM stage need, if the model is served in-house?
Two facts shape the answer. A call doesn’t use the LLM most of the time:
the caller is speaking, or the agent is speaking a reply already
generated. But each call carries a conversation history, and to start the
next reply in tens of milliseconds, its keys and values (the prefix
cache, see Prompt Caching) must be kept somewhere between
turns, or recomputed. A simulation of continuous batching, one iteration
at a time, with each call pinned to one GPU so its cache is where its
requests go. The model, an 8B model on H100s, and every assumption are at
the top of capacity.py:
# How many GPUs does the LLM stage of 1,000 simultaneous voice calls need?
# An idealized simulation of continuous batching ("Serving Metrics and Prefill–Decode Disaggregation"), one iteration
# at a time, for an 8B model in BF16 on H100s (80 GB).
# Calls: each sends the LLM a request every ~20 s on average (the caller
# talks for a few seconds; the agent speaks its ~45-word answer for ~17 s);
# arrivals are random (Poisson), 50 per second in all.
# A request: a 2,000-token conversation so far plus 150 new tokens, and
# 60 output tokens. Each call is pinned to one GPU, so its cache is there.
# An iteration decodes one token for every active request and prefills the
# requests admitted this iteration. Its time is the larger of reading the
# weights and every KV cache it touches at 3.35 TB/s, and the arithmetic at
# 40% of 989 TFLOPS.
# Where the 2,000 tokens of each call's history live between turns decides
# everything (2,000 tokens x 128 KiB = 262 MB per call, 262 GB for 1,000):
# "HBM" : kept in GPU memory, so only the 150 new tokens are prefilled,
# but every call's cache must fit beside the weights;
# "re-prefill": nothing kept, all 2,150 tokens prefilled every turn;
# "CPU" : kept in host memory and copied back over PCIe (~25 GB/s
# effective) when the call's next request arrives.
N, W, BW, FLOPS, HBM = 8e9, 16e9, 3.35e12, 0.4 * 989e12, 80e9
KV, CONTEXT, NEW, OUT, MAX_BATCH, PCIE = 131072, 2000, 150, 60, 200, 25e9
CALLS, TURN_S = 1000, 20.0Its simulate() draws Poisson arrivals and queues each request on its
call’s GPU. Each iteration of a GPU admits waiting requests into a batch of
at most 200 and takes the larger of its memory reads and its arithmetic:
admit = waiting[g][: max(0, MAX_BATCH - len(active[g]))]
waiting[g] = waiting[g][len(admit):]
reads = W + (len(active[g]) + len(admit)) * CONTEXT * KV # weights + every cache touched
dt = max(reads / BW, 2 * N * (len(active[g]) + prefill * len(admit)) / FLOPS)
clock[g] = now + dt
if active[g]:
itl.append(dt)
active[g] = [r - 1 for r in active[g] if r > 1] + [OUT - 1] * len(admit)
ttft += [clock[g] - a for a in admit]It records every request’s time to first token and every iteration’s time, the gap between tokens:
1000 calls, a request every 20 s per call; 2,000 + 150 prompt tokens, 60 out
history kept in GPUs TTFT p50 TTFT p99 ITL p50 ITL p99
HBM 1 1000 calls' caches don't fit in 80 GB
HBM 2 500 calls' caches don't fit in 80 GB
HBM 4 250 calls' caches don't fit in 80 GB
HBM 6 9 ms 16 ms 5.0 ms 6.2 ms
HBM 8 8 ms 16 ms 4.9 ms 6.2 ms
HBM 12 8 ms 14 ms 4.9 ms 6.1 ms
re-prefill 1 overloaded: requests queue without bound
re-prefill 2 overloaded: requests queue without bound
re-prefill 4 8715 ms 22437 ms 20.4 ms 2440.8 ms
re-prefill 6 194 ms 1321 ms 5.3 ms 261.2 ms
re-prefill 8 106 ms 554 ms 5.0 ms 87.9 ms
re-prefill 12 90 ms 349 ms 4.9 ms 87.1 ms
CPU 1 22 ms 41 ms 6.5 ms 13.3 ms
CPU 2 20 ms 31 ms 5.5 ms 6.8 ms
CPU 4 19 ms 27 ms 5.1 ms 6.3 ms
CPU 6 19 ms 27 ms 5.0 ms 6.2 ms
CPU 8 19 ms 26 ms 4.9 ms 6.2 ms
CPU 12 18 ms 24 ms 4.9 ms 6.1 msThe compute is the easy part. With the histories offloaded to CPU memory and copied back over PCIe when a call’s turn comes, a single GPU keeps the time to first token at 22 ms (median) and 41 ms (99th percentile), including the copy. An iteration takes about 6.5 ms, so at 50 requests a second, each decoding 60 tokens, only about 20 requests are decoding at any moment. Speech is slow, so the GPU spends most of each call serving someone else.
Memory is the hard part. 1,000 histories of 2,000 tokens are 262 GB of KV cache. Kept in GPU memory, they need 6 GPUs just to fit, whatever the compute needs. Recomputed every turn, they make each request a 2,150-token prefill: 8 GPUs and the 99th percentile is still over half a second. Kept in CPU memory, they cost host RAM and PCIe bandwidth instead (here about 13 GB/s of copies, half of what one GPU’s link sustains), which is cheaper than either. The design question is not “how many GPUs” but “where does each call’s state live between turns”, and serving systems now support exactly this kind of tiered KV storage.
Where the conversation caches live decides the GPU count; the compute fits on one.
Data: voice-agent-system-design/capacity.pyThis is a model, and the assumptions matter: an 8B model is small for reliable multi-step tool use, and a 70B-class model needs about nine times the compute, more than twice the KV cache per token, and at least two GPUs per copy; copies back from CPU memory are assumed not to queue on the PCIe link; real serving reaches a fraction of the roofline (How a GPU Runs Deep Learning) assumed here; peak traffic is burstier than random; and each tool call or turn-detection model adds requests. A production deployment would multiply the answer for headroom and failures. The STT and TTS models, which run continuously while anyone is speaking, need the same sizing, and often more GPUs than the LLM.
Cost per call-minute follows from the GPU count. Six GPUs at an illustrative $2.50 an hour each serve 60,000 call-minutes an hour, about $0.00025 per call-minute for the LLM. Speech recognition, synthesis and telephony, priced per minute, usually cost far more than that, which is one more reason to size them as carefully.
Observability
Every turn should produce a trace with one span per stage: VAD end of turn, STT final, LLM time to first token (TTFT), first clause, TTS first audio, plus the network legs measured from the client. The latency that matters is the caller’s, from their last word to the agent’s first sound, at the 50th, 95th and 99th percentiles; any stage’s distribution shifting shows up in it. Barge-in rate (how often callers interrupt) and the share of turns where the agent cut the caller off are the quality signals that pure latency misses. Turn detection in particular is a trade-off curve: replaying recorded calls through the detector at several silence thresholds gives the false-cutoff rate against added latency, and the operating point is a product decision.
Where this leads
- The Notebook Assistant, End to End applies the same habit to the series’ running example: one question traced through every stage with its time, and the cost per answer worked out from measured token counts on the same kind of H100 arithmetic.
Takeaways
- Design from the latency budget: about a second, shared by transport, turn detection, STT, the LLM and TTS.
- End-of-turn detection, waiting for silence, is the largest single item in the budget; semantic turn detection cuts it.
- Every stage must stream: partial transcripts, clause-by-clause LLM output, chunked TTS audio.
- WebRTC over UDP, with ICE, STUN and TURN for connectivity, carries the audio; lost packets are cheaper than late ones.
- Speech-to-speech models cut latency and keep tone; cascades keep modularity, transcripts and a free choice of LLM.
- Speech is slow, about three or four tokens a second, so an 8B LLM is busy only a small fraction of each call; for 1,000 calls the binding constraint is where 262 GB of conversation caches live between turns: 6 GPUs to hold them, 8 or more to recompute them (p99 first token still 0.55 s at 8), or one GPU with the caches in CPU memory, in an idealized simulation.
References
Start with Stivers et al. for the human turn-taking gap the budget is built around, then Moshi for a full-duplex speech-to-speech model; the WebRTC and ICE RFCs cover the transport.
- Stivers et al., Universals and cultural variation in turn-taking in conversation (PNAS, 2009): the gap of about 200 ms between speakers, across ten languages, that sets the latency target.
- RFC 8825, Overview: Real-Time Protocols for Browser-Based Applications (WebRTC): the transport the audio travels on.
- RFC 8489, Session Traversal Utilities for NAT (STUN): discovering the public address a client appears from.
- RFC 8656, Traversal Using Relays around NAT (TURN): the relay used when no direct path works.
- RFC 8445, Interactive Connectivity Establishment (ICE): how candidate paths are tried in turn.
- Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper, 2022): the open speech recognizer and its 30-second windows.
- Défossez et al., Moshi: a speech-text foundation model for real-time dialogue (2024): a full-duplex speech-to-speech model and its reported latency of about 200 ms.
- OpenAI, Hello GPT-4o (2024): the launch post’s audio response times, 232 ms at best and 320 ms on average.
- OpenAI, Realtime API reference: client secrets (as of October 2026): the session’s
server_vadsettings, including the 500 ms default forsilence_duration_ms.