Tag: capacity-planning
All the articles with the tag "capacity-planning".
System Design for AI: A Real-Time Voice Agent, Millisecond by Millisecond
Updated:A voice agent has about one second between the user stopping and the agent starting, and five components share it. Requirements and a worked latency budget stage by stage, where end-of-turn detection turns out to be the biggest item; transport over WebRTC; streaming speech recognition and synthesis; the LLM's time to first token; cascaded versus speech-to-speech models; barge-in and tool calls mid-turn; and capacity for 1,000 simultaneous calls from a simulation of continuous batching, which needs far fewer GPUs than intuition says because speech is slow.