Tag: system-design
All the articles with the tag "system-design".
The Notebook Assistant, End to End
Updated:The series' running example, built as one working system: a question-answering assistant for this blog, with a semantic cache, hybrid retrieval, a cross-encoder reranker, a prompt that fences retrieved text as data, a 3B model that cites its passages, and a citation check. One question traced through every stage; a scorecard on 150 questions, 15 the blog doesn't answer, 30 repeats and a planted injection, with the judge checked by hand; the cost per answer, refining the series map's estimate; and what to change next.
System Design for AI: A Real-Time Voice Agent, Millisecond by Millisecond
Updated:A voice agent has about one second between the user stopping and the agent starting, and five components share it. Requirements and a worked latency budget stage by stage, where end-of-turn detection turns out to be the biggest item; transport over WebRTC; streaming speech recognition and synthesis; the LLM's time to first token; cascaded versus speech-to-speech models; barge-in and tool calls mid-turn; and capacity for 1,000 simultaneous calls from a simulation of continuous batching, which needs far fewer GPUs than intuition says because speech is slow.