Posts
All the articles I've posted.
vLLM vs. SGLang vs. TensorRT-LLM: Comparing LLM Serving Architectures
Updated:vLLM, SGLang and TensorRT-LLM now share one design: continuous batching over a paged KV cache with prefix reuse, run from PyTorch with CUDA graphs. What differs is defaults, hardware and ecosystem. Measured: on multi-turn chat with room for the working set, the default prefix caches reused 92.0% of prompt tokens in SGLang, 90.9% in vLLM and 92.0% in TensorRT-LLM; with a smaller cache, TensorRT-LLM's fell furthest. The rest is each engine's source, read at its latest release.
System Design for AI: A Real-Time Voice Agent, Millisecond by Millisecond
Updated:A voice agent has about one second between the user stopping and the agent starting, and five components share it. Requirements and a worked latency budget stage by stage, where end-of-turn detection turns out to be the biggest item; transport over WebRTC; streaming speech recognition and synthesis; the LLM's time to first token; cascaded versus speech-to-speech models; barge-in and tool calls mid-turn; and capacity for 1,000 simultaneous calls from a simulation of continuous batching, which needs far fewer GPUs than intuition says because speech is slow.
Where Models Run: Cloud, Device, and Routing Between Models
Updated:Where a model runs is an optimization over cost, latency, privacy and quality, and routing turns it from one decision per product into one per query. The arithmetic of cloud against device; what fits in a phone's memory; hybrid designs like Apple's; routers, cascades and learned routing; how routing relates to mixture-of-experts; and a cascade measured between a 1.5B and a 3B model on 150 checkable questions, where routing by the kind of question beat routing on the small model's confidence.
World Models, JEPA and Recursive Self-Improvement: Three Bets Beyond the Next Token
Updated:Three research directions for what comes after next-token prediction, each with what is shown and what is hoped. World models learn a simulator and train or plan inside it: a learned pendulum imagined forward drifts 3° in 15 steps and 15° in 100, one reason imagination horizons are short. JEPA predicts embeddings instead of pixels: on a toy world, the naive version collapses to a point, and a stop-gradient, an EMA target with a predictor, or VICReg's variance terms prevent it. And self-improvement: what systems like AlphaEvolve have actually done, separated from what remains speculation.