Posts
All the articles I've posted.
SGLang and RadixAttention: Sharing the KV Cache Across Requests
Updated:Many requests begin with tokens the server has already processed. SGLang keeps every computed prefix in a radix tree and schedules to reuse it. Measured: a cached 2,048-token prefix cut a first token from 2.1 s to 0.15 s; on multi-turn chat, the radix tree and vLLM's hashed blocks reused about the same, 92% and 91% of prompt tokens; the order requests run in moved reuse from 8% to 70% on a shared-prefix workload, at the price of making early arrivals wait; and jump-forward decoding saved a fifth of the forward passes on a JSON schema.
Small and Specialized Models: When Not to Use a Frontier LLM
Updated:Most production model calls don't need a frontier model, and many don't need text at all: they need a decision. What small means in memory and speed, worked out and checked against a measurement; how small models stay capable (data, overtraining, distillation); and an experiment on this series' own question router, where a 23-million-parameter encoder beats a 1.5-billion-parameter language model on accuracy and calibration, and on speed on an Apple M4 laptop, measured with bootstrap intervals and a careful readout of the language model. Then System One decision models, which return typed answers with probabilities instead of text, with a vendor's claims labeled as such.
Cross-Entropy and the Softmax Gradient: One Loss for Every Classifier
Updated:From two routes to four: the softmax, the loss every language model is trained with and where it comes from in information theory, why its gradient is again just prediction minus target, how to compute it without overflow, label smoothing, a four-way router read through its confusion matrix, and what the same loss costs in memory at a 128,256-token vocabulary.
Streaming Tokens to the User: SSE, WebSockets, and the Wait the Reader Feels
Updated:Streaming doesn't make a model generate any faster. It moves the wait the reader feels from the whole answer to the first token. A streaming server on Python's standard library, delivering a real model's tokens three ways: one response, server-sent events, and a WebSocket written by hand. Measured: time to first text, bytes on the wire, reading speed against generation speed, what happens when the user leaves, and how a buffering proxy quietly undoes all of it.