Tag: latency
All the articles with the tag "latency".
Speculative Decoding, Derived: Draft Models, N-grams, Medusa and EAGLE
Updated:Decoding is memory-bound, so checking k guessed tokens costs about as much as generating one. A rejection-sampling rule makes the output exactly the target model's distribution whatever the guesser, proved and checked by sampling. The expected tokens per step derived and checked, the speedup with the draft's cost, and an interactive simulator; draft models, prompt lookup, Medusa heads and EAGLE; and acceptance measured on real models for two kinds of task.
Prompt Caching: Paying Once for a Prefix
Updated:API prompt caching is the KV cache and prefix sharing sold as a price. What is actually cached, why only an exact prefix can hit, breakpoints and lifetimes, the write and read prices and the break-even worked from published multipliers, the time a cached prefix saves measured on a 1.5B model, the gigabytes a provider holds for one, how to order a prompt, and how agents break their own cache.
Small and Specialized Models: When Not to Use a Frontier LLM
Updated:Most production model calls don't need a frontier model, and many don't need text at all: they need a decision. What small means in memory and speed, worked out and checked against a measurement; how small models stay capable (data, overtraining, distillation); and an experiment on this series' own question router, where a 23-million-parameter encoder beats a 1.5-billion-parameter language model on accuracy and calibration, and on speed on an Apple M4 laptop, measured with bootstrap intervals and a careful readout of the language model. Then System One decision models, which return typed answers with probabilities instead of text, with a vendor's claims labeled as such.
Streaming Tokens to the User: SSE, WebSockets, and the Wait the Reader Feels
Updated:Streaming doesn't make a model generate any faster. It moves the wait the reader feels from the whole answer to the first token. A streaming server on Python's standard library, delivering a real model's tokens three ways: one response, server-sent events, and a WebSocket written by hand. Measured: time to first text, bytes on the wire, reading speed against generation speed, what happens when the user leaves, and how a buffering proxy quietly undoes all of it.