Posts
All the articles I've posted.
Long Context, and Where It Breaks: Lost in the Middle
Updated:A model that accepts a long prompt doesn't use all of it equally. The lost-in-the-middle result reproduced on a 1.5B model: a key-value lookup that's perfect at 2,000 tokens and loses things in the middle by 8,000. Why the needle-in-a-haystack test misses it, what RULER and Context Rot measure instead, and the causes: softmax dilution worked out, attention sinks measured, RoPE's reach, and training on short documents. Then what to do about it: put things where they're read, retrieve less, compact, train longer.
Measuring a Classifier Honestly: Precision, Recall, Thresholds and Calibration
Updated:An 88.5% accurate router that misses 38% of the questions it exists to catch. Why accuracy hides that, how precision, recall and ranking metrics expose it, how to choose a threshold from what each mistake costs without leaking the test set, whether the router's probabilities mean what they say, and what happens when Vietnamese questions arrive.
The Notebook Assistant, End to End
Updated:The series' running example, built as one working system: a question-answering assistant for this blog, with a semantic cache, hybrid retrieval, a cross-encoder reranker, a prompt that fences retrieved text as data, a 3B model that cites its passages, and a citation check. One question traced through every stage; a scorecard on 150 questions, 15 the blog doesn't answer, 30 repeats and a planted injection, with the judge checked by hand; the cost per answer, refining the series map's estimate; and what to change next.
Paged Attention: Managing the KV Cache Like Virtual Memory
Updated:A server can't know how long an answer will be, so a KV cache stored in one contiguous piece must be reserved for the worst case. Measured on requests from the two datasets the vLLM paper used, ShareGPT and Alpaca, that leaves 3 to 63% of the reservation holding real tokens; 16-token blocks with a block table, as an operating system pages memory, hold 90 to 98%. A paged cache built for a real model and checked token for token, attention over scattered blocks as a GPU kernel, the block size, sharing with copy-on-write, prefix caching by block hashes, and what the paper's throughput numbers do and don't say.