Posts
All the articles I've posted.
Reasoning Models and Test-Time Compute
Updated:A reasoning model spends generated tokens as compute: trained with verifiable rewards, it learns to think before answering, and gets better the longer it thinks. What test-time compute means, how RL produces long chains and distillation passes them to small models, thinking budgets, and what 1,000, 10,000 and 50,000 thinking tokens cost in KV cache and time on two real models. Then a 1.5B model on checkable word problems, two seeds: answering directly, reasoning step by step, and voting over up to 16 chains, with the tokens each one costs, then on harder eight-step problems, where voting stops helping and a verifier could still gain.
Recursive Language Models: The Prompt as an Environment
Updated:A recursive language model never reads its long input: the input sits in a Python REPL as a variable, and the model writes code to inspect it and calls itself on the pieces. How the idea works, why its three design choices matter, what the paper measured, and a miniature built and run on this blog's whole archive with a 3B model, against direct reading and a fixed map-over-chunks program.
The Residual Stream: How a Transformer Turns Tokens Into Predictions
Updated:How the pieces of a Transformer fit into one model: a token becomes a vector, every attention and MLP sublayer reads it and adds to it, and the result becomes a prediction. Measured in Qwen2.5-1.5B-Instruct: the stream grows from 0.03 to 8.0 in size over 28 blocks, each sublayer adds about a fifth to a half of what it reads, the final prediction appears only in the last few blocks, and on one passage, removing any one middle block costs under a third of a nat.
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.