Posts
All the articles I've posted.
Evaluating LLMs: Metrics, Benchmarks, Judges and Error Bars
Updated:An eval is a measurement instrument, and it needs what any instrument needs: validity, reliability and a known error bar. What to measure and with which metric; benchmarks and contamination; how many questions it takes to tell two models apart (8,077 for 2 points, or as few as 1,570 when the same questions are scored for both); confidence intervals and Cohen's kappa worked; an LLM judge measured on questions with known answers, where a 1.5B judge is sound on questions it can answer, falls back on position on ones it can't, and is fooled by claims of having checked; and arena ratings as Bradley-Terry with their intervals.
FlashAttention: Exact Attention at the Speed of Memory
Updated:Standard attention is slow at long contexts not because of its arithmetic but because it writes an n-by-n matrix of scores to memory and reads it back. FlashAttention computes exactly the same result without ever storing that matrix, using tiling and a softmax computed one block at a time. The online softmax derived and checked to within 1e-15, the memory traffic counted, a fused kernel measured against the three-step version, causal masking, and what FlashAttention 2 and 3, and Flash-Decoding, changed.
From Function Calling to the Agent Loop
Updated:An agent is a model in a loop, with tools and a rule for stopping. What function calling is (a protocol around generated text: the model never runs anything), how a call is emitted and parsed, parallel calls, think-act-observe, stopping, and the transcript as append-only context, with its cost growing quadratically per turn unless cached. Then an agent searching this blog, on 1.5B and 3B models, measured against one retrieval and one answer: whether small models choose to use tools, what forcing a call fixes, and how they cite what they found.
GRPO and Reinforcement Learning from Verifiable Rewards
Updated:When an answer can be checked, the reward model goes away, and sampling a group of answers per question and comparing them within the group gives the advantage without a value network. Verifiable rewards, GRPO's group-relative advantage worked on four samples, its objective and KL estimator, the memory it saves, why groups that all agree teach nothing, and the length bias Dr. GRPO identified, measured on a toy where length never affects the reward.