Tag: evaluation
All the articles with the tag "evaluation".
Predicting Chess Outcomes: A Disciplined ML Walkthrough
Updated:Predicting who wins a Lichess blitz game from its opening, rebuilt end to end on 100,000 games. Against a rating curve refitted to the data, the opening adds nothing at move 3 and a Brier gain of about 0.002 at move 10, on all seven splits tried; a Stockfish evaluation makes it 0.0084. The method is the point: an audited dataset, a leakage contract enforced by tests, a baseline worth beating, and diagnostics before features.
Evaluating LLMs: Metrics, Benchmarks, Judges and Error Bars
Updated:An eval is a measurement instrument, and it needs what any instrument needs: validity, reliability and a known error bar. What to measure and with which metric; benchmarks and contamination; how many questions it takes to tell two models apart (8,077 for 2 points, or as few as 1,570 when the same questions are scored for both); confidence intervals and Cohen's kappa worked; an LLM judge measured on questions with known answers, where a 1.5B judge is sound on questions it can answer, falls back on position on ones it can't, and is fooled by claims of having checked; and arena ratings as Bradley-Terry with their intervals.
Long Context, and Where It Breaks: Lost in the Middle
Updated:A model that accepts a long prompt doesn't use all of it equally. The lost-in-the-middle result reproduced on a 1.5B model: a key-value lookup that's perfect at 2,000 tokens and loses things in the middle by 8,000. Why the needle-in-a-haystack test misses it, what RULER and Context Rot measure instead, and the causes: softmax dilution worked out, attention sinks measured, RoPE's reach, and training on short documents. Then what to do about it: put things where they're read, retrieve less, compact, train longer.
Measuring a Classifier Honestly: Precision, Recall, Thresholds and Calibration
Updated:An 88.5% accurate router that misses 38% of the questions it exists to catch. Why accuracy hides that, how precision, recall and ranking metrics expose it, how to choose a threshold from what each mistake costs without leaking the test set, whether the router's probabilities mean what they say, and what happens when Vietnamese questions arrive.