Tag: calibration
All the articles with the tag "calibration".
Measuring a Classifier Honestly: Precision, Recall, Thresholds and Calibration
Updated:An 88.5% accurate router that misses 38% of the questions it exists to catch. Why accuracy hides that, how precision, recall and ranking metrics expose it, how to choose a threshold from what each mistake costs without leaking the test set, whether the router's probabilities mean what they say, and what happens when Vietnamese questions arrive.
Small and Specialized Models: When Not to Use a Frontier LLM
Updated:Most production model calls don't need a frontier model, and many don't need text at all: they need a decision. What small means in memory and speed, worked out and checked against a measurement; how small models stay capable (data, overtraining, distillation); and an experiment on this series' own question router, where a 23-million-parameter encoder beats a 1.5-billion-parameter language model on accuracy and calibration, and on speed on an Apple M4 laptop, measured with bootstrap intervals and a careful readout of the language model. Then System One decision models, which return typed answers with probabilities instead of text, with a vendor's claims labeled as such.