Measuring a Classifier Honestly: Precision, Recall, Thresholds and Calibration

Features and Overfitting built a router that decides whether a reader’s question belongs with the serving posts. Its best version is a bag of words plus a hand-picked keyword count with an L2 penalty. Measured on held-out test questions pooled over 20 random splits, as this post does below, it reaches 88.5% accuracy.

That sounds good. This post takes the number apart. Underneath it is a router that misses 38% of the serving questions it exists to catch. A router that never catches any at all would still score 75%. Along the way we’ll derive the right threshold from what each kind of mistake costs, and see how easily choosing it leaks the test set. We’ll check whether the router’s probabilities can be trusted, and then feed it 20 Vietnamese questions it has never seen a word of. Every number comes from the same 200-question dataset and the same three routers as Features and Overfitting.

Table of contents

The accuracy trap

A quarter of the questions are about serving. A router that answers “not serving” to every question is right on the other three quarters: 75% accuracy from a model that has learned nothing. Our router’s 88.5% has to be read against that floor, not against zero. Put another way, it cuts the error rate from 25% to 11.5%, removing a little over half of the baseline’s mistakes.

The deeper problem is that accuracy counts every mistake the same. A router can make two kinds of mistake, and they don’t cost the same:

• A miss: a serving question routed elsewhere. The reader asks why their server slows down under load and gets posts about backpropagation.
• A false alarm: another question routed to the serving posts. The reader asks about agent memory and gets posts about the KV cache.

Accuracy adds these together. To see them separately we need the four possible outcomes.

Four outcomes

Every prediction lands in one cell of the confusion matrix:

On one split’s 40 test questions, the router scores:

split 0 test: TP 6  FP 2  FN 4  TN 28

From those four numbers, three questions have separate answers.

When the router says “serving”, how often is it right? That’s precision:

Of the serving questions, how many does it catch? That’s recall:

How often is it right at all? That’s accuracy, $(6 + 28)/40 = 0.85$. Of the three, it’s the least useful here.

Precision and recall pull against each other. A router that says “serving” more often catches more serving questions (recall up), but more of its “serving” answers are wrong (precision down). The 28 true negatives, the easy majority, appear in neither formula. That’s exactly why they are the right metrics when the positive class is the rare one that matters.

When one number is needed, the usual choice is the F1 score, the harmonic mean of the two:

Why the harmonic mean and not the ordinary one? It is dominated by the smaller value. Picture a router that says “serving” to just one question out of 50 serving ones, and happens to be right: precision 1.0, recall 0.02. The ordinary mean is 0.51. The F1 is 0.04. The harmonic mean refuses to let one good number hide one terrible one.

All three routers, compared properly

Pool the test predictions over all 20 splits and compare the three routers from Features and Overfitting. router_models.py retrains them on 20 random splits, with that post’s data loader (router_data.py) and trainer (overfit.py), and keeps each router’s logits on the training, validation, test and Vietnamese questions. Its last lines pool them:

RUNS = [fit_all(seed) for seed in range(20)]

def pooled(name, part):
    """Logits and labels for one model and one part, pooled over 20 splits."""
    z = np.concatenate([run[name][part][0] for run in RUNS])
    y = np.concatenate([run[name][part][1] for run in RUNS])
    return z, y

Full script: router_models.py

measure.py computes every metric in this post from those pooled predictions. Its three scoring functions:

def counts(p, y, t=0.5):
    yes = p >= t
    return ((yes & (y == 1)).sum(), (yes & (y == 0)).sum(),    # TP, FP
            (~yes & (y == 1)).sum(), (~yes & (y == 0)).sum())  # FN, TN

def roc_auc(p, y):
    """Probability that a random positive scores above a random negative."""
    ranks = np.empty(len(p)); ranks[np.argsort(p)] = np.arange(1, len(p) + 1)
    pos, neg = y.sum(), len(y) - y.sum()
    return (ranks[y == 1].sum() - pos * (pos + 1) / 2) / (pos * neg)

def average_precision(p, y):
    """Area under the precision-recall curve (mean precision at each hit)."""
    hits = y[np.argsort(-p)]
    precision = np.cumsum(hits) / np.arange(1, len(hits) + 1)
    return (precision * hits).sum() / hits.sum()

Full script: measure.py

The script prints one split’s confusion matrix, the three routers’ scores, a threshold sweep, and the test cost of three ways to pick a threshold:

split 0 test: TP 6  FP 2  FN 4  TN 28
no penalty     acc 0.833  precision 0.795  recall 0.445  F1 0.571  ROC-AUC 0.861  AP 0.714
L2             acc 0.809  precision 0.776  recall 0.330  F1 0.463  ROC-AUC 0.850  AP 0.688
L2 + keywords  acc 0.885  precision 0.891  recall 0.615  F1 0.728  ROC-AUC 0.888  AP 0.790
threshold 0.1: precision 0.563  recall 0.830  cost 197
threshold 0.2: precision 0.750  recall 0.750  cost 150
threshold 0.3: precision 0.789  recall 0.710  cost 154
threshold 0.4: precision 0.830  recall 0.660  cost 163
threshold 0.5: precision 0.891  recall 0.615  cost 169
threshold 0.6: precision 0.918  recall 0.560  cost 186
threshold 0.7: precision 0.934  recall 0.495  cost 209
threshold 0.8: precision 0.931  recall 0.405  cost 244
test cost at 0.5: 169;  theory's 1/3: 156;  tuned on validation: 167;  tuned on test itself: 149

One note on the sample before reading it. The 800 pooled test predictions aren’t 800 independent questions. The dataset has only 200 English questions, so across the 20 splits most questions land in a test set several times, and the predictions come from 20 different trained routers. Treat these numbers as roughly as precise as ~200 cases, not 800.

The first four lines, as a table:

Recall is the story. Even the best router catches only 61.5% of serving questions, so it misses 77 of 200. The L2 router, which Features and Overfitting preferred for its log-loss, has the worst recall: 0.330. Features and Overfitting’s picture explains why. L2 shrinks the weights, which pulls every probability toward the base rate of 25%, so fewer serving questions cross 0.5. That’s no flaw in the router. It’s a sign that 0.5 is the wrong threshold, and nothing about the problem said it should be 0.5.

Metrics that don’t need a threshold

Precision, recall and F1 all depend on where the threshold is. The last two columns don’t, because they measure only how well the router ranks questions.

• ROC-AUC is the probability that a randomly chosen serving question gets a higher score than a randomly chosen other question. That’s 0.888 for the best router. It’s also the area under the ROC curve, which plots the true-positive rate against the false-positive rate as the threshold sweeps from 1 to 0.
• Average precision (AP) is the area under the precision–recall curve, the trade-off the threshold sweep traces out: 0.790.

When positives are rare, prefer the precision–recall view. The ROC curve’s false-positive rate divides by the large number of negatives, so a router can make many false alarms and still look good. Precision divides by the router’s own “yes” answers and shows those false alarms directly. On this data both metrics rank the three routers the same way, and the combined router wins on every column.

The threshold is a cost decision

The router outputs a probability, and the threshold turns it into a decision. The sweep lines show the trade. Lowering the threshold from 0.5 to 0.2 raises recall from 0.615 to 0.750 and lowers precision from 0.891 to 0.750. Which is better depends only on what the two mistakes cost.

Suppose a miss costs twice a false alarm. A reader sent to the wrong posts is worse off than one handed a few extra serving posts alongside the right ones. The cost column is then $2 \times FN + FP$, so which threshold should we use?

The threshold decision theory predicts

There’s a principled answer, and deriving it is short. If the probability $p$ is honest, meaning that among questions given $p = 0.3$, 30% really are serving, then for a single question:

• routing it to serving costs $(1 - p)\,C_{FP}$ in expectation (wrong with probability $1 - p$);
• routing it elsewhere costs $p\,C_{FN}$.

Route to serving when the first is smaller:

With $C_{FN} = 2\,C_{FP}$, the optimal threshold is $t^* = 1/3$. With equal costs it’s $1/2$, the default and the only case where the default is right. With a miss five times as bad, it’s $1/6$.

Choosing it without touching the test set

The last line of output compares three ways to pick the threshold, each scored by its total cost on the test predictions. For each of the 20 splits, the validation tuning tries every threshold from 0.02 to 0.98 in steps of 0.01 on that split’s validation questions, keeps the cheapest, and charges its cost on that split’s test questions:

# Three ways to pick the threshold when a miss costs twice a false alarm,
# each scored on the test predictions.
grid = np.round(np.arange(0.02, 0.985, 0.01), 2)
cost = lambda p, y, t: 2 * ((p < t) & (y == 1)).sum() + ((p >= t) & (y == 0)).sum()
from_theory = from_validation = 0
for run in RUNS:
    pv, yv = sigmoid(run["L2 + keywords"]["val"][0]), run["L2 + keywords"]["val"][1]
    pt, yt = sigmoid(run["L2 + keywords"]["test"][0]), run["L2 + keywords"]["test"][1]
    t_val = grid[int(np.argmin([cost(pv, yv, t) for t in grid]))]
    from_validation += cost(pt, yt, t_val)
    from_theory += cost(pt, yt, 1 / 3)
in_hindsight = min(cost(sigmoid(z), y, t) for t in grid)
print(f"test cost at 0.5: {cost(sigmoid(z), y, 0.5)};  theory's 1/3: {from_theory};"
      f"  tuned on validation: {from_validation};  tuned on test itself: {in_hindsight}")

Full script: measure.py

Three lessons in four rows.

• Theory beat tuning here. The formula, which uses no data at all, cost 156 against 167 for a threshold tuned on validation. Forty validation questions are too few to place a threshold reliably, and the tuned value moves a lot from split to split. When a principled prior exists, a noisy search can do worse than simply using it.
• The last row is a leak. Tuning on the test predictions claims a cost of 149, better than anything achievable honestly. This is Features and Overfitting’s test-set leak again, in a new place: a threshold is a decision, and a decision made on the test set makes the test set optimistic. (The threshold sweep printed above is exploration on the test set, useful for understanding the trade-off, and not a way to choose.)
• Calibration explains a little of the gap. In hindsight the cheapest threshold sits near 0.2, below theory’s 1/3. Part of that is the probabilities not being fully honest. The temperature fix later in this post finds this router slightly overconfident ($\tau$ median 1.24), so an honest 1/3 sits a little below 1/3 on the raw scale. The rest is noise: the sweep above costs 150 at 0.2 and 154 at 0.3, both within 5 of the hindsight minimum of 149.

Try it: drag the threshold and change the cost ratio. The open circle marks the cheapest threshold on these same 800 predictions, which is the hindsight number from the table and optimistic for the same reason.

Can we trust a probability of 0.7?

A router’s probabilities are calibrated if they mean what they say: among all questions scored near 0.7, about 70% are serving. Calibration matters whenever anything downstream uses the number itself rather than just the decision. Examples include the threshold rule above, a fallback that runs a slower search when the router is unsure, and a dashboard reporting “we expect 40 serving questions today”. A model can rank well and be badly calibrated, or be calibrated and rank poorly. They are different properties.

To check calibration, group predictions into bins by their probability and compare each bin’s average prediction to how often its questions were actually serving. That comparison is a reliability diagram, written out here as a table:

def reliability(p, y, bins=5):
    """Mean predicted vs observed rate per probability bin, and the ECE."""
    edges = np.linspace(0, 1, bins + 1)
    which = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)
    rows, ece = [], 0.0
    for k in range(bins):
        m = which == k
        if m.any():
            rows.append((edges[k], edges[k + 1], m.sum(), p[m].mean(), y[m].mean()))
            ece += m.mean() * abs(p[m].mean() - y[m].mean())
    return rows, ece

for name in ["no penalty", "L2 + keywords"]:
    z, y = pooled(name, "test")
    rows, ece = reliability(sigmoid(z), y)
    print(f"{name}: ECE {ece:.3f}, Brier {((sigmoid(z) - y) ** 2).mean():.3f}")
    for lo, hi, n, conf, rate in rows:
        print(f"   p in [{lo:.1f}, {hi:.1f}): {n:3d} predictions, "
              f"mean p {conf:.2f}, actually serving {rate:.2f}")

# How big is ECE for probabilities that are calibrated by construction?
p = sigmoid(pooled("L2 + keywords", "test")[0])
rng = np.random.default_rng(0)
floor = [reliability(p, (rng.random(len(p)) < p).astype(float))[1] for _ in range(500)]
print(f"ECE of perfectly calibrated labels: mean {np.mean(floor):.3f},"
      f" 95th percentile {np.percentile(floor, 95):.3f}")

Full script: calibration.py

no penalty: ECE 0.109, Brier 0.134
   p in [0.0, 0.2): 632 predictions, mean p 0.03, actually serving 0.12
   p in [0.2, 0.4):  40 predictions, mean p 0.28, actually serving 0.62
   p in [0.4, 0.6):  34 predictions, mean p 0.50, actually serving 0.79
   p in [0.6, 0.8):  33 predictions, mean p 0.72, actually serving 0.64
   p in [0.8, 1.0):  61 predictions, mean p 0.92, actually serving 0.84
L2 + keywords: ECE 0.037, Brier 0.094
   p in [0.0, 0.2): 600 predictions, mean p 0.06, actually serving 0.08
   p in [0.2, 0.4):  41 predictions, mean p 0.31, actually serving 0.44
   p in [0.4, 0.6):  37 predictions, mean p 0.49, actually serving 0.54
   p in [0.6, 0.8):  35 predictions, mean p 0.71, actually serving 0.89
   p in [0.8, 1.0):  87 predictions, mean p 0.94, actually serving 0.93
ECE of perfectly calibrated labels: mean 0.016, 95th percentile 0.027
no penalty     tau median 1.99 (range 1.03-3.08):  ECE 0.109 -> 0.031
L2             tau median 1.14 (range 0.73-1.67):  ECE 0.061 -> 0.043
L2 + keywords  tau median 1.24 (range 0.28-2.03):  ECE 0.037 -> 0.027

Read the unpenalized router’s rows from top to bottom and a pattern appears.

• The bottom is too low. It put 632 predictions below 0.2 with an average of 0.03, and 12% of them were serving questions. The next bin is worse: scored 0.28, serving 62% of the time.
• The top is too high. Scored 0.92, serving 84% of the time.

Too low at the bottom and too high at the top is what overconfidence looks like: every probability is pushed toward 0 or 1, further than the evidence supports. It’s the pattern Features and Overfitting predicted. On separable training data its weights grew large, and would have kept growing without limit, and large weights mean extreme probabilities. The combined router’s bins track the diagonal far more closely.

The expected calibration error (ECE) turns the table into one number: the gap between predicted and observed rates in each bin, weighted by how many predictions fall in the bin:

For the unpenalized router’s first bin that term is $\frac{632}{800} \times |0.03 - 0.12| = 0.07$, most of its total of 0.109. The combined router’s ECE is 0.037.

Three cautions come with ECE.

• It depends on the bins. Here three quarters of the predictions sit in the first bin, so the other four bins rest on 35–87 predictions each. The usual remedy is equal-mass bins, each holding the same number of predictions.
• It is biased upward by noise. The line after the tables resamples the labels from the router’s own probabilities, so the probabilities are perfectly calibrated by construction. ECE still averages 0.016, and exceeds 0.027 one time in twenty. At this sample size an ECE of 0.027 is at the edge of what noise alone produces, and 0.031 is just past it.
• A model can be perfectly calibrated and useless. Predicting the base rate, 0.25, for every question has an ECE of zero.

So calibration is always reported next to a measure of ranking, such as the AUCs above. The Brier score, the mean squared error between probability and outcome, blends both in one number. It’s 0.134 unpenalized and 0.094 combined, against 0.1875 for always predicting 0.25. (Predicting Chess Outcomes uses it for the same reason.)

Fixing calibration with one number

Calibration can often be repaired after training without touching the router’s weights. Temperature scaling divides every logit by one constant $\tau$ before the sigmoid:

• $\tau > 1$ pulls every probability toward 0.5, curing overconfidence.
• $\tau < 1$ sharpens them.
• Ranking is untouched. Dividing by a positive constant never changes which of two questions scores higher, so ROC-AUC and average precision are exactly unchanged. Only the meaning of the numbers moves.

$\tau$ is fit on the validation set by minimizing log-loss. It’s one parameter, so even 40 questions can estimate it roughly. The loop at the end of calibration.py:

# Temperature scaling: one number tau, fit on each split's validation set.
grid = np.exp(np.linspace(np.log(0.2), np.log(5), 400))
for name in ["no penalty", "L2", "L2 + keywords"]:
    before, after, taus = [], [], []
    for run in RUNS:
        zv, yv = run[name]["val"]
        tau = grid[np.argmin([log_loss(sigmoid(zv / t), yv) for t in grid])]
        zt, yt = run[name]["test"]
        before.append(sigmoid(zt)); after.append(sigmoid(zt / tau)); taus.append(tau)
    y = pooled(name, "test")[1]
    e0, e1 = reliability(np.concatenate(before), y)[1], reliability(np.concatenate(after), y)[1]
    print(f"{name:14s} tau median {np.median(taus):.2f} (range {min(taus):.2f}-{max(taus):.2f}):"
          f"  ECE {e0:.3f} -> {e1:.3f}")

Full script: calibration.py

The last three lines of its output show the results.

• The unpenalized router wants $\tau \approx 2$, halving its logits. That cuts its ECE from 0.109 to 0.031, just above the 0.027 that noise alone exceeds one time in twenty.
• The routers L2 already calibrated gain little: 0.061 → 0.043 and 0.037 → 0.027.
• How reliable is the fit? The per-split ranges show it. The median $\tau$ is stable, but a single split’s fitted value can be anywhere from 0.28 to 2.03 for the combined router. Forty questions pin down one parameter only loosely.

The pattern holds far beyond this router. A model trained to push its training loss to zero is systematically overconfident, and one temperature, fit on held-out data, fixes much of it. Large neural networks behave the same way, which is where temperature scaling was popularized. (Its two-parameter generalization, a learned scale and shift on the logit, is Platt scaling.)

The test set that was missing

Everything so far measured the router on held-out questions from the same distribution it trained on: English questions written by the same person. The Notebook Assistant will also get Vietnamese questions. The router has never seen one, so its training vocabulary has no Vietnamese words at all.

The 20 Vietnamese questions are translations of English questions from the dataset, so the content is held fixed and only the language changes. That makes them a clean probe of the language. One other thing changes, though: 10 of the 20 are serving questions, against a quarter in English. So the class balance shifts too, which by itself moves precision and the best threshold.

import numpy as np
from overfit import sigmoid
from router_data import split, featurize, tokens, vietnamese
from router_models import RUNS, pooled

# How much of each Vietnamese question does the English vocabulary see?
(train_set, *_), vocab = featurize(*split(seed=0))
z, y = RUNS[0]["L2 + keywords"]["vi"]
for r, p in zip(vietnamese, sigmoid(z)):
    if r["route"] == "serving":
        seen = [t for t in tokens(r["question"]) if t in vocab]
        print(f"p={p:.2f} {'caught' if p >= 0.5 else 'MISSED'}  seen={seen}  {r['question']}")

# Over all 20 splits: how often is each serving question caught, and how
# often is a non-serving question wrongly sent to serving?
for name in ["no penalty", "L2", "L2 + keywords"]:
    caught = np.mean([sigmoid(run[name]["vi"][0]) >= 0.5 for run in RUNS], axis=0)
    z, y = pooled(name, "vi")
    recall = ((sigmoid(z) >= 0.5) & (y == 1)).sum() / (y == 1).sum()
    false_alarm = ((sigmoid(z) >= 0.5) & (y == 0)).sum() / (y == 0).sum()
    print(f"{name:14s} recall {recall:.3f}  false-alarm rate {false_alarm:.3f}"
          f"  per-question catch rate: {' '.join(f'{c:.2f}' for c in caught[:10])}")

Full script: vietnamese.py

p=0.94 caught  seen=['kv', 'cache', 'a', '70b']  KV cache của mô hình 70B lớn cỡ nào?
p=0.98 caught  seen=['i', 'decode', 'prefill', 'i', 'token']  Tại sao decode chậm hơn prefill trên mỗi token?
p=0.07 MISSED  seen=['i', 'd']  Làm sao phục vụ một mô hình cho nhiều người dùng cùng lúc?
p=0.17 MISSED  seen=['token']  Vì sao token đầu tiên ra chậm như vậy?
p=0.08 MISSED  seen=['a']  Lượng tử hóa ảnh hưởng thế nào đến độ trễ?
p=0.39 MISSED  seen=['gpu', 'i']  Một GPU H100 chạy được bao nhiêu yêu cầu đồng thời?
p=0.59 caught  seen=['continuous', 'batching']  Continuous batching tăng thông lượng ra sao?
p=0.89 caught  seen=['vllm', 'd', 'paged', 'attention']  vLLM có dùng paged attention không?
p=0.07 MISSED  seen=['i', 'i', 'i', 'd']  Tại sao máy chủ chậm lại khi có thêm người dùng?
p=0.31 MISSED  seen=['speculative', 'decoding', 'i']  Speculative decoding tiết kiệm được bao nhiêu thời gian?
no penalty     recall 0.735  false-alarm rate 0.065  per-question catch rate: 0.85 1.00 0.50 1.00 0.00 0.95 0.75 0.80 0.50 1.00
L2             recall 0.505  false-alarm rate 0.000  per-question catch rate: 0.65 1.00 0.05 0.65 0.00 0.80 0.55 0.55 0.05 0.75
L2 + keywords  recall 0.540  false-alarm rate 0.000  per-question catch rate: 1.00 1.00 0.00 0.00 0.00 0.95 1.00 1.00 0.00 0.45

The per-question catch rates are in the same order as the ten questions above. Four findings, each a lesson in its own right:

• English terms rescue part of it. Vietnamese technical writing keeps English terms. “KV cache”, “decode”, “prefill”, “vLLM”, “paged attention” and “continuous batching” appear as-is, and across the 20 splits the combined router catches those questions every time. Not every English term is enough, though. “token” and “speculative decoding” are English too. The question with “token” is missed in every split, and the one with “speculative decoding” in more than half. (The “GPU” question, missed in the split shown above, is caught in 19 of 20.)
• The questions written entirely in Vietnamese are missed by the combined router. “Làm sao phục vụ một mô hình cho nhiều người dùng cùng lúc?” (How do I serve a model to many users at once?) and “Tại sao máy chủ chậm lại khi có thêm người dùng?” (Why does the server slow down as more users join?) are among the most central serving questions in the dataset. It scores them 0.07 in the split above, and misses them in all 20 splits, because every content word in them is unknown.
• The tokenizer shreds Vietnamese. Look at the seen lists. The rule [a-z0-9]+ only knows unaccented Latin letters, so “Tại” becomes the fragments t and i, and “dùng” becomes d and ng. Some fragments match English vocabulary by accident, such as i from “How do I” and d from “d_k”, and a is simply the English article. A tokenizer is a model of what text looks like, and this one’s model is English. This is the case Byte-Pair Encoding makes for starting from bytes: every script is covered, and no character can fall outside the vocabulary.
• The overconfident router catches more, for the wrong reasons. The unpenalized router’s recall is 0.735, well above the combined router’s 0.540, and it even catches the two fully Vietnamese questions in half the splits. It does that through its large weights on the shredded fragments i and d, and on “token” and “speculative decoding”. It’s the right answer for the wrong reason, and it comes with false alarms: it sends 6.5% of the non-serving Vietnamese questions to serving, where both L2 routers send none. Recall alone would have made it look better.

Taken together, better calibration on the training distribution didn’t make the router better under distribution shift, the change from the data a model was built on to the data it actually meets. The English test set predicted neither number. The only way to know how a model behaves on the traffic it will get is to test it on that traffic, and to look at more than one metric while doing so.

What this router still can’t do

• Use word order or context. A bag of words treats “why is training slower than serving?” and “why is serving slower than training?” as the same question, because order is gone. It can’t express that a word means one thing next to one neighbor and another next to a different one.
• Read Vietnamese. It needs features where “phục vụ” and “serve” land in the same place. That’s what multilingual embeddings are for (Embeddings and Contrastive Learning).
• Route to more than two places. The real router has four destinations. That needs the softmax generalization of the sigmoid, and the gradient that comes with it.

Where this leads

• Backpropagation, by Hand and Cross-Entropy and the Softmax Gradient replace the linear router with a neural network. The first derives backpropagation, and the second derives cross-entropy for many classes, which routes to all four parts of the blog. Its confusion matrix becomes 4×4, and precision and recall are computed per route.
• The same measurements, bigger stakes. Precision and recall reappear in retrieval (did we fetch the passage that holds the answer?) and in guardrails (how many harmless requests does a safety filter block?). Calibration reappears in evaluation, where an LLM grading other models’ answers has to have its own scores checked against human judgment. Each of those posts links back here instead of re-deriving the metrics.
• Temperature, again. The $\sigma(z/\tau)$ of this post becomes $\operatorname{softmax}(z/\tau)$ in sampling (Decoding) and distillation (Knowledge Distillation).

Takeaways

• Read accuracy against the base rate. “Always no” scores 75% here, so 88.5% means removing a little over half the baseline’s mistakes, not being right 88.5% of the time from zero.
• Precision asks how often “yes” is right. Recall asks how many true “yes” cases were caught. The true negatives, the easy majority, appear in neither. F1, their harmonic mean, can’t hide one bad number behind a good one. ROC-AUC and average precision measure ranking and need no threshold; prefer the precision–recall view when positives are rare.
• The threshold is a decision about costs, not a property of the model. For calibrated probabilities the optimal threshold is $C_{FP}/(C_{FP} + C_{FN})$: 1/3 when a miss costs twice a false alarm. That data-free choice (test cost 156) beat one tuned on 40 validation questions (167). Tuning on the test set claimed 149, which is a leak.
• Calibration is whether probabilities mean what they say. The unpenalized router was overconfident, too low at the bottom and too high at the top, with an ECE of 0.109. One temperature fit on validation data cut it to 0.031 without changing a single ranking. Remember ECE’s noise floor, 0.016 here even for perfect calibration.
• Pooled predictions from reused splits are fewer independent cases than they look: 800 predictions, about 200 distinct questions.
• Distribution shift can’t be predicted from the test set. On Vietnamese, English terms rescued part of the recall, and the fully Vietnamese questions were missed by the calibrated router. The overconfident router caught more, through shredded-token accidents and with false alarms.

References

Start with Fawcett for ROC analysis and Davis and Goadrich for when to prefer precision–recall. Then read Guo et al. on calibration: it is short, and it is the paper behind temperature scaling.

• van Rijsbergen, Information Retrieval (2nd ed., 1979): precision, recall and the effectiveness measure that became F1 (chapter 7).
• Fawcett, An Introduction to ROC Analysis (Pattern Recognition Letters, 2006): ROC curves and AUC as the probability of ranking a positive above a negative.
• Davis and Goadrich, The Relationship Between Precision-Recall and ROC Curves (ICML, 2006): why precision–recall is the right view when positives are rare.
• Elkan, The Foundations of Cost-Sensitive Learning (IJCAI, 2001): the cost-based optimal threshold.
• Brier, Verification of Forecasts Expressed in Terms of Probability (Monthly Weather Review, 1950): the Brier score.
• Platt, Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods (in Advances in Large Margin Classifiers, MIT Press, 1999): Platt scaling.
• Niculescu-Mizil and Caruana, Predicting Good Probabilities with Supervised Learning (ICML, 2005): reliability diagrams, and which models are over- or under-confident.
• Naeini, Cooper and Hauskrecht, Obtaining Well Calibrated Probabilities Using Bayesian Binning (AAAI, 2015): expected calibration error.
• Guo, Pleiss, Sun and Weinberger, On Calibration of Modern Neural Networks (ICML, 2017): temperature scaling, and why deep networks are overconfident.
• Quiñonero-Candela, Sugiyama, Schwaighofer and Lawrence (eds.), Dataset Shift in Machine Learning (MIT Press, 2009): the kinds of distribution shift, including a change in the class balance.