Evaluating LLMs: Metrics, Benchmarks, Judges and Error Bars

Every decision in this series that compared two things (two retrievers, two fine-tunes, two decoding settings) rested on an evaluation, and most of them came with a warning about seeds or sample size. This post is about evaluation itself. The core idea is borrowed from measurement in any science: an eval is an instrument. It has to measure the thing that matters for the decision (validity), give the same answer when nothing changed (reliability), and come with an error bar, or its readings can’t be compared. A language-model judge is an instrument too, and it has to be calibrated like one.

Table of contents

What to measure

Start from the decision the eval serves. “Is model B better than model A?” is too vague to measure. “On our customers’ support questions, does B give a correct, complete answer more often, without getting slower?” can be. That framing names the data (the product’s own traffic, not a public benchmark), the property (correctness, completeness) and the constraint (latency).

The metric then depends on what an answer looks like:

• Exact or programmatic checks, when the answer is checkable: a multiple-choice letter, a number, code that must pass tests, JSON that must validate. These are the cheapest and most reliable metrics, and GRPO and Reinforcement Learning from Verifiable Rewards used them as rewards.
• Reference-based overlap, such as BLEU or ROUGE, which count n-grams shared with a reference answer. They were built for translation and summarization, where outputs are close to a reference; for open-ended answers they reward wording, not correctness, and a correct answer phrased differently scores low.
• Embedding similarity to a reference (Embeddings and Contrastive Learning), which tolerates rewording but still can’t tell a confident wrong answer that shares the reference’s vocabulary from a right one.
• Human judgment, the reference standard for open-ended quality, and slow and expensive.
• An LLM judge, cheaper than people and scalable, and biased in ways measured below.

Perplexity (What “Generative” Means) belongs on this list for base models: it measures how well a model predicts text, which tracks capability but says nothing about whether answers are helpful.

Benchmarks and contamination

Public benchmarks make models comparable: MMLU (Hendrycks et al., 2020) for knowledge across 57 subjects, HELM (Liang et al., 2022) for a broad, standardized battery, SWE-bench for coding agents, and many more. They share two weaknesses. They saturate: once frontier models score in the high 90s, a benchmark stops separating them. And they leak: a benchmark’s questions published on the web end up in later models’ training data, so a high score may reflect memory rather than ability. Checks for contamination look for verbatim overlap between benchmark and training data, or for a model doing much better on a benchmark’s public questions than on freshly written questions of the same kind. The practical defense is a private eval: questions written for the task at hand, kept out of anything public, and refreshed.

How many questions?

Suppose two models score around 70%, and the eval has to detect a 2-point difference. The number of questions this takes is a power calculation, the same one a clinical trial uses. From sample_size.py:

z = NormalDist().inv_cdf
za, zb = z(0.975), z(0.80)
p1, p2 = 0.70, 0.72
# Unpaired: the two models answer different random questions.
n_unpaired = (za + zb) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2)) / (p2 - p1) ** 2
print(f"unpaired (different questions): {math.ceil(n_unpaired):,} questions per model")
# Paired: both answer the same questions, and only the questions they
# disagree on carry information (McNemar). If they disagree on a share d of
# questions, the per-question difference has variance d - (p2 - p1)^2,
# about d. (Independent errors would make them disagree on
# p1 (1 - p2) + p2 (1 - p1) = 41%: the unpaired case.)
for d in (0.30, 0.15, 0.08):
    n = (za + zb) ** 2 * d / (p2 - p1) ** 2
    print(f"paired, models disagree on {d:.0%} of questions: {math.ceil(n):,} questions")

Full script: sample_size.py

The same script also prints the Wilson intervals, the interval on a difference, and the Cohen’s kappa worked in the kappa section below:

unpaired (different questions): 8,077 questions per model
paired, models disagree on 30% of questions: 5,887 questions
paired, models disagree on 15% of questions: 2,944 questions
paired, models disagree on 8% of questions: 1,570 questions

70 right of 100: accuracy 70%, 95% interval 60.4% to 78.1% (about ±8.8 points)
700 right of 1,000: accuracy 70%, 95% interval 67.1% to 72.8% (about ±2.8 points)
7,000 right of 10,000: accuracy 70%, 95% interval 69.1% to 70.9% (about ±0.9 points)

500 questions, 95% interval on the difference in accuracy: unpaired ±5.6 points, paired (disagreeing on 15%) ±3.4 points

Cohen's kappa: observed agreement 0.85, chance agreement 0.55 x 0.50 + 0.45 x 0.50 = 0.500, kappa = (0.85 - 0.500) / (1 - 0.500) = 0.70

With different questions for each model, about 8,000 each. Few evals are that large, which is why a 2-point gap on a 500-question benchmark is usually noise: at 500 questions, the 95% interval on the difference between two models is about ±5.6 points unpaired (second-to-last line of the output).

Scoring both models on the same questions helps a great deal, because it cancels the question-to-question variation. Only the questions where the models disagree carry information about which is better (the logic of McNemar’s test), so the sample size scales with how often they disagree: 1,570 questions if they disagree on 8% of them, 5,887 at 30%. (Two models with independent errors at 70% and 72% would disagree on 41% of questions: the unpaired case is the paired one with no shared difficulty.) At 500 questions and 15% disagreement, the interval on the difference shrinks from ±5.6 to ±3.4 points. Always compare models on the same items, and report the paired difference with its interval: if model B is right and A wrong on $b$ questions, and the reverse on $c$, the difference is $(b - c)/n$, with a 95% interval of about $\pm 1.96\sqrt{b + c - (b - c)^2/n}\,/\,n$.

Three things make real intervals wider than these formulas say. Questions are often clustered: several built from one passage or one template, which fail together, so the effective number of independent questions is smaller; clustered standard errors account for it (Miller, 2024, shows they can be several times the naive ones). Sampling at a temperature above zero adds run-to-run variance: score each question several times, or decode greedily, and include that variance in the error bar. And comparing many variants or many slices of an eval guarantees some false “wins” at 5% each: correct for the number of comparisons, or confirm a winner on fresh data.

The middle block is the interval for a single accuracy. At 100 questions, 70% means “somewhere from 60% to 78%”; at 1,000, ±2.8 points; at 10,000, under a point. The Wilson interval used here behaves better than the textbook $p \pm 1.96\sqrt{p(1-p)/n}$ near 0% and 100% and at small $n$.

Human evaluation and agreement

When people judge answers, they disagree, and the first thing to measure is how much. Raw agreement is misleading, because two raters who label most answers “good” agree often by chance. Cohen’s $\kappa$ corrects for it:

where $p_o$ is the observed agreement and $p_e$ the agreement expected if each rater labeled at random with their own overall rates. In the worked example here, two raters, A and B, label the same 100 answers good or bad:

They agree on 45 + 40 = 85 of the 100 answers, but since each says “good” about half the time (A on 55, B on 50), chance alone would give 0.55 × 0.50 + 0.45 × 0.50 = 50%, and $\kappa = 0.70$: agreement well beyond chance, short of perfect. A common rough reading, from Landis and Koch (1977), is that $\kappa$ above about 0.6 is substantial and above 0.8 near-perfect, but the point is less the label than the use: if raters reach only $\kappa = 0.4$ on a rubric, the rubric is ambiguous, and no model, human or LLM, can be scored reliably against it until it’s fixed. Cohen’s $\kappa$ is for two raters; with more, Fleiss’ $\kappa$ or Krippendorff’s $\alpha$ do the same job. The same statistic calibrates an LLM judge: compute $\kappa$ between the judge and human raters on a labeled sample, and trust the judge only if it agrees with people about as well as people agree with each other.

LLM-as-judge, measured

Using a strong model to grade answers (Zheng et al., 2023) made open-ended evaluation cheap. It comes in two forms: pointwise, where the judge scores one answer against a rubric (G-Eval, Liu et al., 2023, has it reason through rubric steps first, then averages the possible scores weighted by the probabilities the judge gives each score token), and pairwise, where it picks the better of two, usually with a tie option. Pairwise is easier for a judge and maps directly onto arena-style ratings; it is also where the best-known biases show up.

To measure them, a judge needs questions whose right answer is known. A hundred two-digit multiplications, each wrong answer off by a multiple of ten, and twenty capital cities, with each country’s best-known other city as the wrong answer, judged by Qwen2.5-1.5B-Instruct, reading its preference between “A” and “B” from the next-token probabilities (the same trick G-Eval uses). The judge in judge.py:

@torch.no_grad()
def judge(question, first, second, labels=("A", "B")):
    """Returns 1 if the judge prefers the first-shown answer, else 2.
    labels: the letters printed before the first and second answers."""
    prompt = (f"Question: {question}\n\nAnswer {labels[0]}: {first}\n\nAnswer {labels[1]}: {second}\n\n"
              "Which answer is correct? Reply with a single letter, A or B.")
    ids = tok.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True,
                                  return_tensors="pt", return_dict=True)["input_ids"].to(DEVICE)
    logits = model(ids).logits[0, -1]
    first_letter = A if labels[0] == "A" else B
    second_letter = B if labels[0] == "A" else A
    return 1 if logits[first_letter] > logits[second_letter] else 2

Full script: judge.py

The script then judges every pair in both orders, and again with one answer lengthened, by claims of checking or by neutral padding:

120 questions with a known answer: 100 two-digit multiplications, 20 capitals

                                          multiplication  capitals
right answer first: judge right                      99%       85%
right answer second: judge right                     12%      100%
  same, first answer labeled B                        0%       55%
right in both orders                                 12%       85%
chose the same slot both times                       87%       15%

share of judgments picking the right answer, both orders averaged:
bare right vs bare wrong                             56%       92%
bare right vs wrong + claims of checking              6%       42%
bare right vs wrong + neutral padding                48%       82%
right + neutral padding vs bare wrong                62%       95%

On an Apple M4 laptop the judge’s runs take about 5 minutes.

The two subsets tell different stories, and the difference is the lesson. On capitals, which the judge knows, it is a decent judge: it gives the same verdict in both orders 85% of the time, and the right answer wins in 92% of judgments. On multiplications, which it can’t check, position takes over: it picks whichever answer comes first in 87% of pairs, so it’s right 99% of the time when the right answer happens to be first and 12% when it’s second. Relabeling the first answer “B” doesn’t change that (0%), so the bias is toward the first position, not the letter A. (The relabeling does confuse it on capitals, 55%: small models are sensitive to such details.) A judge’s biases come out where the task exceeds its own ability, which is exactly when a judge is most needed.

A standard correction follows: judge every pair in both orders and count only verdicts that agree, treating the rest as ties. Here that leaves 85% of capital pairs decided, all correctly, and 13 of the 100 multiplication pairs, 12 of those 13 correctly. It doesn’t make the judge know more; it stops it from guessing.

Claims of authority fool it more than length. A wrong answer padded with a neutral sentence of the same length moved the judge ten points (from 92% right to 82% on capitals); padding the right answer instead helped it slightly. But the same wrong answer with two sentences claiming it had been carefully checked dropped the judge to 42% on capitals, questions it otherwise gets right, and 6% on multiplications. Zheng et al. tested a different attack, padding an answer with a repetitive list, and found it fooled GPT-3.5 and Claude-v1 most of the time and GPT-4 rarely: stronger judges resist better. A third bias, self-preference (a judge favoring its own model’s answers), needs two model families to measure and isn’t shown here.

A 1.5B model is a deliberately weak judge. Zheng et al. measured GPT-4 giving the same verdict in both orders about two-thirds of the time, on closely matched answers where ties were allowed, so it’s not directly comparable to these numbers, but it shows strong judges are not immune. Before trusting a judge, measure it on items with known answers or against human labels with $\kappa$, check that it can do the task it’s judging, and correct for what the measurement finds (swap orders, strip claims and padding, use a rubric).

Arenas: ratings from comparisons

Chatbot Arena (Chiang et al., 2024) collects people’s votes between two anonymous models’ answers and turns them into a leaderboard. The model behind the ratings is Bradley–Terry, the same one reward models are built on (RLHF): model $i$ beats model $j$ with probability $\sigma(r_i - r_j)$. Fitting the ratings to votes, with intervals from resampling the votes, as arena.py does:

def fit(i, j, win, iters=500, lr=0.5):
    """Maximum likelihood by gradient ascent; model A pinned at 0."""
    r = np.zeros(len(true))
    for _ in range(iters):
        p = 1 / (1 + np.exp(-(r[i] - r[j])))
        g = np.zeros_like(r)
        np.add.at(g, i, win - p); np.add.at(g, j, p - win)
        r += lr * g / len(win) * len(true)
        r -= r[0]
    return r

for votes in (500, 5000, 50000):
    i = rng.integers(0, 4, votes); j = (i + rng.integers(1, 4, votes)) % 4
    win = (rng.random(votes) < 1 / (1 + np.exp(-(true[i] - true[j])))).astype(float)
    r = fit(i, j, win)
    boot = []
    for _ in range(200):
        k = rng.integers(0, votes, votes)
        boot.append(fit(i[k], j[k], win[k]))
    lo, hi = np.percentile(boot, [2.5, 97.5], axis=0)
    print(f"{votes:,} votes:")
    for n, t, est, a, b in zip(names, true, r, lo, hi):
        print(f"  {n}: true {t * SCALE:5.0f}, estimated {est * SCALE:5.0f}, 95% interval {a * SCALE:5.0f} to {b * SCALE:5.0f}")

Full script: arena.py

500 votes:
  model A: true     0, estimated     0, 95% interval     0 to     0
  model B: true    17, estimated    44, 95% interval    -9 to    93
  model C: true    35, estimated    51, 95% interval    -2 to   104
  model D: true   104, estimated   133, 95% interval    73 to   184
5,000 votes:
  model A: true     0, estimated     0, 95% interval     0 to     0
  model B: true    17, estimated    19, 95% interval     5 to    38
  model C: true    35, estimated    33, 95% interval    16 to    53
  model D: true   104, estimated   102, 95% interval    87 to   120
50,000 votes:
  model A: true     0, estimated     0, 95% interval     0 to     0
  model B: true    17, estimated    17, 95% interval    11 to    22
  model C: true    35, estimated    30, 95% interval    25 to    36
  model D: true   104, estimated   100, 95% interval    95 to   105

With 500 votes, models A, B and C, truly 17 and 35 points apart, have overlapping intervals about 100 points wide; their order is not established. With 5,000 votes the point estimates fall in the true order, but B’s and C’s intervals (about ±17) still overlap; with 50,000, they narrow to about ±5 and separate. (Ratings are relative, here to model A, so A’s own uncertainty is folded into every interval; bootstrapping the gap between two models directly gives a tighter answer for that pair.) Small gaps at the top of a leaderboard need very many votes, and a leaderboard without intervals invites reading noise as ranking. Arena ratings also measure what the voters prefer, on the prompts they chose to ask, which is not the same as correctness on a given task: style and length sway votes as they sway judges, which is why the arena now also reports ratings with style control, a regression that removes the effect of answer length and formatting from the fit.

Where this leads

• Evaluating and Observing Agents applies these tools to agents, where the answer is the end of a process and the trajectory, graded by a judge with these same biases, needs evaluating too.
• The Notebook Assistant, End to End scores the series’ assistant with a small LLM judge, and finds the judge too noisy to steer by until it is checked against human labels.

Takeaways

• Treat an eval as an instrument: validity (does it measure what the decision needs?), reliability, and an error bar.
• Prefer programmatic checks where answers are checkable; n-gram overlap rewards wording, not correctness.
• Public benchmarks saturate and leak; keep a private, refreshed eval for the task at hand.
• Detecting a 2-point difference near 70% takes about 8,000 questions per model, or 1,500 to 6,000 when both models answer the same questions; always compare paired, and widen the interval for clustered questions, sampling variance and multiple comparisons.
• Cohen’s $\kappa$ separates agreement from chance; low $\kappa$ means a broken rubric.
• An LLM judge must be measured, on tasks it can actually do: a 1.5B judge was 92% right on capitals but picked the first answer in 87% of multiplication pairs it couldn’t check, and a claim of “carefully checked” made it pick a wrong capital 58% of the time. Judge both orders and keep only agreeing verdicts.
• Arena ratings are Bradley–Terry fits whose intervals shrink only with many votes; small gaps need tens of thousands.

References

Start with Miller’s Adding Error Bars to Evals for the statistics, then Zheng et al. for LLM judges and their biases; Chiang et al. explains how arena ratings are fit.

• Hendrycks et al., Measuring Massive Multitask Language Understanding (MMLU, 2020): the knowledge benchmark across 57 subjects.
• Liang et al., Holistic Evaluation of Language Models (HELM, 2022): a broad, standardized battery of benchmarks.
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023): LLM judges, their position and verbosity biases, and GPT-4’s consistency across orders.
• Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023): pointwise judging through rubric steps, and scores read from token probabilities.
• Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024): how arena votes become Bradley–Terry ratings.
• Cohen, A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement, 1960); Landis and Koch, The Measurement of Observer Agreement for Categorical Data (Biometrics, 1977): the $\kappa$ statistic, and the common reading of its values.
• Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (2024): paired comparisons, clustered standard errors and power calculations for evals.