Tag: llm-as-judge
All the articles with the tag "llm-as-judge".
Evaluating LLMs: Metrics, Benchmarks, Judges and Error Bars
Updated:An eval is a measurement instrument, and it needs what any instrument needs: validity, reliability and a known error bar. What to measure and with which metric; benchmarks and contamination; how many questions it takes to tell two models apart (8,077 for 2 points, or as few as 1,570 when the same questions are scored for both); confidence intervals and Cohen's kappa worked; an LLM judge measured on questions with known answers, where a 1.5B judge is sound on questions it can answer, falls back on position on ones it can't, and is fooled by claims of having checked; and arena ratings as Bradley-Terry with their intervals.