Tag: bradley-terry
All the articles with the tag "bradley-terry".
Evaluating LLMs: Metrics, Benchmarks, Judges and Error Bars
Updated:An eval is a measurement instrument, and it needs what any instrument needs: validity, reliability and a known error bar. What to measure and with which metric; benchmarks and contamination; how many questions it takes to tell two models apart (8,077 for 2 points, or as few as 1,570 when the same questions are scored for both); confidence intervals and Cohen's kappa worked; an LLM judge measured on questions with known answers, where a 1.5B judge is sound on questions it can answer, falls back on position on ones it can't, and is fooled by claims of having checked; and arena ratings as Bradley-Terry with their intervals.
RLHF: From Christiano 2017 to InstructGPT to Today's Recipe
Updated:Nobody can write a reward function for a helpful answer, but people can say which of two answers is better. The Bradley-Terry model turns those comparisons into a reward, worked on three answers; the KL-regularized objective keeps the policy near where it started, and its optimal policy is derived in closed form. Then the 2017 paper, InstructGPT's three steps, and an exact toy RLHF pipeline that shows reward overoptimization appear, over 100 worlds each, when the reward model's error has a direction the policy can reach, and rarely when it doesn't.