Tag: evaluation
All the articles with the tag "evaluation".
The Notebook Assistant, End to End
Updated:The series' running example, built as one working system: a question-answering assistant for this blog, with a semantic cache, hybrid retrieval, a cross-encoder reranker, a prompt that fences retrieved text as data, a 3B model that cites its passages, and a citation check. One question traced through every stage; a scorecard on 150 questions, 15 the blog doesn't answer, 30 repeats and a planted injection, with the judge checked by hand; the cost per answer, refining the series map's estimate; and what to change next.
Evaluating and Observing Agents: Outcomes, Trajectories, Traces and Spans
Updated:An agent's answer can be right for the wrong reasons, wrong for a trivial one, or right only some of the time. Outcome, trajectory and tool-use evaluation; SWE-bench and τ-bench; pass@k against pass^k, with the arithmetic and unbiased estimators; and a small agent instrumented with OpenTelemetry's GenAI span conventions and evaluated from its traces, where most 'failures' turned out to be the answer's format or the harness's own token budget, and the traces showed exactly which.