Tag: observability
All the articles with the tag "observability".
Evaluating and Observing Agents: Outcomes, Trajectories, Traces and Spans
Updated:An agent's answer can be right for the wrong reasons, wrong for a trivial one, or right only some of the time. Outcome, trajectory and tool-use evaluation; SWE-bench and τ-bench; pass@k against pass^k, with the arithmetic and unbiased estimators; and a small agent instrumented with OpenTelemetry's GenAI span conventions and evaluated from its traces, where most 'failures' turned out to be the answer's format or the harness's own token budget, and the traces showed exactly which.