Evaluating and Observing Agents: Outcomes, Trajectories, Traces and Spans

Evaluating LLMs treated a model’s answer as the thing to evaluate. An agent’s answer is the end of a process: several model calls, tool calls, intermediate results. That changes what evaluation has to look at. The final answer can be right because the agent did the work properly, or by luck; wrong because of a real mistake, or because it wrote “about twelve thousand” where a number was expected. And the same agent, asked the same thing twice, can do both. This post covers how to evaluate all of that, and the instrumentation that makes it possible.

Table of contents

Three levels of evaluation

• Outcome: did the task end in the right state? The answer is correct, the tests pass, the database row exists. This is what users care about, and the only level many benchmarks score.
• Trajectory: did the agent get there the right way? It called the tools it should, in a sensible order, without detours, without actions it shouldn’t take (deleting a file, emailing a customer) on the way. Two agents with the same outcome score can differ completely here, and the one that guessed right will fail on the next variation.
• Tool use: was each call well formed and appropriate? Right tool, valid arguments, result actually used.

Agent benchmarks mostly score outcomes in environments built to make the outcome checkable. SWE-bench (Jimenez et al., 2023) gives an agent a real GitHub issue and repository and runs the project’s own tests on its patch. τ-bench (Yao et al., 2024) puts an agent in a simulated customer service conversation, with a simulated user and tools over a database, and checks the database’s final state against the goal, along with information the agent had to tell the user. τ-bench also introduced the metric this post turns to next, because it asks the question production cares about: does the agent succeed every time? Its strongest agent at publication succeeded on about 60% of retail tasks in one try, and on about a quarter in all of eight.

Agent evaluations also need what model evaluations don’t: an environment that resets. An agent with tools changes things (files, databases, sent messages), so each run needs a fresh sandbox, as SWE-bench’s per-task containers and τ-bench’s database resets provide, or runs contaminate each other and side effects leak into the real world.

pass@k and pass^k

Run an agent $k$ times on one task. Two different questions:

• pass@k: does at least one of the $k$ attempts succeed? Right when answers can be checked and the best kept, as a coding benchmark with unit tests can.
• pass^k: do all $k$ succeed? Right when every attempt goes to a user, as in customer service, where $k$ customers asking the same thing should all be helped.

For a task the agent solves with probability $p$ per attempt, they are $1 - (1 - p)^k$ and $p^k$:

from math import comb

# pass@k: at least one of k attempts succeeds (what a benchmark with
# retries, or a human picking the best of k, cares about). pass^k: all k
# succeed (what a user who asks k times, or k customers, cares about).
# For one task with per-attempt success p: 1 - (1 - p)^k and p^k.
print(f"{'p':>5s} " + " ".join(f"{'pass@' + str(k):>8s}" for k in (1, 4, 8)) + "   " +
      " ".join(f"{'pass^' + str(k):>8s}" for k in (1, 4, 8)))
for p in (0.5, 0.8, 0.9, 0.95, 0.99):
    print(f"{p:5.2f} " + " ".join(f"{1 - (1 - p) ** k:8.3f}" for k in (1, 4, 8)) + "   " +
          " ".join(f"{p ** k:8.3f}" for k in (1, 4, 8)))

# From n attempts with c successes, unbiased estimates for k <= n
# (Chen et al. 2021 for pass@k; the same counting gives pass^k).
def pass_at(n, c, k): return 1 - comb(n - c, k) / comb(n, k)
def pass_hat(n, c, k): return comb(c, k) / comb(n, k)
print(f"\n5 attempts, 3 succeed: pass@1 {pass_at(5, 3, 1):.2f}, pass@3 {pass_at(5, 3, 3):.2f}, "
      f"pass^3 {pass_hat(5, 3, 3):.2f}")

Full script: pass_k.py

    p   pass@1   pass@4   pass@8     pass^1   pass^4   pass^8
 0.50    0.500    0.938    0.996      0.500    0.062    0.004
 0.80    0.800    0.998    1.000      0.800    0.410    0.168
 0.90    0.900    1.000    1.000      0.900    0.656    0.430
 0.95    0.950    1.000    1.000      0.950    0.815    0.663
 0.99    0.990    1.000    1.000      0.990    0.961    0.923

5 attempts, 3 succeed: pass@1 0.60, pass@3 1.00, pass^3 0.10

They move in opposite directions. An agent that succeeds 80% of the time looks nearly perfect by pass@8 (1.000) and unreliable by pass^8 (0.17): across eight customers, it fails at least one five times in six. Even 99% per attempt gives pass^8 of 0.92. τ-bench reported exactly this pattern: agents whose single-attempt success looked respectable fell sharply as $k$ grew. Report $k$, and pick the direction that matches the deployment.

From $n$ attempts with $c$ successes, both have unbiased estimates for any $k \le n$, by counting subsets (pass@k’s from Chen et al., 2021, pass^k’s from τ-bench): $1 - \binom{n-c}{k}/\binom{n}{k}$ and $\binom{c}{k}/\binom{n}{k}$. With 3 of 5 successes, pass@3 is 1.0 (any 3 attempts include a success) and pass^3 is 0.1 (1 of the 10 possible triples is all successes).

Traces and spans

Trajectory evaluation needs a record of the trajectory, and that is what tracing provides. A trace is the record of one request through a system; it is made of spans, each a timed operation with a name, attributes, and a parent, so they form a tree. Distributed systems have used this for years, and OpenTelemetry is the vendor-neutral standard for it. Its GenAI semantic conventions (still marked “development”) fix the names for AI work, so any backend can read any agent’s traces:

• an agent run is a span named invoke_agent {agent name};
• each model call is chat {model}, with gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens;
• each tool call is execute_tool {tool name}, with gen_ai.tool.name, gen_ai.tool.call.id, and, when opted in, the arguments and result;
• gen_ai.operation.name on every span says which kind it is, and gen_ai.response.finish_reasons says why a model call stopped;
• model calls are client spans (a call to another service), tool calls usually internal ones; a failed operation sets error.type and the span’s error status.

Message contents (prompts, responses, tool arguments and results) are opt-in, because they may hold personal data; deciding what to record is a privacy decision as much as a debugging one. Production systems usually redact personal data before storing contents, and sample traces (keep all failures and slow runs, a fraction of the rest) to control volume.

An agent, traced and evaluated

A small agent to evaluate: Qwen2.5-1.5B-Instruct with one tool, a calculator, on 20 word problems that each need two arithmetic steps on 3- and 4-digit numbers, run 5 times each at temperature 0.7. Every model call and tool call becomes a record shaped like an OpenTelemetry span, with the GenAI attribute names (plain dictionaries here, to keep the code self-contained; the OpenTelemetry SDK would add span kinds, wall-clock timestamps and export to a tracing backend).

In traced_agent.py, a span is a dictionary with its trace, its parent and a start time:

def span(trace, name, parent, **attrs):
    s = {"trace_id": trace["id"], "span_id": uuid.uuid4().hex[:16], "parent_id": parent, "name": name,
         "start": time.perf_counter(), "attributes": attrs}
    trace["spans"].append(s)
    return s

run() opens a root invoke_agent span for the whole run. Each turn of the agent then opens a chat span under it, generates, and records the token counts, the finish reason and the reply:

def run(question, seed, max_turns=5):
    torch.manual_seed(seed)
    trace = {"id": uuid.uuid4().hex, "spans": []}
    root = span(trace, "invoke_agent warehouse-helper", None, **{"gen_ai.operation.name": "invoke_agent",
                "gen_ai.agent.name": "warehouse-helper"})
    messages = [{"role": "system", "content": "Use the calculator tool for every arithmetic step. "
                 "End with: Answer: <number>"}, {"role": "user", "content": question}]
    answer = lenient = final = None
    for _ in range(max_turns):
        prompt = tok.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True, tokenize=False)
        ids = tok(prompt, return_tensors="pt").input_ids.to(DEVICE)
        s = span(trace, f"chat {MODEL}", root["span_id"], **{"gen_ai.operation.name": "chat",
                 "gen_ai.provider.name": "local", "gen_ai.request.model": MODEL, "gen_ai.request.temperature": 0.7,
                 "gen_ai.request.max_tokens": 512})
        out = model.generate(ids, max_new_tokens=512, do_sample=True, temperature=0.7, top_p=1.0, top_k=0,
                             repetition_penalty=1.1)
        new = out[0, ids.shape[1]:]
        text = tok.decode(new, skip_special_tokens=True)
        s["end"] = time.perf_counter()
        stopped = new[-1].item() in (tok.eos_token_id, tok.convert_tokens_to_ids("<|im_end|>"))
        s["attributes"].update({"gen_ai.usage.input_tokens": ids.shape[1], "gen_ai.usage.output_tokens": len(new),
                                "gen_ai.response.finish_reasons": ["stop" if stopped else "length"],
                                "gen_ai.output.messages": [{"role": "assistant", "content": text}]})

Full script: traced_agent.py

Tool calls get an execute_tool span the same way, with the arguments and the result or an error type. traced_agent.py saves all 100 traces to traces.json.

The evaluation then reads only the traces and the answer key. From each trace, evaluate_traces.py extracts the facts it scores:

def facts(r):
    spans = r["trace"]["spans"]
    chats = [s["attributes"] for s in spans if s["attributes"]["gen_ai.operation.name"] == "chat"]
    tools = [s["attributes"] for s in spans if s["attributes"]["gen_ai.operation.name"] == "execute_tool"]
    final = chats[-1]["gen_ai.output.messages"][0]["content"]
    strict = re.search(r"Answer:\s*\$?(-?[\d,]+)", final)
    nums = re.findall(r"-?\d[\d,]*", final)
    return {"final": final, "tools": tools,
            "truncated": any(c["gen_ai.response.finish_reasons"] == ["length"] for c in chats),
            "strict": int(strict.group(1).replace(",", "")) if strict else None,
            "lenient": int(nums[-1].replace(",", "")) if nums else None,
            "tool_has_truth": str(key[r["task"]]) in [t.get("gen_ai.tool.call.result", "").split(".")[0] for t in tools]}

and sorts every run into a trajectory:

def trajectory(f, truth):
    right = f["lenient"] == truth
    if f["truncated"]:
        return "cut off by the token budget" + (" (still right)" if right else "")
    if right:
        return "right, the calculator produced the answer" if f["tool_has_truth"] else "right, the calculator didn't produce it"
    if f["tools"] and not f["tool_has_truth"]:
        return "wrong: asked the calculator the wrong question"
    if f["tool_has_truth"]:
        return "wrong: calculator had it, reply differs"
    return "wrong: no calculator"

Full script: evaluate_traces.py

The rest of the script computes pass@k and pass^k both ways, counts the trajectories and calculator calls, sums the spans into a dashboard, and prints one trace:

outcome scored as asked ('Answer: <number>'), 20 tasks x 5 runs:
  pass@1 0.15   pass^1 0.15
  pass@3 0.39   pass^3 0.00
  pass@5 0.55   pass^5 0.00
  tasks solved in all 5 runs: 0 of 20
outcome scored by the last number in the reply, 20 tasks x 5 runs:
  pass@1 0.93   pass^1 0.93
  pass@3 1.00   pass^3 0.81
  pass@5 1.00   pass^5 0.70
  tasks solved in all 5 runs: 14 of 20

trajectories (100 runs):
   81  right, the calculator produced the answer
   11  right, the calculator didn't produce it
    4  wrong: no calculator
    3  wrong: asked the calculator the wrong question
    1  cut off by the token budget (still right)
  calculator calls per run: none 14, one 84, two or more 2 (the instructions asked for one per arithmetic step: two)
  runs with no 'Answer:' line: 83; of those, replies saying 'the answer is': 19

dashboard: 100 runs, 186 model calls, 88 tool calls
  per run: 1.9 model calls, 454 tokens in, 81 out; per successful run: 575 tokens
  run latency p50 2.5 s, p95 10.9 s; model call p50 1.55 s; tool call p50 33 us

one trace (task 9, right answer 12975):
  invoke_agent warehouse-helper                0 ms  +  2468 ms  
      chat qwen2.5-1.5b-instruct                   1 ms  +  1594 ms  in 213 out 33 tokens, finish stop
      execute_tool calculator                   1595 ms  +     0 ms  {'expression': '8113-(423+7*755)'} -> 2405
      chat qwen2.5-1.5b-instruct                1596 ms  +   872 ms  in 268 out 12 tokens, finish stop
  final reply: 'The warehouse holds 2405 boxes now.'

On an Apple M4 laptop the 100 runs take about 6 minutes.

The first score was a measurement error, twice. Scored as instructed, by an Answer: <number> line, the agent passes 15% of runs. Scored by the last number in its final reply, 93%. Most replies gave the number in a plain sentence, like the trace’s “The warehouse holds 2405 boxes now”; 19 said “the answer is” without the colon. The agent did the task and loosely followed the format. And an earlier version of this experiment, with a smaller token budget per model call, scored lower and seemed to show the model doing arithmetic in its head; the traces showed that most of those replies had been cut off mid-sentence and scored by whatever number came last. With 512 tokens and each call’s finish reason recorded (gen_ai.response.finish_reasons), one run hit the limit, and it was still right. The harness’s budgets, parser and sampling settings are part of the evaluation, and they belong in the trace, so that a surprising score can be checked against what the model actually said.

The trajectory separates good answers from lucky ones. Of the 93 right answers, 81 came out of the calculator. In 11, no calculator result contained the answer: the model did at least part of the arithmetic itself and got it right. Those count as successes by outcome and as process failures by trajectory, and the four runs where unaided arithmetic went wrong show what the habit costs. Three runs used the calculator correctly on the wrong expression. The trace above is one: the agent computed 8113 − (423 + 7 × 755), subtracting the deliveries instead of adding them, and the calculator faithfully returned 2405. Every span in that trace is healthy; the error is in what the model asked for, which no tool-level metric catches.

And one rule went unchecked until the traces were counted. The instructions asked for the calculator “for every arithmetic step”, two per problem. Only 2 runs made two calls; 84 made exactly one, which in the trace above held both steps in one expression. Whether that’s a violation depends on what the rule was for, but an outcome score could never have shown it.

pass^k is the sobering number. At 93% of runs right, 14 of the 20 tasks were solved in all five runs, and pass^5 averages 0.70. (Here that happens to match $0.93^5$; in general the average of per-task pass^k differs from the average success rate raised to the $k$, because failures concentrate in some tasks, and it’s the per-task view that a user meets.) With five runs per task, each task’s pass^5 estimate is just “all five passed or not”, and with 20 tasks of one template, these numbers describe this agent on this problem, nothing broader.

Rule-based categories like these don’t scale to open-ended tasks, where “the right route” can’t be written as a check. There, trajectories are graded by an LLM judge reading the trace against a rubric, with all the judge biases of Evaluating LLMs and the same need to calibrate it against people.

The dashboard lines come from the same spans: 1.9 model calls and about 535 tokens per run, 575 tokens per successful run (the number to budget with), and a median run of 2.5 seconds, almost all of it model time; the tool calls take microseconds. In a real system the tool spans would hold network calls and database queries, and the same breakdown would show where the time goes.

Observability is not evaluation

The same traces serve two purposes that are easy to conflate. Evaluation asks whether the agent is good, on a fixed set of tasks, before a change ships. Observability asks what the agent is doing in production, on traffic nobody chose: latency and cost per run, how many model calls and tool calls a run takes, which tools fail, where time goes. The dashboard lines above are the start of it. In production, the useful views are distributions, not averages (a p95 run that makes ten model calls is a different problem from a slow model), cost per successful task rather than per call, and alerts on shifts: a jump in calls per run often means an agent has started looping. The two meet in a loop: production traces of failures become new evaluation tasks.

Where this leads

• Guardrails and Prompt Injection turns to what an agent with tools can be made to do by the text it reads, and why bounding its tools matters more than filtering.
• Computer-Use Agents applies the pass^k arithmetic to long tasks on a screen, where a task fails if any step does.

Takeaways

• Evaluate agents at three levels: the outcome, the trajectory that produced it, and each tool call.
• pass@k (any of $k$ succeeds) and pass^k (all $k$ succeed) diverge quickly: 80% per attempt is 1.00 by pass@8 and 0.17 by pass^8. Choose the one that matches how answers reach users.
• Traces made of spans are what make trajectory evaluation possible; OpenTelemetry’s GenAI conventions name them (invoke_agent, chat, execute_tool) so any tool can read them.
• A small agent scored 15% by the requested answer format and 93% by the number it actually gave, after an earlier run’s smaller token budget had silently truncated replies; its traces showed both problems, which right answers skipped the calculator, and which wrong ones asked it the wrong question. 14 of 20 tasks were right in all five runs.
• Observability watches production with the same traces; production failures become tomorrow’s eval tasks.

References

Start with τ-bench for pass^k and why reliability matters in production, then OpenTelemetry’s GenAI semantic conventions for how to trace an agent.

• Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023): the coding-agent benchmark scored by each project’s own tests.
• Yao et al., τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024): simulated customer-service tasks, the pass^k metric and its estimator.
• Chen et al., Evaluating Large Language Models Trained on Code (2021): the unbiased pass@k estimator.
• OpenTelemetry, Semantic conventions for generative AI (development status): the span names and gen_ai.* attributes the agent’s traces use.