Engineering Around the Model: Harnesses, Loops, Graphs and the Definition of Done

The same model can make a useless agent or a reliable one. What differs is everything around it: what it’s told, what it can see and do, when it stops, and how anyone knows it’s finished. That surrounding code is the harness. Claude Code’s documentation puts it in one sentence: the harness is “the layer around the model that provides the tools and manages the context the model sees” (How Claude Code works, discussed in How Coding Agents Work). This post is about engineering that layer, and it ends with a measurement of one part of it: a check of whether the work is done.

Table of contents

The parts of a harness

• Instructions and context: the system prompt, the task, project files, and what’s retrieved or remembered (Context Engineering and Agent Memory).
• Tools: their definitions, their implementations, and their error messages, which are the model’s only window onto what went wrong.
• The loop controller: what happens after each model reply: run a tool, ask the user, check the work, stop.
• State: the transcript, plus anything kept outside it (files, a database, checkpoints) so a run can be resumed or inspected.
• Verification: whatever decides that the task is done.
• Budgets: limits on steps, tokens, time and money, and what happens when one runs out.
• Observability: traces of every step (Evaluating and Observing Agents).

The model is one component among these, and often not the one to improve first.

Loop design

Stopping. The simplest loop stops at the first reply without a tool call (see From Function Calling to the Agent Loop). That trusts the model to know when it’s done. Alternatives: a dedicated “finish” tool whose arguments are the final answer (so stopping is an explicit act with a structured output), or stopping only when a check passes, the subject of the rest of this post.

Retries. Two different failures need two different responses. A tool or API error (a timeout, a rate limit) is retried by the harness, with backoff, without bothering the model. A model error (malformed tool arguments, a call to a tool that doesn’t exist) goes back to the model as an error message it can act on: the error text is a prompt, and a specific one (“expression contains a letter; only numbers and + - * / are allowed”) gets fixed where a generic one doesn’t.

Budgets. Every loop needs limits: steps, tokens, wall-clock time, cost. The interesting question is what happens at the limit. Returning nothing wastes the work done; returning the best partial result, saving state so a person or a later run can continue, or escalating are all better than silence. And budgets should be measured, not guessed: the distribution of steps successful runs actually take says where the limit belongs.

Graphs

A loop has one shape. When a task has known stages (plan, then search, then draft, then review), branches (route by request type) or points where a person must approve, it’s clearer as a graph: nodes for steps, edges for what can follow what, shared typed state, and a checkpoint after each step. LangChain and LangGraph, from the Inside built the same agent as a loop and as a LangGraph graph: the graph cost a few more lines and a fraction of a millisecond per run, and bought pausing before a tool call, resuming, and a history of every step.

The trade is between flexibility and control. A loop lets the model decide the order of everything; a graph fixes some order in code. Anthropic’s guide to building agents (2024) frames this as workflows, where code orchestrates the model along predefined paths (prompt chaining, routing, parallelization, an evaluator that feeds back to a generator), versus agents, where the model directs its own process, and advises using the simplest that works. Fixed structure where the task’s shape is known, model autonomy where it isn’t.

The definition of done

An agent that decides for itself when it’s finished will sometimes decide wrong: it stops early, reports success on unfinished work, or answers from memory instead of checking. The fix is to make “done” a check the harness runs, not a claim the model makes. For code, the check is the test suite; for a data task, a schema validation or a reconciliation; for a document, a list of required sections. The best checks need no knowledge of the right answer, only of what a right answer must satisfy, so they can run in production, not just in evaluation.

Here is the experiment, on the calculator agent that Evaluating and Observing Agents uses to evaluate agents. It is right 93% of the time on an Apple M4 laptop (85% on the rerun), and its traces show that some right answers don’t come from the calculator it was told to use, and some wrong ones are mental arithmetic. A check the harness can run without knowing the answer: the final number must equal the calculator’s most recent result. If it doesn’t, the harness tells the agent and lets it continue; if the budget runs out first, the harness returns the agent’s last answer anyway, the best partial result rather than nothing. The same model, problems and random seeds, with and without the check, up to eight model calls per run, every transcript kept in runs.json. On an Apple M4 laptop both harnesses run in about 10 minutes. The harness is run in done_check.py; checked switches the definition of done on:

@torch.no_grad()
def run(question, seed, checked, max_calls=8):
    torch.manual_seed(seed)
    messages = [{"role": "system", "content": "Use the calculator tool for every arithmetic step. "
                 "End with: Answer: <number>"}, {"role": "user", "content": question}]
    last, calls, answer, given = None, 0, None, []    # given: (call number, answer) for every answer the agent gives
    while calls < max_calls:
        prompt = tok.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True, tokenize=False)
        ids = tok(prompt, return_tensors="pt").input_ids.to(DEVICE)
        out = model.generate(ids, max_new_tokens=512, do_sample=True, temperature=0.7, top_p=1.0, top_k=0,
                             repetition_penalty=1.1)
        text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
        calls += 1
        messages.append({"role": "assistant", "content": text})
        tool_calls = re.findall(r"<tool_call>\s*(\{.*?\})\s*</tool_call>", text, re.S)
        if tool_calls:
            for c in tool_calls:
                try:
                    r = calculator(json.loads(c)["arguments"]["expression"])
                except (ValueError, KeyError, TypeError):
                    r = "error: malformed tool call"
                if not r.startswith("error"):
                    last = float(r)
                messages.append({"role": "tool", "content": r})
            continue
        answer = last_number(text)
        given.append((calls, answer))
        if not checked or (answer is not None and last is not None and answer == last):
            return answer, calls, messages, given
        messages.append({"role": "user", "content": f"Your answer {answer} is not the calculator's last result. "
                         "Use the calculator for each step, then give its final result as the answer."})
    return answer, calls, messages, given                # budget exhausted: the last answer, unverified

Full script: done_check.py

The rest of done_check.py loads the model, defines the calculator tool and the 20 problems, runs each harness three times per problem, saves every transcript to runs.json, and prints the table. A run’s success under a smaller budget is read off from the call at which it finished.

plain    right 93%
checked  right 97%

harness    budget 2   budget 3   budget 4   budget 6   budget 8   (share right)
plain           93%        93%        93%        93%        93%
checked         93%        92%        92%        95%        97%
plain: model calls per run, mean 1.9, max 2; used all 8 calls 0 of 60
checked: model calls per run, mean 2.5, max 8; used all 8 calls 3 of 60
paired: the check fixed 2 plain failures and broke 0 plain successes

On the M4 the check raised accuracy from 93% to 97%. Because both harnesses use the same seeds, each pair of runs is identical until the check first objects, so the comparison is exact: it fixed 2 of the plain harness’s 4 failures and broke none of its successes. The transcripts in runs.json, which the script saves but doesn’t print, show what it caught: an answer worked out in prose (5242 − 623 = 4619, then a wrong sum), sent back until it came from the calculator. With four failures in sixty runs, that is a small sample, and the rerun shows how small: there both harnesses were right 85% of the time, and the check fixed 3 plain failures and broke 3 plain successes. Fixed 2 and broke 0 on one machine, fixed 3 and broke 3 on another: this sample cannot show that the check helps. What holds in both runs is narrower: the check changes which runs succeed, and it catches only what it checks.

The M4 run’s transcripts also show what it can’t catch. One failure passed the check in both harnesses: the agent asked the calculator for 8113 − (423 + 7 × 755), subtracting the deliveries instead of adding them, and answered 2405, exactly the calculator’s result. The check verifies where the answer came from, not whether the question put to the calculator was right. Another run fixed its mental arithmetic and then made a different error with the calculator. A check that would catch these needs to know more about the task: recompute the answer a second, independent way and compare, or check the expression against the problem’s structure. In code, the test suite plays that role, which is why coding agents (How Coding Agents Work) gain so much from verification.

The check cost 2.5 model calls per run against 1.9, a third more, and more than that in tokens, since each extra round rereads a longer context; three runs used all eight calls. On the rerun it cost 3.0 against 1.8, and nine runs used all eight. Because the harness returns the last answer when the budget runs out, a two-call budget loses nothing relative to the plain harness (93% for both on the M4, 87% against 85% on the rerun). At three and four calls the checked harness scores 92% on the M4, a point behind, and pulls ahead at six; on the rerun it never pulls ahead after two calls. A harness that returned nothing at the limit would instead count every unfinished run as a failure: what a harness does at its limits is part of the design.

Long-running harnesses

Tasks that take hours outlive any one context window, and two failure modes dominate, as Anthropic’s account of harnesses for long-running coding agents (2025) describes them: the agent tries to do too much at once, and a later session looks around, sees progress, and declares the job done. Its harness answers both with the ideas above, made persistent:

• An initializer session turns the request into a feature list, a JSON file of specific, testable requirements, each with a passes field that starts false, and an instruction that removing or editing tests is unacceptable. The definition of done is written down before work starts.
• Each later session starts by getting its bearings (the working directory, the git log, a progress file), picks the highest-priority feature not yet passing, works on that one, tests it end to end as a user would (through browser automation, not just unit tests), and leaves a commit and a progress note for the next session.
• Git provides both the record and the undo.

Evaluation harnesses

The same machinery evaluates agents (Evaluating LLMs, Evaluating and Observing Agents). An evaluation harness runs each task in a fresh environment (SWE-bench builds a container per task), applies the agent’s result, runs the check, and records the trace. Everything above applies, with one more requirement: the harness itself must be tested, because its bugs become the results. In Evaluating and Observing Agents, the first run of this agent reports a dozen failures that are really its own token limit.

Where this leads

• LangChain and LangGraph, from the Inside builds an agent as a loop and as a graph, and measures what the graph’s checkpoints cost and buy.
• How Coding Agents Work looks at the harnesses of production coding agents, where the test suite is the definition of done and hooks enforce rules outside the model.
• Evaluating and Observing Agents evaluates this calculator agent from its traces, and finds failures that are the harness’s own.

Takeaways

• The harness is everything around the model: context, tools, loop control, state, verification, budgets and observability.
• Retry infrastructure errors in the harness; send model errors back to the model as specific, actionable messages.
• Measure where budgets belong, and decide what happens at the limit.
• Use a graph when the task’s stages, branches or approvals are known; a loop when they aren’t.
• A definition of done catches what it checks. Measured: requiring the answer to be the calculator’s last result raised accuracy from 93% to 97% on an Apple M4 laptop (2 fixes, no breaks) and left it at 85% on a rerun (3 fixes, 3 breaks), so this sample cannot show that it helps. It cost a third to two-thirds more calls, and on the M4 still passed an answer computed from the wrong expression.
• Long-running harnesses write the definition of done down first (a feature list with pass flags), work on one item at a time, and leave a trail for the next session.

References

Start with Anthropic’s Building effective agents for workflows against agents, then its Effective harnesses for long-running agents for the feature-list pattern. Claude Code’s documentation gives the definition of a harness quoted at the start.

• Anthropic, Building effective agents (2024): workflows against agents, and the advice to use the simplest that works.
• Anthropic, Effective harnesses for long-running agents (2025): the initializer session, the feature list with pass flags, and one feature per session.
• Anthropic, How Claude Code works (documentation, 2026): the harness as “the layer around the model”.
• Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023): the evaluation harness that builds a container per task.