From Function Calling to the Agent Loop

The posts from Chain-of-Thought and Prompt Chaining to Building RAG built single calls: a prompt, retrieved context, an answer. An agent makes the model decide what to do next. The definition this series uses is short: a model in a loop, with tools, and a rule for when to stop. Everything in Part X: Agents elaborates one of those three parts.

Table of contents

Chatbots, workflows and agents

Three designs, by who decides the steps:

• A chatbot answers each message in one call. No steps.
• A workflow runs model calls in an order the programmer fixed: the chains of Chain-of-Thought and Prompt Chaining, a RAG pipeline. The code decides the steps.
• An agent runs model calls in a loop where the model decides the next step: which tool to call, with what arguments, and when it’s done.

Workflows are predictable and cheaper; agents handle tasks whose steps can’t be known in advance, at the price of variable cost and new ways to fail. Anthropic’s Building Effective Agents draws the same line (“workflows are systems where LLMs and tools are orchestrated through predefined code paths,” agents those where models “dynamically direct their own processes”) and gives the usual advice: find “the simplest solution possible,” which “might mean not building agentic systems at all”; for many applications, “optimizing single LLM calls with retrieval and in-context examples is usually enough.” The experiment below tests that advice on small models.

Function calling is text plus a protocol

A model can’t run code. Function calling (or tool use) is a protocol around the text it generates:

1. The prompt describes the available tools: a name, a description, and a JSON schema for the arguments. The chat template puts them in the system prompt.
2. The model, trained to do so, emits a call as structured text: for the Qwen models used here, JSON between <tool_call> tags. Toolformer (Schick et al., 2023) showed a model could teach itself to insert API calls into its own text; instruction-tuned models are now trained on tool-use conversations directly.
3. The application parses it, runs the function, and appends the result to the transcript as a tool message.
4. The model is called again, now seeing the result.

Here’s a whole exchange as the model sees it, rendered through Qwen2.5’s chat template:

import json
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download

# What function calling looks like as text: one exchange rendered through
# Qwen2.5's chat template. The tool's schema goes into the system prompt;
# the model's call is JSON between <tool_call> tags; the application's
# result goes back as a tool message, which the template wraps in
# <tool_response> tags. (Search result abridged.)
tok = AutoTokenizer.from_pretrained(snapshot_download("Qwen/Qwen2.5-1.5B-Instruct",
                                                      revision="989aa7980e4cf806f80c7fef2b1adb7bc71aa306"))
tools = [{"type": "function", "function": {"name": "search_posts", "description": "Search the blog.",
          "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}]
call = {"name": "search_posts", "arguments": {"query": "online softmax running maximum"}}
result = [{"post": "flashattention-explained", "heading": "The online softmax", "snippet": "Keep a running maximum m..."}]
messages = [{"role": "system", "content": "You answer questions about a technical blog, using its tools."},
            {"role": "user", "content": "How can a maximum and a running sum be updated incrementally?"},
            {"role": "assistant", "content": f"<tool_call>\n{json.dumps(call)}\n</tool_call>"},
            {"role": "tool", "content": json.dumps(result)}]
print(tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False))

Full script: chat_template.py

<|im_start|>system
You answer questions about a technical blog, using its tools.

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "search_posts", "description": "Search the blog.", "parameters": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}}}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call><|im_end|>
<|im_start|>user
How can a maximum and a running sum be updated incrementally?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "search_posts", "arguments": {"query": "online softmax running maximum"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
[{"post": "flashattention-explained", "heading": "The online softmax", "snippet": "Keep a running maximum m..."}]
</tool_response><|im_end|>
<|im_start|>assistant

Everything is text in one sequence. The tools are a block of JSON in the system prompt, with instructions on the format to use. The call is text the model wrote. And the tool’s result goes back inside a user turn, wrapped in <tool_response> tags, which is why text that comes back from a tool can carry instructions the model may follow (see Guardrails and Prompt Injection).

The model never runs the function; the loop does. That is also where validation and permissions belong (see Guardrails and Prompt Injection): the model proposes, the code disposes. A call can be malformed (invalid JSON, a missing argument, a tool that doesn’t exist), so the parser has to handle failure, usually by returning the error to the model as the tool’s result. Serving engines can rule out syntactically malformed calls by constrained decoding: at each step, mask every token that would break the schema’s grammar, as SGLang does for structured output (SGLang and RadixAttention). That guarantees the form, not the sense: the right tool with wrong arguments still gets through. APIs also let the application force a tool call: tool_choice: "required" in OpenAI’s API, {"type": "any"} in Anthropic’s. The experiment below approximates it by starting the model’s reply with <tool_call>.

A model can also emit several calls in one turn, parallel calls, when they don’t depend on each other: three searches at once instead of three round trips. The loop below runs every call it finds in a reply before calling the model again.

Think, act, observe

The loop is three moves, repeated: the model thinks (in text, or in a reasoning model’s hidden chain), acts (emits a tool call), and observes (reads the result appended to its context). It stops on one of:

• the model answers without a tool call: done, by its own judgment;
• a turn limit, a token budget, or a time budget: done, by the system’s;
• an error the loop can’t recover from.

Every real agent needs the second kind. A model’s judgment that it is done can fail in both directions: stopping before the work is finished, or never stopping. In the experiment below, the turn limit never fired; the first rule did all the stopping, including for a model that “finished” without doing any work.

The transcript is the memory, and it grows

Each turn appends the model’s call and the tool’s result, and the next call resends everything. Turn $t$ reads all $t - 1$ previous turns, so the tokens read over a whole run grow with the square of the number of turns:

# What an agent's transcript costs. Every turn resends everything so far: a
# P-token prefix (system prompt and tools), then g new tokens per turn (the
# model's call plus the tool's result). Turn t reads P + (t - 1) g tokens of
# context, so T turns read T P + g T (T - 1) / 2 in all: quadratic in T.
# With prompt caching ("Prompt Caching"), turn 1 writes the prefix to the cache at
# 1.25x the base input price; each later turn reads everything sent before
# at 0.1x and writes the previous turn's new tokens at 1.25x.
P, g, price = 5_000, 1_500, 3 / 1e6           # tokens; dollars per input token (an illustrative $3 per million)
print(f"prefix {P:,} tokens, {g:,} new per turn, ${price * 1e6:.0f} per million input tokens")
print(f"{'turns':>6s} {'final context':>14s} {'tokens read in all':>19s} {'uncached':>9s} {'cached':>8s}")
for T in (5, 10, 20, 50):
    total = T * P + g * T * (T - 1) // 2
    cached = P * 1.25 + sum((P + (t - 2) * g) * 0.1 + g * 1.25 for t in range(2, T + 1))   # reread, plus the new turn written
    print(f"{T:6d} {P + (T - 1) * g:12,d}   {total:17,d}   ${total * price:7.2f}  ${cached * price:6.2f}")

Full script: transcript_cost.py

prefix 5,000 tokens, 1,500 new per turn, $3 per million input tokens
 turns  final context  tokens read in all  uncached   cached
     5       11,000              40,000   $   0.12  $  0.05
    10       18,500             117,500   $   0.35  $  0.10
    20       33,500             385,000   $   1.16  $  0.23
    50       78,500           2,087,500   $   6.26  $  0.90

At 50 turns, a run reads 2 million tokens, though its final context is only 78,500. Prompt Caching cuts the bill to $0.90 from $6.26, because each turn re-reads the earlier turns from the cache at a tenth of the price. It doesn’t change the shape: the cached reads still grow with the square of the turns, only ten times more slowly, and past about 25 turns here they’re most of the cost. The prices are Anthropic’s for a 5-minute cache; a 1-hour cache writes at twice the base price, other providers price cache reads differently, and very short prefixes aren’t cached at all. All of it holds while the transcript is append-only: editing earlier turns, the context engineering of Context Engineering, breaks the cache from the edit on.

An experiment: an agent on this blog

The definition says the model decides what to do next. A fair test of that is a task where retrieval is the obvious next step, run two ways:

• a workflow: search the blog with the question, put the top three results in the prompt, and answer, one model call;
• an agent: the same search as a tool, search_posts(query), plus read_section(post, heading), and the loop above, up to six turns.

The questions are the 30 from Beyond Vector RAG, three each about 10 posts, each answered by one known section and written to resemble it, which favors searching with the question itself, the workflow’s move. Each answer must end with Source: <post>. The search is the hybrid of BM25 and MiniLM from Semantic and Hybrid Search, and both designs see the same results: a post’s name, a section heading and the section’s first 60 words. The agent runs three ways: as is; with the system prompt adding “always call search_posts before answering”; and with that prompt plus a forced first call. Every variant runs on Qwen2.5-1.5B-Instruct and on the 3B model, greedy. “Tokens read” counts the input of every call, not the output. The models run on the GPU and download about 9 GB together; on an Apple M4 laptop both sizes take about 19 minutes.

agent.py builds the search over the blog’s sections, the two tools and their schemas, and the system prompts. The part that makes it an agent is the loop: generate, run any tool calls the reply contains, append the results, and generate again until a reply has no call:

@torch.no_grad()
def generate(messages, tools=TOOLS, force_call=False):
    """force_call starts the reply with "<tool_call>", approximating an API's
    tool_choice = "required": the model must start a tool call."""
    prompt = tok.apply_chat_template(messages, tools=tools, add_generation_prompt=True, tokenize=False)
    prefix = "<tool_call>\n" if force_call else ""
    x = tok(prompt + prefix, return_tensors="pt").input_ids.to(DEVICE)
    out = lm.generate(x, max_new_tokens=200, do_sample=False, temperature=None, top_p=None, top_k=None)
    return prefix + tok.decode(out[0, x.shape[1]:], skip_special_tokens=True), x.shape[1]

def agent(question, system=SYSTEM, force_first=False, max_turns=6):
    messages = [{"role": "system", "content": system}, {"role": "user", "content": question}]
    read, bad, calls_made = [], 0, 0
    for turn in range(1, max_turns + 1):
        text, n = generate(messages, force_call=force_first and turn == 1); read.append(n)
        calls = re.findall(r"<tool_call>\s*(\{.*?)\s*</tool_call>", text, flags=re.S)
        if not calls:                                   # no call: the model says it's done
            return text, read, bad, calls_made
        messages.append({"role": "assistant", "content": text})
        for c in calls:                                 # the loop runs the tools, not the model
            try:
                call = json.JSONDecoder().raw_decode(c)[0]; result = RUN[call["name"]](**call["arguments"])
            except Exception as e:
                bad += 1; result = f"error: {type(e).__name__}: {e}"
            calls_made += 1
            messages.append({"role": "tool", "content": json.dumps(result)})
    return "(ran out of turns)", read, bad, calls_made

Full script: agent.py

both_models.py runs it with each model size.

Full script: both_models.py

In the table below, “found” counts questions where a search returned the right post, “cites any” answers with a Source line, “last context” the input tokens of the final call, “no tool” agent runs that never called a tool, and “malformed” tool calls that failed to parse or run. The “cites right” column uses a lenient scorer, replaced after the table:

446 sections; 30 questions; Qwen2.5-1.5B-Instruct, greedy

                                  found  cites any  cites right  model calls  tool calls  last context  tokens read  no tool  malformed
one retrieval, one answer            23         28           17          1.0         1.0           427          427        0          0
agent loop                            0         26            0          1.0         0.0           286          286       30          0
agent, 'always search first'          0         25            0          1.0         0.0           309          309       30          0
agent, first call forced             18          1            2          2.0         1.0           710        1,021        0          0
446 sections; 30 questions; Qwen2.5-3B-Instruct, greedy

                                  found  cites any  cites right  model calls  tool calls  last context  tokens read  no tool  malformed
one retrieval, one answer            23         30           23          1.0         1.0           427          427        0          0
agent loop                            4         29            3          1.2         0.2           371          428       24          0
agent, 'always search first'          9         28            3          1.4         0.4           491          624       17          0
agent, first call forced             21         24           10          2.0         1.0           720        1,030        0          0

That table’s “cites right” column came from a scorer that was too lenient: it counted the post’s name anywhere after “Source:”, or anywhere in the answer if it had no source line, and it accepted a list of several posts. The logged answers are rescored so that a citation counts only when the source line names exactly one of the blog’s posts, the right one. rescore.py reads the answers agent.py logged and classifies each source line:

SLUGS = {os.path.basename(p)[:-4] for p in glob.glob("content/posts/*.mdx")}
def cited(answer):
    if "Source:" not in answer:
        return "no source line", None
    line = answer.split("Source:")[-1]
    named = {s for s in SLUGS if s in line}
    if len(named) == 1:
        return "one real post", named.pop()
    if len(named) > 1:
        return "several posts", None
    return "placeholder or invented", None

Full script: rescore.py

Qwen2.5-1.5B-Instruct
                                    found  cites right   source line: none / one real post / several / placeholder or invented
  one retrieval, one answer            23           17      2 / 26 /  0 /  2
  agent loop                            0            0      4 /  0 /  0 / 26
  agent, 'always search first'          0            0      5 /  0 /  0 / 25
  agent, first call forced             18            0     29 /  0 /  1 /  0
Qwen2.5-3B-Instruct
                                    found  cites right   source line: none / one real post / several / placeholder or invented
  one retrieval, one answer            23           23      0 / 30 /  0 /  0
  agent loop                            4            3      1 /  4 /  0 / 25
  agent, 'always search first'          9            3      2 /  3 /  1 / 24
  agent, first call forced             21           10      6 / 15 /  1 /  8

Left to decide, the models mostly didn’t search. The 1.5B model answered all 30 questions from memory, with or without the instruction to search first, and gave sources it had never seen: without the instruction, 26 of its 30 answers had a placeholder or invented source line, such as Source: <post>, the placeholder copied from the prompt; with it, it invented titles like “[Blog Post Title]”. The 3B model searched for 6 of the 30, and the explicit instruction raised that only to 13. “Always call search_posts” is text the model weighs against its own sense that it already knows the answer, and a small model asked about attention believes it does.

Forced to search, they found the post, and then the citations broke. With the first call forced, the 3B model’s own query put the right post among the results 21 times, nearly as often as searching with the question itself (23); the 1.5B model’s, 18 times. The 1.5B model then dropped the source line in 29 of 30 answers, and cited the right post none of the 30 times. The 3B model cited the right post 10 times. Its other 20 answers: no source line (6), a real post that wasn’t the right one (5), several posts (1), and a placeholder or a source it made up (8), such as “Source: [section]”, “blog-post-about-id-generation” and a placeholder example.com URL. A citation is generated text like any other, which is why the checks in Engineering Around the Model compare a claimed source with what the tools actually returned.

No agent ever took a second step. Every forced run made exactly two model calls: the forced search and the answer. No model ever read a section or searched again with a better query, so the loop’s advantage, taking more than one step, never came into play.

The workflow won. With the 3B model, one search with the question and one answer cited the right post for 23 of 30, at 427 input tokens per question; the best agent managed 10, at 1,030. For a task whose steps are known in advance (search, then answer), the steps belong in code, which is the advice quoted above, in numbers.

How far this reaches is limited: 30 questions written to match their sections, one prompt wording, greedy decoding, and two small models. It shows that small models under-use tools on questions they think they know, and misreport where an answer came from; it isn’t a verdict on agents. Larger models trained for long tool-use sessions take more steps, which is where an agent earns its cost: tasks whose steps can’t be fixed ahead of time. What carries over is how to measure one: count whether tools were called, what they returned, and whether the answer matches it, not only whether the answer reads well.

Where this leads

• Agent Architectures compares ways to organize the thinking around tools: ReAct, planning first, and reflection.
• Agent Memory gives an agent memory beyond its transcript.

Takeaways

• An agent is a model in a loop, with tools and a stopping rule; a workflow is model calls in an order the code fixes. Use the simplest one that works.
• Function calling is a protocol around generated text: tool schemas in the prompt, a call written as JSON, a result sent back as a message. The model never runs anything; the application parses the call, should validate it (this post’s loop only parses), runs it, and returns the result.
• The transcript is resent every turn, so tokens read grow with the square of the number of turns. Prompt caching reads the earlier turns at a tenth of the price, which cut the cost sevenfold at 50 turns here, but it doesn’t change the shape, and only while the transcript is appended to, never edited.
• On 30 questions about this blog, small models left to decide rarely searched (1.5B never, 3B for 6 of 30), wrote sources they hadn’t seen, and never took a second step; one retrieval and one answer beat every agent variant. Forcing the first call fixed the searching but not the citing.
• Measure agents by what they did, not only what they said: whether a tool was called, what it returned, and whether the answer matches it. Check the scorer too: a lenient one credited citations here that the answers never made.

References

Start with Anthropic’s Building Effective Agents for when an agent is worth it, then Qwen’s function-calling documentation for the protocol at the token level; Toolformer shows a model teaching itself to call tools.

• Schick et al., Toolformer: Language Models Can Teach Themselves to Use Tools (NeurIPS, 2023): a model that teaches itself to insert API calls into its text.
• Anthropic, Building Effective Agents (2024): workflows versus agents, and the advice to start simple tested here.
• Qwen Team, Qwen2.5 Technical Report (2024), and the Qwen function-calling documentation for the <tool_call> format.