Reasoning Models and Test-Time Compute followed models trained to reason at length. Long before those, people found that an ordinary model reasons better when asked to show its work. This post is about why that works, and about its engineering cousin: splitting a task into a chain of model calls, each small enough to check.
Writing the steps out in one call kept a 1.5B model at 92% on eight dependent operations. Splitting the steps across calls lost accuracy to compounding errors, and a check by self-agreement won none of it back.
- One forward pass is a fixed amount of computation. Writing intermediate results buys more passes, and puts each result in the context for the next step: chain-of-thought is computation bought with tokens. “Let’s think step by step” alone took text-davinci-002 from 17.7% to 78.7% on MultiArith.
- A chain splits a task into separate calls, each simpler and checkable, at the price of latency and errors that compound: 95% per step is 60% after ten steps.
- On a 1.5B model, answering directly failed at two dependent operations (18%); one reasoned call stayed at 92–100% up to eight, and matched or beat every chain.
- A check helps only as far as it can detect errors: accepting a step only when two samples agreed scored 68–72% at eight steps, the same as no check, at twice the calls.
Table of contents
Open Table of contents
One forward pass is a fixed-depth circuit
A Transformer computes each token with a fixed number of layers (The Residual Stream): however hard the question, the answer token gets roughly the same amount of computation, about 2 FLOPs per parameter plus attention over the context. A problem that needs dependent steps, where each step needs the previous one’s result, must fit all into those layers. With few layers and large , it can’t. Theory makes this precise, under standard complexity assumptions and as inputs grow: without intermediate tokens, a constant-depth Transformer can’t solve some problems that it can when allowed to write intermediate steps, such as evaluating arithmetic expressions (Feng et al., 2023) or tracking the state of a finite automaton (Merrill and Sabharwal, 2024).
Generated tokens change that. Each one is another full pass through the model, and it goes back into the context, where later tokens can read it. Writing “17 times 3 is 51” turns a value the model would have had to hold in its activations into text it can attend to. Chain-of-thought is serial computation bought with tokens.
The same layers, used once or used once per written step. Each intermediate result becomes text the next pass can read, instead of a value held in activations. The example is illustrative.
Chain-of-thought, zero-shot and few-shot
Wei et al. (2022) showed it with few-shot prompts: a handful of worked examples, each with its reasoning written out, before the question. Models of around 100B parameters and up then wrote their own reasoning and got much better at arithmetic and logic problems; smaller ones wrote fluent but illogical chains, and sometimes got worse. Kojima et al. (2022) found that no examples are needed: appending “Let’s think step by step” (zero-shot chain-of-thought) took InstructGPT (text-davinci-002) from 17.7% to 78.7% on the MultiArith word problems, and from 10.4% to 40.7% on GSM8K.
One sentence appended to the prompt, no examples: the model’s own written steps did the rest.
Data: Kojima et al. (2022)Self-consistency (Wang et al., 2023) samples several chains at a temperature above zero (Decoding) and takes the majority final answer: different chains make different mistakes, and right answers agree with each other more often than wrong ones do. Reasoning Models and Test-Time Compute measured it on checkable problems, along with what each extra chain costs.
On reasoning models (see Reasoning Models and Test-Time Compute), asking for chain-of-thought is mostly redundant: they were trained to reason before answering, and do it whether asked or not.
Prompt chaining
A chain splits a task into a sequence of separate model calls, each with its own prompt, where one call’s output becomes part of the next call’s input: extract the facts, then check them, then draft, then format. Compared with one long prompt that asks for everything:
- Each step is simpler, so a smaller or cheaper model can often do it.
- Each step can be checked before the next one runs: a parser, a schema, a test, a calculator, or another model call. A step that fails its check can be retried, or the chain stopped, before the error spreads.
- Each step can be inspected: when the final answer is wrong, the intermediate outputs say where it went wrong.
A chain of calls with a check between them. The checks are what a single long prompt can’t have: a place to catch an error before the next step builds on it.
The cost is latency, one round trip per step, and error propagation: a later step trusts what an earlier one produced. If every step is right with probability and errors are never caught, a chain of steps is right with probability about : 95% per step is 77% after 5 steps and 60% after 10. Validation between steps can break that product, as far as it catches the errors.
Uncaught errors multiply. Even 95% per step leaves a ten-step chain wrong two times in five.
An experiment: serial steps, six ways
serial_steps.py builds 40 problems for each length , each a start
number and operations to apply in order (“add 17”, “multiply by 3”),
and runs six arms on them: one call answering directly, one call reasoning
first, a chain of one call per operation answering directly or reasoning
first, and the direct chain with each step checked, by agreement between
two samples or against the true value. The per-step prompts, the chain and
the two checks:
STEP = "Take {x} and {s}. Answer with just the number."
STEP_R = "Take {x} and {s}. Work it out, then give the result as 'Answer: <number>'."
def chain(start, steps, step_fn, record=None):
x = start
for s, f, v in steps:
y = step_fn(x, s, f(x, v))
if record is not None:
record.append({"x": x, "op": s, "right": y == f(x, v)})
x = y if y is not None else x
return x
def agree(x, s, truth):
for _ in range(3): # accept a value two samples agree on
a, b = (number(ask(STEP.format(x=x, s=s), 12, sample=True)) for _ in range(2))
if a is not None and a == b:
return a
return a # no agreement: keep the last sample
def oracle(x, s, truth):
for attempt in range(4):
y = number(ask(STEP.format(x=x, s=s), 12, sample=attempt > 0))
if y == truth:
return y
return yThe rest of the file asks the model with every decoding parameter given
explicitly, scores each arm (the two sampled checks with three seeds), and
saves every greedy chained step to chain_steps.json for the error
analysis at the end, which step_errors.py runs.
k = 1
direct 100% 1.0 calls per problem
reasoned 100% 1.0 calls per problem
chained 98% 1.0 calls per problem
chained, reasoned 100% 1.0 calls per problem
checked by agreement 98%/98%/98% (3 seeds) 2.0 calls per problem
checked by oracle 98%/98%/98% (3 seeds) 1.1 calls per problem
k = 2
direct 18% 1.0 calls per problem
reasoned 100% 1.0 calls per problem
chained 95% 2.0 calls per problem
chained, reasoned 100% 2.0 calls per problem
checked by agreement 95%/95%/98% (3 seeds) 4.1 calls per problem
checked by oracle 98%/98%/95% (3 seeds) 2.1 calls per problem
k = 3
direct 2% 1.0 calls per problem
reasoned 100% 1.0 calls per problem
chained 100% 3.0 calls per problem
chained, reasoned 98% 3.0 calls per problem
checked by agreement 100%/100%/100% (3 seeds) 6.0 calls per problem
checked by oracle 100%/100%/100% (3 seeds) 3.0 calls per problem
k = 4
direct 2% 1.0 calls per problem, 1 hit the token cap
reasoned 98% 1.0 calls per problem
chained 92% 4.0 calls per problem
chained, reasoned 95% 4.0 calls per problem
checked by agreement 98%/95%/95% (3 seeds) 8.0 calls per problem
checked by oracle 98%/95%/95% (3 seeds) 4.2 calls per problem
k = 6
direct 0% 1.0 calls per problem
reasoned 98% 1.0 calls per problem
chained 80% 6.0 calls per problem, 2 hit the token cap
chained, reasoned 88% 6.0 calls per problem
checked by agreement 78%/78%/80% (3 seeds) 12.4 calls per problem
checked by oracle 82%/85%/85% (3 seeds) 6.5 calls per problem
k = 8
direct 2% 1.0 calls per problem
reasoned 92% 1.0 calls per problem
chained 70% 8.0 calls per problem, 3 hit the token cap
chained, reasoned 82% 8.0 calls per problem
checked by agreement 72%/68%/72% (3 seeds) 16.8 calls per problem
checked by oracle 72%/80%/82% (3 seeds) 8.8 calls per problem
one greedy chained step right: 97.1% overall; multiply 98.7%, add or subtract 96.3%; starting from a number of 1,000 or more 81.6% (49 steps)Six ways to do the same serial task, on Qwen2.5-1.5B-Instruct, with 40 problems at each length . On an Apple M4 laptop the whole run takes about 65 minutes. With 40 problems, a 95% interval around 70% runs from about 55% to 82%, so differences under about 15 points are within noise; read the table for its large gaps and its trends.
Direct answers collapse at the first dependent step; one reasoned call stays on top all the way. Every chain decays with length, and even a check that knew the true value didn’t catch up with the single reasoned call.
Data: the output aboveAsked for the number directly, the model fails as soon as one step depends on another. One operation: 100%. Two: 18%. Four or more: 0–2%. A 1.5B model can do “start with 23, multiply by 4” in one pass, but not compose a second operation on a result it never wrote down. That’s consistent with the serial-depth argument above, though the theory is about growing inputs, not two steps; what it shows here is a small model’s limit, not a proof about depth.
Asked to work step by step, in one call, it was nearly perfect: 100% up to three steps, 98% at four and six, 92% at eight. Writing each intermediate result gives every step a pass of its own and puts the result in the context for the next step to read.
Chaining did no better than reasoning in one call. A chain of direct
answers (each call sees the previous result and one operation, and
replies with a number) fell to 70% at eight steps; the same chain with
each call working its step out first reached 82%. Working each step out
was ahead at four, six and eight steps (95, 88 and 82% against 92, 80 and
70%), a consistent direction but within noise at 40 problems, and both
stayed at or below the single reasoned call’s 92%. On the eight-step
problems, the direct chain’s per-step accuracy, computed from the logged
steps in chain_steps.json (95.6%, from step_errors.py), raised to the
eighth power gives the chain’s 70%: consistent with independent errors
compounding.
Big numbers raise the error rate. A step that started from a number
of 1,000 or more was right 81.6% of the time, against 97.9% for steps
starting below 1,000, and long problems produce big intermediate values,
so per-step accuracy falls as grows. But big numbers aren’t most of
the errors: 19 of the 28 wrong steps started below 1,000, several of them
around zero or with negative numbers, in the steps logged to
chain_steps.json
(Full script: step_errors.py). A few failures
were also format, not arithmetic: five direct steps ran into the
12-token cap, and a reply with no number left the previous value in
place.
Checking each step helped only when the check knew the answer. Two checks were tried on the direct chain:
- Agreement, a check that doesn’t know the answer: sample each step twice and accept a value only if both agree, retrying up to three pairs. Over three seeds it scored 68–72% at eight steps, the same as no check, at twice the calls. Likely reasons, not separately measured: the model’s errors are consistent enough that two samples agree on the same wrong value, and the samples, drawn at temperature 0.7, are a little less accurate than the unchecked chain’s greedy answers.
- The oracle, the best case: compare each step with the true value and retry until right, up to four tries. It reached 72–82% over three seeds, for under one extra call per problem (8.8 calls against 8). That it fell short of 100% means some steps failed all four tries; the retries weren’t logged, but a step the model gets wrong systematically stays wrong when resampled.
So a check is only as good as what it can detect, and a model checking itself against itself detects little. For this task, the best design is one call that reasons step by step, and, since every step is arithmetic, a calculator (From Function Calling to the Agent Loop’s tools) would make it exact. A chain earns its cost when the steps are different jobs, need different prompts or models, or produce something that can be checked by something other than the model.
Where this leads
- Prompt Caching makes the long, repeated prefixes of chains and agents cheaper.
- From Function Calling to the Agent Loop turns chains into agents: the model chooses the next step itself, with tools.
Takeaways
- A forward pass is a fixed amount of computation; generated tokens add serial steps. Chain-of-thought is computation bought with tokens.
- Asked for the answer directly, a 1.5B model failed at two dependent operations (18%); asked to work step by step in one call, it was right 92–100% of the time up to eight.
- Prompt chaining splits a task into simpler, inspectable calls, at the price of latency and compounding errors. Here a chain of direct answers reached 70% at eight steps (82% when each call worked its step out), against 92% for one reasoned call.
- A check between steps helps only as far as it can detect errors: a check by agreement between two samples changed nothing, likely because the model repeats its own mistakes; even a perfect check left some steps wrong after four tries.
- Don’t chain what one call can do well; chain when the steps are different jobs, or when something other than the model can check them.
References
Start with Wei et al. and Kojima et al. for few-shot and zero-shot chain-of-thought, and Wang et al. for self-consistency; Feng et al. and Merrill and Sabharwal give the theory of why intermediate steps add computation.
- Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS, 2022): few-shot chain-of-thought, and its dependence on model scale.
- Kojima, Gu, Reid, Matsuo and Iwasawa, Large Language Models are Zero-Shot Reasoners (NeurIPS, 2022): zero-shot chain-of-thought, and the MultiArith and GSM8K numbers.
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR, 2023): self-consistency, the majority over sampled chains.
- Feng et al., Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective (NeurIPS, 2023): why constant-depth Transformers need intermediate steps for arithmetic expressions.
- Merrill and Sabharwal, The Expressive Power of Transformers with Chain of Thought (ICLR, 2024): what intermediate steps add, including tracking a finite automaton.