Reasoning Models and Test-Time Compute

Every model so far spent the same compute on an easy question as on a hard one: one forward pass per generated token, and about as many tokens either way. A reasoning model doesn’t. Before it answers, it generates a long chain of intermediate reasoning, often thousands of tokens, and the harder the problem, the longer the chain. Generated tokens are compute, spent at test time, when the question is asked, instead of at training time.

Table of contents

Test-time compute

There are two ways to spend more compute on a question after training:

• Longer chains. Asking a model to reason step by step before answering (chain-of-thought prompting, Wei et al., 2022) improves multi-step problems, because each generated step becomes context for the next: the model can compute intermediate results it couldn’t hold in one forward pass.
• More chains. Sample several and combine them: take the majority answer (self-consistency, Wang et al., 2023), or score each with a verifier and keep the best (best-of-$N$).

OpenAI’s o1 (2024) made the first one a trained ability: its accuracy rose steadily with the amount of reinforcement learning and, separately, with the amount of thinking it was allowed at test time, both on a log scale. Snell et al. (2024) studied the trade directly: on problems a smaller model can sometimes solve, spending test-time compute well can beat a model 14 times larger given the same total compute (when the model serves few queries relative to its pretraining, so inference compute is the smaller budget), while on the hardest problems, extra thinking helps little and a larger model wins.

How RL produces long chains

GRPO and Reinforcement Learning from Verifiable Rewards’s GRPO with a checkable reward is enough. DeepSeek-R1-Zero was trained from a base model with only a correctness reward and a format reward; its prompt told it to think before answering, never for how long. Its answers grew from hundreds of tokens to thousands over training. A plausible mechanism, not one the paper isolates: on hard problems, the samples that happened to check their work, or try another approach, were the ones that got the reward, and GRPO made them more likely. The DeepSeek-R1 paper describes an “aha moment”, a point where the model began to stop and re-evaluate its own steps.

Distilling reasoning into small models

DeepSeek-R1’s authors found that distilling a large reasoning model’s chains into a smaller one beat running RL on the smaller one. They fine-tuned Qwen2.5 and Llama models on about 800,000 samples curated with R1 (about 600,000 reasoning traces, plus 200,000 others), plain supervised fine-tuning, which is Knowledge Distillation’s sequence-level distillation. The 32B distilled model scored 72.6% on AIME 2024; the same 32B base trained with large-scale RL directly (their R1-Zero-Qwen-32B, over 10,000 RL steps) reached 47.0%. Even the 1.5B distilled model, built on Qwen’s math-specialized 1.5B base, reached 28.9%. The paper’s own caution is that RL on small models “requires enormous computational power and may not even achieve the performance of distillation”. A small model can learn to reason from examples of reasoning much more cheaply than it can discover reasoning by trial and error.

Thinking budgets

Since the chain’s length is the cost, APIs let the caller bound or steer it. Some take a token budget: Anthropic’s budget_tokens on its older thinking models, a target the model may stop well short of, while the request’s maximum output stays the hard ceiling. More now take a coarse effort level, which the model interprets: OpenAI’s reasoning_effort, and the effort setting of Anthropic’s newer adaptive thinking, which has replaced fixed budgets on its latest models. A model trained with RL tends to use what it needs, more for harder questions; cutting a chain short usually costs accuracy, and research methods such as s1’s budget forcing cut a chain off, or extend it, at decode time to fit a given length.

What thinking costs

Every thinking token is a decoded token: one forward pass, which reads the weights and the whole KV cache so far (The KV Cache). Worked for two real models:

# What thinking tokens cost, for one request on two real configurations
# (from their config.json files):
#   Qwen2.5-32B: 64 layers, 8 key/value heads of 128 (GQA, "The KV Cache"), BF16;
#   DeepSeek-R1 (671B, 37B active): 61 layers of multi-head latent
#   attention, caching a 512-number latent plus a 64-number RoPE key per
#   token per layer ("How LLM Architecture Evolved"), BF16.
# Decode time, batch 1, bandwidth-bound ("Small and Specialized Models"): each token reads the
# active weights plus the whole KV cache so far, at one H100's 3.35 TB/s.
# Qwen2.5-32B fits on one H100; DeepSeek-R1 needs many GPUs to hold its
# 671B parameters, so its row only illustrates the reading: its active
# weights and cache at one GPU's bandwidth, without the communication
# between GPUs that a real deployment adds ("Mixture of Experts").
H100 = 3.35e12
models = {
    "Qwen2.5-32B": dict(kv=2 * 64 * 8 * 128 * 2, weights=32.5e9 * 2),
    "DeepSeek-R1": dict(kv=61 * (512 + 64) * 2, weights=37e9 * 1),   # FP8 weights, the format it was trained in
}
print(f"{'':12s} {'KV per token':>13s}   {'thinking tokens':>15s} {'KV cache':>9s} {'decode time':>12s} {'avg tokens/s':>13s}")
for name, m in models.items():
    for n in (1_000, 10_000, 50_000):
        # time = sum over tokens of (weights + KV so far) / bandwidth
        seconds = (n * m["weights"] + m["kv"] * n * (n - 1) / 2) / H100
        print(f"{name:12s} {m['kv'] / 1024:9.1f} KiB   {n:15,d} {m['kv'] * n / 2**30:7.2f} GiB {seconds:10.1f} s {n / seconds:13.0f}")

Full script: cost.py

              KV per token   thinking tokens  KV cache  decode time  avg tokens/s
Qwen2.5-32B      256.0 KiB             1,000    0.24 GiB       19.4 s            51
Qwen2.5-32B      256.0 KiB            10,000    2.44 GiB      197.9 s            51
Qwen2.5-32B      256.0 KiB            50,000   12.21 GiB     1068.0 s            47
DeepSeek-R1       68.6 KiB             1,000    0.07 GiB       11.1 s            90
DeepSeek-R1       68.6 KiB            10,000    0.65 GiB      111.5 s            90
DeepSeek-R1       68.6 KiB            50,000    3.27 GiB      578.5 s            86

Ten thousand thinking tokens on a 32B model hold 2.4 GiB of KV cache for that one request and take over three minutes at batch 1 on an H100 SXM. At 50,000 the cache reaches 12 GiB, and reading it slows every token: by the end, about 43 tokens a second by the same formula, against 51 at the start (47 on average). DeepSeek-R1’s multi-head latent attention (How LLM Architecture Evolved), introduced in DeepSeek-V2 to shrink the cache for inference generally, holds about a quarter as much per token, a 671B mixture-of-experts model against a 32B dense one, which matters more the longer the output. In serving terms, a reasoning request holds memory that could have served many short requests, for minutes, and the bill grows with the chain.

An experiment: thinking on checkable problems

The two kinds of test-time compute from the start of this post, measured on a 1.5B model, Qwen2.5-1.5B-Instruct, on problems with checkable answers: 100 three-step arithmetic word problems, each answered with just the number, with a written chain of reasoning, and with 16 sampled chains combined by majority vote. Two seeds for the sampled chains. On an Apple M4 laptop the two seeds took 2,570 and 1,828 seconds, as printed below, and all of this post’s code runs in about 86 minutes:

GEN = dict(do_sample=False, repetition_penalty=1.0, top_k=None, top_p=None, temperature=None, max_new_tokens=320)
SAMPLE = dict(do_sample=True, temperature=0.7, top_p=1.0, top_k=0, repetition_penalty=1.0, max_new_tokens=320,
              num_return_sequences=16)
res = {"direct": [], "chain": [], "samples": []}
tokens = {"direct": [], "chain": [], "samples": []}
t0 = time.perf_counter()
for q, ans in problems:
    if seed == 0:                                                   # greedy answers don't depend on the seed
        for kind, prompt in (("direct", q + " Answer with just the number."),
                             ("chain", q + " Think step by step, then give the final answer as 'Answer: <number>'.")):
            x = chat(prompt)
            out = m.generate(x, **GEN)[0, x.shape[1]:]
            res[kind].append(number(tok.decode(out, skip_special_tokens=True)) == ans)
            tokens[kind].append(int((out != tok.pad_token_id).sum()))
    x = chat(q + " Think step by step, then give the final answer as 'Answer: <number>'.")
    outs = m.generate(x, **SAMPLE)[:, x.shape[1]:]
    res["samples"].append([number(tok.decode(o, skip_special_tokens=True)) for o in outs])
    tokens["samples"].append([int((o != tok.pad_token_id).sum()) for o in outs])
print(f"seed {seed}: {len(problems)} problems, {time.perf_counter() - t0:.0f} s")
if seed == 0:
    for kind in ("direct", "chain"):
        print(f"  greedy, {kind:6s}: {np.mean(res[kind]):.0%} right, {np.mean(tokens[kind]):.0f} tokens per answer")
S, truth = res["samples"], [a for _, a in problems]
print(f"  sampled chain, one sample: {np.mean([[s == t for s in ss] for ss, t in zip(S, truth)]):.0%} right, "
      f"{np.mean(tokens['samples']):.0f} tokens per answer")
for k in (1, 2, 4, 8, 16):
    vote = [Counter([s for s in ss[:k] if s is not None]).most_common(1) for ss in S]
    acc = np.mean([bool(v) and v[0][0] == t for v, t in zip(vote, truth)])
    any_right = np.mean([t in ss[:k] for ss, t in zip(S, truth)])
    print(f"  majority of {k:2d}: {acc:.0%} right ({k * np.mean(tokens['samples']):5.0f} tokens per question); "
          f"some sample right: {any_right:.0%}")

Full script: tts.py

The rest of tts.py builds the 100 problems from four templates with a fixed seed and loads the model from lm.py. The two seeds come from tts_seeds.py, which runs tts.py 0 and then tts.py 1.

seed 0: 100 problems, 2570 s
  greedy, direct: 1% right, 3 tokens per answer
  greedy, chain : 100% right, 215 tokens per answer
  sampled chain, one sample: 97% right, 178 tokens per answer
  majority of  1: 97% right (  178 tokens per question); some sample right: 97%
  majority of  2: 98% right (  357 tokens per question); some sample right: 100%
  majority of  4: 100% right (  713 tokens per question); some sample right: 100%
  majority of  8: 100% right ( 1426 tokens per question); some sample right: 100%
  majority of 16: 100% right ( 2853 tokens per question); some sample right: 100%
seed 1: 100 problems, 1828 s
  sampled chain, one sample: 98% right, 175 tokens per answer
  majority of  1: 99% right (  175 tokens per question); some sample right: 99%
  majority of  2: 99% right (  351 tokens per question); some sample right: 100%
  majority of  4: 100% right (  702 tokens per question); some sample right: 100%
  majority of  8: 100% right ( 1404 tokens per question); some sample right: 100%
  majority of 16: 100% right ( 2808 tokens per question); some sample right: 100%

(One difference from the harder run below: this script reads the last number of a reply even without “Answer:”, which is lenient toward the chain arms; the harder run requires it.)

The chain did the work. Answering with just the number, the model got 1 problem in 100 right; writing out its reasoning first, all 100, at 215 tokens an answer instead of 3. That’s the whole idea of a reasoning model at small scale: the extra tokens are where the computation happens.

On these problems, more chains had nothing left to add. A greedy chain was already right every time. Sampling at temperature 0.7, which voting needs, cost a few points (97–98% per sample), and a majority of four samples won them back, at four times the tokens. Self-consistency can only help where single chains disagree, so the next test uses harder problems: Chain-of-Thought and Prompt Chaining’s serial arithmetic with eight dependent operations, where one greedy chain is right 92% of the time. The script reuses that post’s problem generator: the folder holds a copy of that post’s experiment script, serial_steps.py, saved here as chain.py, and tts_hard.py runs the part of it that makes the problems. It prints one line per problem, problem N: c/16 samples right; those 40 lines are left out of the output below, and hard_stats.py reads them back from the saved log, tts_hard.out.

src = open("chain.py").read()
exec(src[src.index("N = int("):src.index("m = model()")])          # the problem generator of "Chain-of-Thought and Prompt Chaining", same seed
probs = problems[8]
m = model()
SAMPLE = dict(do_sample=True, temperature=0.7, top_p=1.0, top_k=0, repetition_penalty=1.0)
def number(text):
    if "Answer:" not in text:
        return None
    import re
    nums = re.findall(r"-?\d+", text.split("Answer:")[-1].replace(",", ""))
    return int(nums[-1]) if nums else None
torch.manual_seed(0)
answers, tokens = [], []
for start, steps, answer in probs:
    prompt = (f"Start with {start}. " + " ".join(f"Then {s}." for s, _, _ in steps) +
              " Work it out step by step, then give the final result as 'Answer: <number>'.")
    x = tok.apply_chat_template([{"role": "user", "content": prompt}], add_generation_prompt=True,
                                return_tensors="pt", return_dict=True)["input_ids"].to(DEVICE)
    out = m.generate(x.repeat(16, 1), max_new_tokens=380, **SAMPLE)
    gen = out[:, x.shape[1]:]
    answers.append([number(tok.decode(g, skip_special_tokens=True)) for g in gen])
    tokens.append(int((gen != tok.pad_token_id).sum()) / 16)
    print(f"problem {len(answers)}: {sum(a == answer for a in answers[-1])}/16 samples right", flush=True)

Full script: tts_hard.py

After the loop, tts_hard.py prints the majority-vote table and how the problems split by share of samples right.

40 problems, eight operations each; 202 tokens per sample
  majority of  1:  82% right (   202 tokens per question); some sample right:  82%
  majority of  2:  88% right (   403 tokens per question); some sample right:  90%
  majority of  4:  92% right (   807 tokens per question); some sample right:  95%
  majority of  8:  92% right (  1614 tokens per question); some sample right:  95%
  majority of 16:  92% right (  3227 tokens per question); some sample right:  98%
problems by share of samples right: none 1, under half 2, half or more 25, all 12

A caution on the script’s middle rows: “majority of $k$” votes over the first $k$ samples, and a tie goes to the earlier answer, so the rows for two and four samples depend on sample order (two samples is really “the first, or the second if the first gave no answer”). The order-free numbers are the plurality of all 16, and two computed from the per-problem counts in the log:

import re
from math import comb

# From tts_hard.out's per-problem counts (c of n = 16 samples right):
# the average accuracy of one sample, and the unbiased estimate of the
# chance that at least one of k samples is right (pass@k, Chen et al.,
# 2021): 1 - C(n - c, k) / C(n, k), averaged over problems. That is what a
# perfect verifier choosing among k samples would score.
counts = [int(m) for m in re.findall(r"^problem \d+: (\d+)/16", open("tts_hard.out").read(), re.M)]
n = 16
print(f"{len(counts)} problems; one sample right on average: {sum(counts) / (n * len(counts)):.1%}")
for k in (1, 2, 4, 8, 16):
    p = sum(1 - comb(n - c, k) / comb(n, k) for c in counts) / len(counts)
    print(f"  pass@{k:<2d} (some of {k:2d} samples right): {p:.1%}")

Full script: hard_stats.py

40 problems; one sample right on average: 86.6%
  pass@1  (some of  1 samples right): 86.6%
  pass@2  (some of  2 samples right): 93.1%
  pass@4  (some of  4 samples right): 94.9%
  pass@8  (some of  8 samples right): 96.2%
  pass@16 (some of 16 samples right): 97.5%

Voting bought back what sampling cost, and no more. One sampled chain was right 86.6% of the time, about six points below the greedy chain’s 92% (37 of 40). Part of that loss may be the sampling setting itself: pure temperature 0.7 over the whole vocabulary, not the narrower top-p and top-k the model’s defaults use. The plurality of all 16 samples was right on 37 of the 40 problems, 92%, back to the greedy chain and no further, at 16 times its tokens. On all but at most one of those 37, at least half the samples had the right answer: a rerun on an NVIDIA RTX 3090 (its saved outputs) also voted 37 of 40 right, one of them by a plurality below half. On the other three, most samples were wrong.

A verifier could have gone further. Some sample was right on 39 of the 40 problems by sixteen samples (pass@16, 97.5%), two problems more than the vote. Picking that sample needs something that can tell right from wrong, a checker or a trained verifier, which is best-of-$N$: here it would beat voting, and only because these answers can be checked. These are 40 problems and one seed, so each problem is 2.5 points. The paper results above, where test-time compute rivals a much larger model, come from models trained to reason and from problems hard enough that the first chain is often wrong; at 1.5B on arithmetic, the first lesson is simpler: let the model write its steps.

Is the chain the reasoning?

A chain of thought looks like an explanation, but nothing forces it to be the computation that produced the answer. Turpin et al. (2023) biased models with few-shot prompts in which the right answer was always option (A), and with other biasing features such as a suggested answer: the models’ answers shifted toward the bias, dropping accuracy by as much as 36%, while their written explanations rationalized the biased answers and systematically failed to mention the bias. A chain is evidence about how a model reached an answer, not a guarantee, which matters when chains are used to audit a model’s decisions.

When not to use one

A reasoning model is the wrong tool for Small and Specialized Models’s System One calls: lookups, classifications, extraction, short answers a small model gets right directly. It adds seconds to minutes of latency and many times the tokens, for problems that didn’t need multi-step work. Use one where steps compound: math, code, planning, and analysis with several dependent parts.

Where this leads

• Chain-of-Thought and Prompt Chaining turns to what goes into the context, starting with chains written by prompting alone, then Prompt Caching makes a repeated prefix cheap, and Context Engineering treats the context window as the model’s working memory.
• vLLM Architecture, KV Cache Compression and Speculative Decoding, Derived come back to serving long generations: batching, cache management and speculative decoding.

Takeaways

• Test-time compute is spending more tokens per question: longer chains, more chains, or both. Reasoning models are trained, by reinforcement learning on checkable rewards, to spend it.
• Small models learn to reason more cheaply by distillation from a large reasoning model’s chains than by reinforcement learning of their own.
• Every thinking token is a decoded token: it costs time, and its KV cache occupies memory for the whole request.
• On a 1.5B model, writing the reasoning took three-step problems from 1% to 100%. On harder eight-step problems, majority voting over sampled chains (one sample: 86.6%) reached the greedy chain’s 92% and no more; some sample was right on 97.5% of problems, which only a verifier can exploit.
• A chain of thought is evidence about how a model answered, not a guarantee; models can rationalize answers their chains don’t explain.
• Don’t use a reasoning model for System One work: lookups, classifications and short answers don’t need the extra seconds.

References

Start with Wei et al. for chain-of-thought and Wang et al. for self-consistency, then the DeepSeek-R1 paper for how RL produces long chains and why distillation wins at small sizes; Snell et al. covers when thinking longer beats a bigger model.

• Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (NeurIPS, 2022): chain-of-thought prompting, the first kind of test-time compute.
• Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR, 2023): self-consistency, majority voting over sampled chains.
• OpenAI, Learning to Reason with LLMs (2024): o1, with accuracy rising with RL and with thinking time.
• Snell, Lee, Xu and Kumar, Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024): when test-time compute beats a model 14 times larger, and when it doesn’t.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025): R1-Zero’s growing chains, and distillation beating RL for small models.
• Turpin, Michael, Perez and Bowman, Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (NeurIPS, 2023): biased prompts and chains that rationalize the biased answer.
• Muennighoff et al., s1: Simple Test-Time Scaling (2025): budget forcing.
• Chen et al., Evaluating Large Language Models Trained on Code (2021): the unbiased pass@k estimator.
• Anthropic, Extended thinking and OpenAI, Reasoning models (documentation, accessed October 2026): thinking budgets and effort levels.