Recursive Language Models: The Prompt as an Environment

Long Context, and Where It Breaks measured how models lose things in long inputs, and Context Engineering managed an agent’s context by clearing, compacting and remembering. Both treat the context window as the place the input has to go. A recursive language model (RLM; Zhang, Kraska and Khattab, 2025) starts from the opposite assumption: the long input never enters the model at all. It’s stored as a variable in a programming environment, and the model writes code to look at it, and calls itself on the parts it wants read.

Table of contents

Context rot, and why bigger windows don’t settle it

The key-value task of Long Context, and Where It Breaks lost facts in the middle of 8,000 tokens; the needle task at the same lengths didn’t. The difference was what the model had to do with the text. The RLM paper makes that precise by sorting tasks by how the work grows with input length:

• Constant: find one needle. The answer depends on a fixed amount of the text however long it gets. Frontier models now pass these at a million tokens.
• Linear: label every line and aggregate (the paper’s OOLONG benchmark). The answer depends on nearly all of the text.
• Quadratic: aggregate over pairs of lines (OOLONG-Pairs). The answer depends on nearly all pairs.

On GPT-5 (a 272K-token window), the paper’s Figure 1 shows the constant task holding up as inputs grow from 2¹³ to 2²⁰ tokens, and the linear and quadratic ones degrading earlier and faster. On OOLONG-Pairs, GPT-5 and Qwen3-Coder-480B scored an F1 of at most 0.1%. A bigger window lets more text in; it doesn’t let the model do more work per token of it. That degradation with length is what the paper, after Hong et al. (2025), calls context rot.

The common general-purpose fix is compaction (see Context Engineering): summarize when the context gets long. It works when early details can be dropped; it fails exactly on the linear and quadratic tasks, where every detail might matter.

The idea: put the prompt in the environment

An RLM is a scaffold with the same interface as a model, a string in and a string out. Inside:

1. Start a REPL (a persistent Python session) and store the whole prompt in it as a variable, context.
2. Give the model only metadata: the length, a short prefix, how to access it.
3. Loop: the model writes code; the code runs; the model sees only a truncated piece of what the code printed. Variables persist between turns.
4. The code can call a function, llm_query in the paper’s implementation, that runs a fresh model call on any string the code builds: a slice of context, a list of extracted lines, a question.
5. The loop ends when the model gives a final answer. In the paper’s implementation it writes FINAL(answer), or FINAL_VAR(name) to return a variable it built in the REPL.

Recursion is step 4: the model calls a model, inside a program it wrote. With depth 1 the sub-call is a plain model call; with depth 2 or more, the sub-call is itself an RLM with its own REPL.

This is an agent loop (From Function Calling to the Agent Loop) whose one tool is a code interpreter. What makes it different from a CodeAct agent is where the data lives.

Three design choices that matter

The paper contrasts its algorithm with a “deceptively similar” one that has a code tool, a search tool, a sub-model tool and a finish action. It names three differences:

1. A symbolic handle to the prompt. The similar design puts the prompt in the model’s context, and so inherits the window’s limits and falls back to compaction. The RLM never copies the prompt into the window; it holds a name for it.
2. The answer comes from the environment. If the model must write its answer as its own output, the answer can’t be longer than the model’s context window. An RLM can build its answer in a variable, stitched together from many sub-calls.
3. Sub-calls from code. A sub-model tool lets the model delegate a few tasks it writes out by hand, one tool call each. A sub-model function inside the REPL can be called in a loop over a thousand slices. This is what makes linear and quadratic work possible: the model writes for line in lines: llm_query(...) instead of performing each call itself.

The third is the core. It moves the loop over the data out of the model’s token stream and into a program.

What the paper measured

On four long-context tasks (BrowseComp-Plus over 1,000 documents, OOLONG, OOLONG-Pairs and LongBench-v2 CodeQA), with GPT-5 as the root model and GPT-5-mini for sub-calls, the RLM’s scores were higher than compaction’s by a median of 26%, CodeAct with sub-calls’ by 130%, and Claude Code’s by 13%, at comparable average cost. Some particulars:

• On BrowseComp-Plus (6–11 million input tokens), the RLM averaged $0.99 a question, against an extrapolated $1.50–$2.75 for GPT-5-mini simply reading it all, and beat the compaction and retrieval baselines by over 29%.
• On OOLONG-Pairs, F1 went from at most 0.1% to 58.0% (GPT-5) and 23.1% (Qwen3-Coder).
• The REPL matters most for length; sub-calls matter for density. Depth 0, the REPL with no sub-calls, already scaled past the window; on CodeQA with Qwen3-Coder it beat every sub-calling variant. On the information-dense tasks, sub-calls were necessary: without them the model fell back on keyword heuristics.
• Costs are skewed. The median RLM run was cheaper than the median base-model run, but the mean was higher, pulled up by outlier runs where the RLM struggled to find an answer.
• Training helps a small model. Qwen3-8B fine-tuned on 1,000 filtered trajectories of Qwen3-Coder-480B acting as an RLM, on unrelated tasks (RLM-Qwen3-8B), improved on the untrained Qwen3-8B by a median of 28% across the four tasks.

The appendix of things that didn’t work is as useful as the results:

• Models without enough coding skill struggled as RLMs; Qwen3-8B, before training, is the paper’s example.
• One system prompt didn’t fit all models: Qwen3-Coder needed an extra line telling it not to make a sub-call on everything, or it launched thousands for simple tasks.
• Telling a final answer from a thought was brittle; models sometimes returned their plan as the answer.
• Blocking, sequential sub-calls made runs slow.

A miniature, run on this blog

The setup, small enough to read in one sitting:

• Input: every post on this blog as prose (code, math and links removed), concatenated: about 991,000 characters, roughly 256,000 tokens. That’s far beyond what a 3B model on a laptop can read at once. This post and three others that describe similar experiments are left out.
• Root model: Qwen2.5-3B-Instruct, a general instruct model rather than a coding one, greedy, at most 8 turns, each turn’s printed output cut to 2,000 characters. The REPL prints the value of a final bare expression, as Python’s own does.
• System prompt: a shortened version of the paper’s: the context’s length and format, what llm_query is for, the advice to split the context and query each piece, and the paper’s own example, a loop that asks each chunk for “the magic number.” It doesn’t mention anything the tasks ask about.
• Sub-calls: llm_query(text) runs the same model on up to 20,000 characters (about 5,000 tokens of this text, to keep memory in bounds), and returns an error for anything longer.
• Tasks: 10, all checkable by a program:
  • 5 needle tasks, each with a code like cobalt-512 planted in a random post;
  • 5 counting tasks: how many posts mention a term in their prose (a linear task).
• Two comparisons:
  • the model reading the input directly, cut to its first 8,000 tokens;
  • a fixed program with the same sub-calls: map llm_query over 19,000-character chunks, leaving room for the question (needles), or over every post in pieces of that size (counts), and combine. It’s what a good RLM might write, but written by me, with each sub-call’s reply capped at a few tokens. A needle counts as found only if exactly one distinct code comes back, the right one.

The prose comes from clean, from the first code block of Building RAG, which sits in the folder as common.py: it strips code, math and components from a post and reduces links to their text. The posts are a snapshot of the blog at commit 73b66ae, unpacked from blog-73b66ae.tar.gz. On an Apple M4 laptop the whole run takes about an hour and a half, almost all of it the fixed program’s sub-calls.

The heart of rlm.py is the sub-call, the system prompt and the REPL loop:

SUB = []                                                  # every sub-call's question and reply
def llm_query(text, max_new=300):
    if len(text) > 20000:                                 # ~5k tokens: keeps memory in bounds on this laptop
        return f"error: llm_query takes at most 20,000 characters; this text has {len(text):,}"
    reply = chat([{"role": "user", "content": text}], max_new=max_new)
    SUB.append({"asked": text[-160:], "reply": reply[:160]})
    return reply

SYSTEM = """You are answering a query about a context stored in a Python REPL as the variable `context`, a string of {n:,} characters (a blog's posts, each starting with a line '=== POST: <title> ==='). You cannot see it directly. Write Python in ```python blocks; you see only what you print, cut to 2,000 characters. Variables persist between turns, so use them as buffers. The REPL also has llm_query(text), which asks another model about up to 20,000 characters of text; use it when you need to understand meaning, not just match strings. Make sure your code covers the whole context before you answer.

A good strategy is to look at the context first, decide how to split it, query an LLM per piece with a specific question, collect the answers, and combine them. For example, to find a magic number in a long context:
```python
answers = []
for i in range(0, len(context), 19000):
    answers.append(llm_query("What is the magic number in this text? Reply NONE if there is none.\\n" + context[i:i + 19000]))
print([a for a in answers if "NONE" not in a])
```
When you are done, reply with FINAL(your answer), or FINAL_VAR(name) to return a REPL variable, and nothing else."""

LOG = []
def run(code, env):
    """Execute like a REPL: the value of a final bare expression is printed."""
    tree = ast.parse(code)
    last = tree.body.pop() if tree.body and isinstance(tree.body[-1], ast.Expr) else None
    exec(compile(tree, "<repl>", "exec"), env)
    if last is not None:
        value = eval(compile(ast.Expression(last.value), "<repl>", "eval"), env)
        if value is not None:
            print(repr(value))

def rlm(context, question, max_turns=8):
    env = {"context": context, "llm_query": llm_query, "re": re}
    messages = [{"role": "system", "content": SYSTEM.format(n=len(context))}, {"role": "user", "content": question}]
    LOG.append(messages)                                  # every turn is kept for inspection
    for _ in range(max_turns):
        reply = chat(messages)
        messages.append({"role": "assistant", "content": reply})
        m = re.search(r"FINAL(_VAR)?\((.*?)\)\s*$", reply.strip(), re.S)
        if m:
            return str(env.get(m.group(2).strip(), "")) if m.group(1) else m.group(2).strip()
        code = "\n".join(re.findall(r"```(?:python|repl)\n(.*?)```", reply, re.S))
        buf = io.StringIO()
        try:
            with contextlib.redirect_stdout(buf):
                run(code, env)
            out = buf.getvalue() or "(no output)"
        except Exception as e:
            out = f"error: {type(e).__name__}: {e}"
        messages.append({"role": "user", "content": "Output:\n" + out[:2000]})
    return None

The rest of the script loads the model, builds the corpus and the ten tasks, and runs the two comparisons (direct and scripted) beside the RLM on each task, saving every turn and sub-call to rlm_transcripts.json.

Full script: rlm.py

corpus: 66 posts, 991,185 characters (~255,743 tokens)

task         truth           direct              RLM  calls   sub            fixed program  calls  minutes: direct, RLM, fixed
needle   ochre-876 wrong: Based on           wrong:       5     2   right: 2 hits, 1 codes     53  0.4, 0.5, 9.3
needle   amber-365 wrong: Based on           wrong:       5     2   right: 2 hits, 1 codes     53  0.4, 0.5, 8.6
needle   ochre-514 wrong: Based on           wrong:       5     2   right: 2 hits, 1 codes     53  0.4, 0.5, 8.6
needle   ochre-466 wrong: Based on           wrong:       5     2   right: 2 hits, 1 codes     53  0.4, 0.5, 8.6
needle  cobalt-388 wrong: Based on           wrong:       5     2   right: 2 hits, 1 codes     53  0.4, 0.5, 8.6
count           36         wrong: 3            right      2     0                right: 36     73  0.3, 0.1, 8.2
count           34         wrong: 3       wrong: 147      3     0                wrong: 32     71  0.3, 0.2, 8.1
count           14         wrong: 3            right      3     0                wrong: 12     77  0.3, 0.1, 8.3
count           14         wrong: 2       wrong: 144      3     0                wrong: 10     80  0.3, 0.2, 8.4
count           37         wrong: 3          wrong:       8     0                wrong: 25     75  0.3, 0.3, 8.2

right: direct 0 of 10, RLM 2 of 10, fixed program 6 of 10

The fixed program found every needle and the RLM none; on the counts the RLM got two right and the fixed program one. Neither number says much on its own; the transcripts do.

Reading directly found nothing. The first 8,000 tokens are about 3% of the corpus, and for every needle the model’s reply began “Based on…”, saying in effect that the text held no code, and guessed small numbers for the counts.

On the needles, the RLM delegated, but only the beginning. Given the paper’s advice (split the context, query each piece) and its example loop, it wrote a sub-call, on the first 10,000 characters, about 1% of the text. An excerpt of one needle transcript, from the rlm_transcripts.json the script saves, with long strings cut at ”…”:

assistant | llm_query("What is the archive access code mentioned in the text?\n" + context[:10000])
output    | 'Based on the provided text, there is no specific mention of an "archive access code." ...'
assistant | llm_query("Could you specify the part of the text where the archive access code is mentioned? ..."
          |           + context[:10000])
output    | "Based on the provided text, there is no explicit mention of an archive access code. ..."
assistant | FINAL_VAR(access_code)

It asked the same slice twice, then returned a variable it had never created. All five needle transcripts are identical (greedy decoding, the same first slice, code that doesn’t depend on the code word), so this is one behavior, not five independent tries. The system prompt says to cover the whole context and shows a loop that does; the model wrote the loop’s body without the loop.

On the counts, it never delegated, and it got the code right twice. For “KV cache” it wrote the right program:

matches = 0
for post_start in context.split('=== POST: ')[1:]:
    if 'kv cache' in post_start.lower():
        matches += 1
matches

That answered 36, exactly, in a few seconds and with no sub-calls, and the same idea in one line got “tokenizer” right. The other three went wrong in ways a programmer would catch at a glance: two counted every occurrence instead of every post that mentions the term (147 and 144, against 34 and 14), and one built a regex “pattern” as a tuple, then repeated the same failing code, between “FINAL 14” lines written without their parentheses, until the turns ran out. Three runs also called FINAL() inside the code, where it isn’t defined, and recovered on the next turn.

That’s the paper’s result in miniature. Its REPL-only RLM (depth 0) already handled long inputs on some tasks, because the work was a program, and this model wrote that program correctly two times in five. And it’s the paper’s negative result exactly: Qwen3-8B, untrained, “struggled without sufficient coding abilities,” and this is a general instruct model of under half that size.

The fixed program found all five needles, and undercounted four of the five counts. Mapping llm_query over 19,000-character chunks returned exactly one code each time, the right one (plus one off-task reply, “Sure, I can help with that…”), because “archive access code” is a distinctive phrase. Asking each post “does this text mention ‘GPU’?” was harder: on prose alone, with code stripped, the sub-calls said no for some posts that do (32 of 34, 12 of 14, 10 of 14, 25 of 37), and hit “KV cache” exactly, though a total can be right while individual replies are wrong. A one-line string check gets all five; a 3B model reading 19,000 characters for one word misses some.

The right tool differed by task: a sub-call per chunk for the needle, which has to be read for; plain code for the counts, which only have to be matched. Choosing between them per question is what an RLM is for, and it’s what this root model did inconsistently: it chose code for the counts and a sub-call for the needle, and then implemented each one only partly.

What it costs

The corpus is about 256,000 tokens, too long for this model’s window and slow to read in one pass on any model. Per question, from the timings in the output:

• direct: one call, 8,000 tokens in, about 20 seconds, wrong;
• the RLM: 2 to 8 calls of a few hundred tokens each, plus at most two sub-calls of about 3,000 tokens, about half a minute or less; right twice, where it wrote the right code;
• the fixed program: 53 sub-calls of about 5,000 tokens for a needle, the whole corpus read once, about 9 minutes; 71 to 80 for a count, about 8 minutes, run one call at a time.

That’s the general shape. A program that maps a model over every piece costs about what reading everything costs, split into calls that fit, and it buys coverage, not savings. An RLM saves only when its code narrows the work first, and here the cheapest correct answer was the one-line program that needed no model calls at all. The paper reports its costs as comparable to the baselines on average and skewed: the median RLM run cost less than the median base-model run, while outlier runs where the RLM struggled to find an answer pulled the mean up; my reading is that those are runs that never found a narrowing step. Running sub-calls concurrently cuts the time, not the tokens; the paper ran its own one at a time and notes that this made its experiments slow.

Failure modes

• The root can’t program. Everything rests on the root model writing sensible code over a string it can’t see. Below a certain skill this fails outright, as in the run above and in the paper’s Qwen3-8B.
• The first decomposition decides a lot. The paper found that RLMs often recover from a bad first plan, but that the first attempt still drives the result, and that a worked example in the system prompt improves it even when the example is from an unrelated task.
• Runaway sub-calls. A loop over every line is easy to write and expensive to run. Cap the number of sub-calls and the total tokens, as with any agent (see Engineering Around the Model).
• Errors compound with depth. The paper’s Qwen3-Coder made more syntax errors than GPT-5, and deeper recursion did worse for it on average: each sub-RLM can fail the same way.
• Executing model-written code. The REPL runs whatever the model writes, on data that may contain injected instructions (see Guardrails and Prompt Injection). Run it in a sandbox with no network or secrets.
• Ending. A loop that stops on a text pattern can stop on the wrong text.

RLMs, chunking and RAG

• Chunking (map a prompt over fixed pieces, then combine) is an RLM whose program is fixed in advance. It’s cheaper and predictable, and the run above shows how far it gets. The RLM’s addition is choosing the program per question: filter with a regex first, read only the matching pieces, change strategy when a step comes back empty.
• RAG (Building RAG) picks passages by similarity before the model sees anything. It’s fast and cheap, and it can only find what looks like the question. An RLM can do retrieval as one step of a program, and also the exhaustive scans that RAG can’t (count every post that mentions X). Beyond Vector RAG’s agentic RAG sits between them: the model decides what to retrieve, but the corpus still reaches it only through a search tool.
• Compaction discards detail to fit the window; an RLM keeps everything outside the window and reads it as needed.

Where this leads

• Giving Agents Tools and Knowledge gives an agent its tools and procedures through MCP and Skills, and keeps procedures out of the context until they are needed (Skills), as the REPL keeps the input out.
• Many Agents splits a task across agents with their own contexts, the multi-agent form of a sub-call that reads one piece and returns a summary.
• Engineering Around the Model covers the budgets, stopping rules and checks that an RLM’s loop and its sub-calls need.

Takeaways

• An RLM keeps the long input out of the model: the input is a variable in a REPL, the model writes code to inspect it, and code can call the model on any piece.
• What makes it work is calling sub-models from code, in loops, rather than as hand-written tool calls. That’s what makes linear and quadratic work over the input possible.
• In the paper, a frontier root model beat compaction, CodeAct and Claude Code on long-context tasks at comparable average cost; the REPL alone handled length, and sub-calls handled density.
• On this blog, with a 3B root model and a prompt modeled on the paper’s, the RLM answered 2 of 10: it wrote correct code for two counts, and otherwise covered too little of the text or wrote code that didn’t run. A fixed map-over-chunks program found all 5 planted codes, at the cost of reading the whole corpus per question, and still miscounted 4 of 5 counts that one line of code gets right.
• The weak point is the root model’s programming. A fixed decomposition is the safe default; let the model write the program when questions vary too much for one program to fit.

References

Start with the RLM paper, especially its three design choices and its appendix of what didn’t work, then the authors’ rlm library; Chroma’s Context Rot report shows the problem it starts from.

• Zhang, Kraska and Khattab, Recursive Language Models (2025; v3, 2026), and the authors’ rlm library: the method, its three design choices, the results and the appendix of what didn’t work.
• Hong, Troynikov and Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma, 2025): the degradation with input length that the RLM paper starts from.
• Bertsch et al., Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities (2025): the OOLONG benchmark of linear aggregation tasks, which the paper uses and extends to pairs.
• Hsieh et al., RULER: What’s the Real Context Size of Your Long-Context Language Models? (2024): background on synthetic long-context tests, needle retrieval among them.
• Wang et al., Executable Code Actions Elicit Better LLM Agents (CodeAct, 2024): the code-acting agent the RLM is compared against.