Long Context, and Where It Breaks: Lost in the Middle

A model’s context window is the number of tokens it accepts. Whether it uses them all equally is a different question, and the answer is no. Liu et al. (2024) gave models a set of documents, one of which held the answer to a question, and moved that document from the start of the prompt to the end. Accuracy was highest with the answer at the start or the end, and lowest in the middle: a U-shaped curve, lost in the middle. The models had the information in their context and didn’t find it.

This post reproduces the effect on a small model, shows why the most famous long-context test doesn’t see it, and works through the causes, each of which suggests a different fix.

Table of contents

The experiment

Two tasks, each at four prompt lengths, with the thing to find placed at five depths, from the very start (0%) to the very end (100%):

• Key-value lookup (after Liu et al.’s second task): a JSON object of random UUID keys and values, and a question asking for one key’s value. Every other pair is a distractor that looks exactly like the answer.
• Needle in a haystack: one out-of-place sentence, “The passphrase for the notebook assistant is blue-heron-417.”, inserted into this blog’s own prose, and a question asking for the passphrase.

The model is Decoding’s Qwen2.5-1.5B-Instruct, greedy, which accepts 32,768 tokens; its chat template also puts a short default system prompt first. A trial counts as right if the answer contains the exact value. Depth is measured among the pairs (or the words of prose), not in tokens. The prose is blog_train.txt, the frozen snapshot of the blog’s posts that Sequences Before Attention wrote, kept in the companion repo’s data/ folder. On an Apple M4 laptop the two tasks take about 24 and 10 minutes (the “s so far” column), the extra run below about 41, and all of this post’s code about 80.

position.py runs one task. It builds each trial’s prompt, asks the model through ask (the chat template and greedy decoding), and counts the trials right at each length and depth:

def kv_trial(n, depth, rng):
    pairs = [(str(uuid.UUID(int=rng.getrandbits(128))), str(uuid.UUID(int=rng.getrandbits(128)))) for _ in range(n)]
    key, value = pairs[round(depth * (n - 1))]
    data = "{" + ", ".join(f'"{k}": "{v}"' for k, v in pairs) + "}"
    return ask(f"Extract the value corresponding to the specified key in the JSON object below. "
               f"Answer with the value only.\n\nJSON data:\n{data}\n\nKey: \"{key}\"\nValue:"), value

def needle_trial(n, depth, rng):
    start = rng.randrange(0, len(blog_ids) - n)
    words = tok.decode(blog_ids[start:start + n]).split(" ")
    secret = f"{rng.choice(['blue', 'red', 'amber', 'grey'])}-heron-{rng.randrange(100, 1000)}"
    words.insert(round(depth * len(words)), f"(The passphrase for the notebook assistant is {secret}.)")
    return ask(f"Read the text below, then answer the question.\n\n{' '.join(words)}\n\n"
               f"What is the passphrase for the notebook assistant? Answer with the passphrase only."), secret

trial = kv_trial if TASK == "kv" else needle_trial
results = {}
t0 = time.perf_counter()
for n in SIZES:
    for depth in DEPTHS:
        rng = random.Random(f"{TASK}-{n}-{depth}{SALT}")
        right, lengths = 0, []
        for _ in range(TRIALS):
            (length, answer), value = trial(n, depth, rng)
            right += value in answer; lengths.append(length)
        results[(n, depth)] = (right, sum(lengths) / len(lengths))
    row = " ".join(f"{results[(n, d)][0]:>2}/{TRIALS}" for d in DEPTHS)
    print(f"{TASK:6s} ~{results[(n, 0.5)][1]:5.0f} prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: {row}"
          f"   ({time.perf_counter() - t0:.0f} s so far)", flush=True)

Full script: position.py

both_tasks.py runs it on the key-value task, then on the needle:

Full script: both_tasks.py

kv     ~  763 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: 12/12 12/12 12/12 11/12 12/12   (128 s so far)
kv     ~ 2106 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: 12/12 11/12 12/12 12/12 12/12   (340 s so far)
kv     ~ 4106 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: 12/12 11/12 10/12  8/12 12/12   (705 s so far)
kv     ~ 8151 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: 12/12 10/12  6/12 11/12 12/12   (1411 s so far)
needle ~  822 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%:  6/6  6/6  6/6  6/6  6/6   (40 s so far)
needle ~ 2072 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%:  6/6  6/6  6/6  6/6  6/6   (122 s so far)
needle ~ 4072 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%:  6/6  6/6  6/6  6/6  6/6   (269 s so far)
needle ~ 8072 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%:  6/6  5/6  6/6  6/6  6/6   (581 s so far)

The key-value lookup at 8,000 tokens loses things in the middle. To firm that up, and to see what happens further out, more_trials.py runs the same script with 12 more trials at 8,000 tokens (new random objects) and a 16,000-token length:

Full script: more_trials.py

kv     ~ 8142 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: 12/12  7/12  9/12  9/12 12/12   (936 s so far)
kv     ~16189 prompt tokens: right at depth 0 / 25 / 50 / 75 / 100%: 12/12  2/12 10/12  5/12 12/12   (2448 s so far)

Together:

(The 8,100-token line pools both runs, 24 trials per depth; the others have 12. The 2,100-token line is left out: it sits on the 800-token one.)

Three things stand out:

• The ends held. With the key first or last, the model was right in all 144 trials, at every length.
• The middle fails more as the prompt grows. Pooling the three middle depths: 97% right at 800 and at 2,100 tokens, 81% at 4,100, 72% at 8,100 and 47% at 16,200, half the model’s 32,768-token window.
• The shape within the middle isn’t settled. At 8,100 tokens the two runs (independent draws) disagree on which middle depth is worst: at 25%, 10 then 7 of 12; at 50%, 6 then 9. At 16,200 tokens the key at 25% was found 2 times in 12 and at 50% 10 times, a gap larger than chance, which this post doesn’t explain. What holds at every length is the U’s outline: ends that hold, and a middle that gets worse as the prompt grows. It’s the shape Liu et al. report for much larger models.

How does it fail? Logging the answers for the 12 objects of the 8,100-token, 50%-depth cell that scored 6 of 12, and asking the same questions with the key stated before the data as well as after. key_first.py regenerates those objects from the same seed, asks each question both ways with the same ask, and sorts every answer:

def close(a, b):                              # equal length and at most 2 characters differ
    return len(a) == len(b) and sum(x != y for x, y in zip(a, b)) <= 2

n, depth = 120, 0.5
rng = random.Random(f"kv-{n}-{depth}")
target = round(depth * (n - 1))
counts = {"after only": {}, "before and after": {}}
for trial in range(12):
    pairs = [(str(uuid.UUID(int=rng.getrandbits(128))), str(uuid.UUID(int=rng.getrandbits(128)))) for _ in range(n)]
    key, value = pairs[target]
    data = "{" + ", ".join(f'"{k}": "{v}"' for k, v in pairs) + "}"
    task = "Extract the value corresponding to the specified key in the JSON object below. Answer with the value only."
    for way, content in (("after only", f"{task}\n\nJSON data:\n{data}\n\nKey: \"{key}\"\nValue:"),
                         ("before and after", f"{task}\n\nKey: \"{key}\"\n\nJSON data:\n{data}\n\nKey: \"{key}\"\nValue:")):
        answer = ask(content)
        values = [v for _, v in pairs]
        if value in answer:
            kind = "right"
        elif any(v in answer for v in values):
            kind = f"another pair's value (pair {next(i for i, v in enumerate(values) if v in answer)}, target {target})"
        elif any(close(w.strip('"`.'), value) for w in answer.split()):
            kind = "right value, a character or two wrong"
        else:
            kind = f"none of these: {answer[:40]!r}"
        counts[way][kind] = counts[way].get(kind, 0) + 1
for way, c in counts.items():
    print(f"{way}: " + "; ".join(f"{k}: {v}" for k, v in sorted(c.items(), key=lambda kv: -kv[1])))

Full script: key_first.py

after only: right: 6; none of these: '57615999-1111-2222-3333-444444444444': 1; right value, a character or two wrong: 1; another pair's value (pair 40, target 60): 1; another pair's value (pair 55, target 60): 1; another pair's value (pair 52, target 60): 1; another pair's value (pair 50, target 60): 1
before and after: right: 12

Of the six misses, four were the value of another pair, 5 to 20 pairs before the target; one was the right value with a character wrong; one was made up. “Lost” mostly means found the wrong neighbor: the model located roughly the right region and then matched imprecisely. The second line is the subject of the last section.

Why the needle test is too easy

The needle task is the most famous long-context test, and here it’s nearly perfect: 119 of 120 right, at every length and depth up to 8,000 tokens. Long-context models advertise needle results at a million tokens with the same all-green grid. The key-value task, at the same lengths, loses things in the middle.

The difference is what the model has to tell apart. The needle is the only sentence about a passphrase, in prose about something else: a question about passphrases matches it and nothing else, so any attention head that compares the question with the text finds it. In the key-value task, every pair looks like the answer: same format, same length, random characters. The model has to match 36 characters of key exactly, among up to 240 candidates of identical form. Real long-context work looks more like the second task: find the clause that applies in a long contract, the right function among similar ones in a codebase, the one relevant paragraph among many on the same topic.

Benchmarks since have moved that way. RULER (Hsieh et al., 2024) adds needles with distractors, several needles, chains of references to follow, and counting and aggregation tasks; of the models tested, all of which claimed 32K tokens or more, only half stayed satisfactory at 32K. Context Rot (Hong et al., 2025) showed degradation with length across 18 models on tasks as simple as repeating a list of words, worse when the question resembles the answer less and when distractors are present, and, unexpectedly, worse when the irrelevant text is coherent than when it’s shuffled (a vendor’s technical report, which measures the effects without pinning down their causes). RULER also gives the effect a name: a model’s effective context length is the longest input at which it still scores as well as a reference model does at 4K tokens, and it’s typically far below the advertised window.

Causes

Softmax dilution

Attention spreads one unit of weight over every earlier token (The Transformer’s Three Attention Patterns). Suppose one key, the relevant one, scores $\delta$ above $n - 1$ others that all score the same. Its share of the attention is

which, for a fixed margin, falls as the context grows:

import math

# Softmax dilution: one relevant key scores delta above n - 1 others that all
# score the same. Its share of the attention is e^delta / (e^delta + n - 1).
for delta in (4, 8):
    shares = ", ".join(f"{n:>7,} keys {math.exp(delta) / (math.exp(delta) + n - 1):6.1%}" for n in (100, 1_000, 10_000, 100_000))
    print(f"score margin {delta}: {shares}")
print(f"margin needed for half the attention among 100,000 keys: ln(99,999) = {math.log(99_999):.1f}")

Full script: dilution.py

score margin 4:     100 keys  35.5%,   1,000 keys   5.2%,  10,000 keys   0.5%, 100,000 keys   0.1%
score margin 8:     100 keys  96.8%,   1,000 keys  74.9%,  10,000 keys  23.0%, 100,000 keys   2.9%
margin needed for half the attention among 100,000 keys: ln(99,999) = 11.5

A margin of 8 in the scores gives the relevant key three quarters of the attention among 1,000 keys and 3% among 100,000. To hold half the attention among 100,000 keys, it needs a margin of 11.5, so the queries and keys have to become sharper as the context grows, and a model trained mostly on short texts never had to learn that. With many similar distractors the problem is worse: they don’t score equally low, they score almost as high as the answer. The margin needed grows as $\ln n$, which is the link to Where Am I? Position in Attention, from Sinusoids to RoPE: a model with no position information lost a “seen before” task at four times its training length, and scaling the scores by $\log n$, at test time only, brought it back from 81% to 96%. Similar in spirit, some models scale their attention scores with the log of the length (Qwen’s LogN scaling), and YaRN adjusts the attention temperature when extending context.

Dilution explains why longer prompts are harder, but not why the middle is harder than the ends: the formula has no position in it. With the key first or last, this model kept enough margin at 16,000 tokens to be right every time, so something position-dependent has to sit on top of it.

Attention sinks

sinks.py reads the attention weights out of the model for a 1,024-token passage of the blog and measures, layer by layer, the share that lands on the first token, on the four most recent tokens, and on the rest:

# 32-bit floats: with the explicit ("eager") attention needed to read the
# weights out, 16-bit scores overflow in this model.
model = AutoModelForCausalLM.from_pretrained(PATH, dtype=torch.float32, attn_implementation="eager").to(DEVICE).eval()
text = open("blog_train.txt").read()
ids = torch.tensor([tok(text[200_000:206_000]).input_ids[:1024]], device=DEVICE)
with torch.no_grad():
    attn = model(ids, output_attentions=True).attentions          # per layer: (1, heads, T, T)
T = ids.shape[1]
rows = []
for layer, a in enumerate(attn):
    a = a[0, :, 64:, :].float().mean(0)                         # heads averaged; queries 64..T-1
    q = torch.arange(64, T, device=a.device)
    first = a[:, 0].mean().item()
    recent = torch.stack([a[i, max(0, int(qi) - 3):int(qi) + 1].sum() for i, qi in enumerate(q)]).mean().item()
    rows.append((layer, first, recent, 1 - first - recent))
for layer, first, recent, rest in rows:
    if layer % 3 == 0 or layer == len(rows) - 1:
        print(f"layer {layer:2d}: first token {first:5.1%}, last 4 tokens {recent:5.1%}, the other ~1,000 {rest:5.1%}")
print(f"averaged over all {len(rows)} layers: first token {sum(r[1] for r in rows) / len(rows):.1%}, "
      f"last 4 tokens {sum(r[2] for r in rows) / len(rows):.1%}; a uniform split would give the first token "
      f"{sum(1 / (q + 1) for q in range(64, T)) / (T - 64):.2%}")

The rest of the script counts the heads that park most of their attention on the first token, then changes which token comes first, to see whether the sink follows the token or the position.

Full script: sinks.py

layer  0: first token  0.0%, last 4 tokens 60.4%, the other ~1,000 39.6%
layer  3: first token 35.1%, last 4 tokens 21.5%, the other ~1,000 43.4%
layer  6: first token 49.5%, last 4 tokens 17.3%, the other ~1,000 33.3%
layer  9: first token 31.7%, last 4 tokens 20.2%, the other ~1,000 48.1%
layer 12: first token 34.9%, last 4 tokens 14.3%, the other ~1,000 50.8%
layer 15: first token 16.8%, last 4 tokens 28.7%, the other ~1,000 54.4%
layer 18: first token 20.2%, last 4 tokens 33.0%, the other ~1,000 46.9%
layer 21: first token 30.6%, last 4 tokens 12.1%, the other ~1,000 57.3%
layer 24: first token 46.8%, last 4 tokens 17.3%, the other ~1,000 36.0%
layer 27: first token  0.1%, last 4 tokens 13.0%, the other ~1,000 86.9%
averaged over all 28 layers: first token 30.5%, last 4 tokens 21.1%; a uniform split would give the first token 0.29%
heads in layers 1-26 putting over half their attention on the first token: 48 of 312; under 5%: 19
passage at character 100,000                 starts with 'rive'      : most attention on position 0 ('rive'), 29.0%
passage at character 200,000                 starts with ' then'     : most attention on position 0 (' then'), 30.5%
passage at character 300,000                 starts with 'Parameter' : most attention on position 0 ('Parameter'), 32.3%
passage at character 400,000                 starts with '7'         : most attention on position 1 ('"'), 30.9%
first passage, first token moved to the end  starts with ' climbs'   : most attention on position 0 (' climbs'), 30.4%
first passage with '7'    in front: most attention on position 1 (' then'), 30.4%
first passage with '3'    in front: most attention on position 1 (' then'), 30.4%
first passage with 'A'    in front: most attention on position 0 ('A'), 30.3%
first passage with '.'    in front: most attention on position 0 ('.'), 30.4%
first passage with '\n'   in front: most attention on position 0 ('\n'), 30.4%
first passage with 'the'  in front: most attention on position 0 ('the'), 30.4%

Averaged over the 28 layers, the very first token of a 1,024-token passage receives 30% of all attention, a hundred times its share if attention were spread evenly, and the four most recent tokens another 21%. The rest of the context, about a thousand tokens, gets half. Some layers are different: the first layer looks mostly at the last few tokens (the query’s own included), and the last layer spreads widely. In most layers, on average, a third of the attention goes to the first position, and it’s concentrated in some heads: 48 of the 312 heads in layers 1 to 26 put more than half of theirs there.

It’s mostly not the token’s meaning. Passages beginning with “rive”, ” then” and “Parameter”, the first passage rotated so it begins with ” climbs”, and the same passage behind “A”, ”.”, a newline or “the” all put about 30% on their first token. The exception is telling: a passage that starts with a digit (“7”, or “3” in its place) puts the sink on the second token instead. Xiao et al. (2024) called these attention sinks: since softmax must put its weight somewhere, heads that have nothing useful to attend to learn to park it on a token every query can see. Which token gets the role is something the model builds, unusually large activations on that token (Sun et al., 2024), normally for whatever comes first; why a leading digit doesn’t get it in this model, this post can’t say. The sink is harmless, even useful: it lets a head effectively switch itself off. It matters for long context in one concrete way: evicting the first tokens’ keys and values from a model trained with full attention breaks it, which is why streaming methods pin them (Attention Alternatives is the home of this topic). It is not where content is read: in the key-value prompts the sink is the chat template’s first token, dozens of tokens before the first key.

Position and training data

Two more causes, which this small experiment can’t separate from the others:

• Position. Beyond a model’s training length, RoPE’s rotation makes far-apart query-key pairs land at angles training never produced (see Where Am I? Position in Attention, from Sinusoids to RoPE, which also covers position interpolation and YaRN for extending context). At 16,000 tokens this model is inside its trained length, so that doesn’t apply here. What can apply is that long distances are rare in training even within the window: An et al. (2024) trace short effective context lengths to how seldom far relative positions occur in training.
• Training data. Most documents are short. A model trained on text where the relevant context is usually nearby learns a prior for the nearby, and for the start, where instructions and titles go. Long-context training sets are built by packing or synthesizing long documents, and how many tasks in them truly need distant information decides what the model learns to use.

The U shape itself is consistent with two position effects: the end of the prompt is closest to the question, where recency helps, and early tokens are visible to every later position in every layer, which biases a causal model toward them (Wu et al., 2025). Attention sinks are a separate fact about the start: they mark where heads park attention, not where content is read.

What to do about it

Each cause suggests its own remedy, and in practice they’re combined:

• Put the question where it helps. On the objects that scored 6 of 12 above, stating the key before the data as well as after scored 12 of 12. In a causal model a token can only use what came before it, so with the question first, every data token is processed already knowing what’s being looked for (Liu et al. found the same for key-value lookup, though little gain on document QA). Providers’ guides differ in detail: Anthropic’s says to put long documents first and the query last, and reports up to 30% better answers that way; OpenAI’s GPT-4.1 guide says instructions at both ends work best, and if only once, before the context. One trade-off: a question placed before a fixed block of documents changes every request’s prefix, which defeats prompt caching of those documents (see Prompt Caching); documents first, question last is the cache-friendly order.
• Send less. Retrieval (Building RAG) selects the relevant passages instead of sending everything, so there’s less middle to get lost in, and a reranker (Reranking and Late Interaction) can put the best ones first or last.
• Compact. Summarize old turns and tool outputs and drop what’s done (Context Engineering), so the context holds what’s needed now.
• Keep the input outside the context. A recursive language model (Recursive Language Models) stores a long input as a variable and has the model read it in pieces, through code.
• Train for it. Longer training sequences, data whose answers depend on distant text, and position schemes that extend gracefully. That’s the model builder’s fix; the others are the application builder’s.
• Measure on the real task. A needle grid says little. Test the model on the application’s own documents, with the relevant part at several depths, at the lengths it will actually be sent.

Where this leads

• The KV Cache measures what a long context costs in memory, and the alternatives to full attention in Attention Alternatives exist largely because of attention’s long-context cost.
• Prompt Caching and Context Engineering, and the retrieval posts of Part IX: Retrieval, are the application-side answers to this post: deciding what goes into the context, and where.

Takeaways

• A context window is what a model accepts, not what it uses evenly. On a key-value lookup, a 1.5B model was always right with the key at the start or end, and in the middle went from 97% right at 2,000 tokens to 47% at 16,000.
• Needle-in-a-haystack tests miss this: one odd sentence is easy to find (119 of 120 here). Distractors that look like the answer, several needles, and reasoning over what’s found are what break models, and RULER and Context Rot measure them.
• Most misses weren’t blanks but neighbors: another pair’s value, 5 to 20 pairs before the target. Stating the question before the data as well fixed every miss in the cell tested (6 of 12 to 12 of 12).
• Softmax dilution: with a fixed score margin, the relevant token’s share of attention falls toward $e^{\delta}/n$ once $n$ is much larger than $e^{\delta}$; holding it needs a margin that grows as $\ln n$. It explains why long prompts are harder, not why the middle is.
• Attention sinks: about 30% of attention goes to the first token, almost whatever it is (a leading digit is an exception in this model). They mark where heads park attention, not where content is read.
• Position extrapolation and short training documents add to it. The fixes are on both sides: put important text at the ends, send less (retrieve, compact), train on long documents that need their length, and test at the lengths and depths the application uses.

References

Start with Liu et al., which has the original U-shaped plots, then RULER for what a harder benchmark measures. Xiao et al. is the attention-sink paper; read it before Attention Alternatives.

• Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, Lost in the Middle: How Language Models Use Long Contexts (TACL, 2024): the U-shaped curve.
• Hsieh et al., RULER: What’s the Real Context Size of Your Long-Context Language Models? (COLM, 2024): tasks beyond retrieval, and effective context length.
• Hong, Troynikov and Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance (Chroma, 2025): degradation with length on simple tasks, 18 models.
• An et al., Why Does the Effective Context Length of LLMs Fall Short? (2024): short effective context from rarely trained long distances.
• Wu, Wang, Jegelka and Jadbabaie, On the Emergence of Position Bias in Transformers (ICML, 2025): the causal mask’s bias toward early positions.
• Sun, Chen, Kolter and Liu, Massive Activations in Large Language Models (COLM, 2024): the huge activations behind attention sinks.
• Anthropic, Prompting best practices: long context: documents first, the query last.
• OpenAI, GPT-4.1 prompting guide: instructions at both ends of long context.
• Xiao, Tian, Chen, Han and Lewis, Efficient Streaming Language Models with Attention Sinks (ICLR, 2024): attention sinks.