Tag: long-context
All the articles with the tag "long-context".
Long Context, and Where It Breaks: Lost in the Middle
Updated:A model that accepts a long prompt doesn't use all of it equally. The lost-in-the-middle result reproduced on a 1.5B model: a key-value lookup that's perfect at 2,000 tokens and loses things in the middle by 8,000. Why the needle-in-a-haystack test misses it, what RULER and Context Rot measure instead, and the causes: softmax dilution worked out, attention sinks measured, RoPE's reach, and training on short documents. Then what to do about it: put things where they're read, retrieve less, compact, train longer.
Recursive Language Models: The Prompt as an Environment
Updated:A recursive language model never reads its long input: the input sits in a Python REPL as a variable, and the model writes code to inspect it and calls itself on the pieces. How the idea works, why its three design choices matter, what the paper measured, and a miniature built and run on this blog's whole archive with a 3B model, against direct reading and a fixed map-over-chunks program.
Where Am I? Position in Attention, from Sinusoids to RoPE
Updated:Attention on its own can't tell word order apart, proved in two lines. How position gets in: added sinusoids, learned embeddings, ALiBi's distance penalty, and RoPE's rotation, derived so the query-key score depends only on relative distance. Then a tiny Transformer trained with each, tested on sequences four times longer than it trained on, on one task that needs exact relative position and one that needs to look back over any distance, and what the results say about long-context extension.