It strikes me that our industry has become somewhat obsessed with a single metric of progress: the size of the context window. We’ve watched as physical limits have climbed from a few thousand tokens to millions, yet we continue to encounter a frustrating phenomenon we can call “Context Rot”. Even when a model can physically “see” a million tokens, its ability to reason over them effectively—to aggregate information or find subtle patterns—often degrades long before the window is full.

A recent paper from researchers at MIT CSAIL introduces a pattern that addresses this not by making the “memory” larger, but by changing how the model interacts with the data. They propose Recursive Language Models (RLMs), a strategy that treats a long prompt not as a direct input to a neural network, but as an external environment to be programmatically explored.

The leaky abstraction of the context window In traditional software engineering, we are familiar with the concept of “out-of-core” algorithms. When a dataset is too large to fit into a system’s fast main memory (RAM), we don’t just stop, but we write code that cleverly fetches chunks of data from a larger, slower disk.

The standard LLM pattern, however, has been “in-core” by default. We try to shove everything into the Transformer’s attention mechanism at once. The problem is that as the context grows, the signal-to-noise ratio drops, and even frontier models like GPT-5 begin to exhibit catastrophic failures on complex tasks. This is particularly true for “information-dense” problems—tasks where the answer depends on nearly every line of the prompt, rather than just a single hidden “needle”.

Prompt as environment pattern

The RLM shift is a fundamental change in architectural pattern. Instead of Prompt as Input, we move to Prompt as Object.

When an RLM receives an arbitrarily long prompt, it doesn’t feed it to the model. Instead, it initializes a Python Read-Eval-Print Loop (REPL) environment and loads the prompt as a string variable. The LLM is then given a system prompt that encourages it to interact with this variable symbolically.

The model might:

  • Peek: Print the first and last few lines to understand the structure.
  • Filter: Use regex or keyword searches to find relevant snippets.
  • Decompose: Write a loop to split the prompt into manageable “smart chunks”.
  • Recurse: Call a sub-instance of itself (often a smaller, cheaper model) to process those chunks and return a summary or specific data point.

By offloading the context to the REPL, the “effective” context window is no longer limited by the Transformer’s architecture, but by the memory of the execution environment.

Scaling inference-time compute

What I find most compelling about this research is that it represents a move toward scaling inference-time compute. Rather than relying on a single, expensive forward pass, the RLM uses a series of smaller, strategic steps.

The results are quite striking:

  • Extreme scale: RLMs have successfully handled inputs up to 10 million tokens—two orders of magnitude beyond current context windows.
  • Robustness to complexity: On the OOLONG-Pairs benchmark—a task requiring the model to aggregate pairs of data points across a large set—base models achieved F1 scores of less than 0.1%. The RLM approach achieved 58%.
  • Cost efficiency: For massive documents (6M-11M tokens), the RLM with GPT-5 was actually cheaper than standard retrieval or summarization baselines because the model could selectively view only the necessary context.

Trade-offs and “It Depends”

As with any architectural choice, there are no silver bullets—only trade-offs.

  • Latency: Because RLMs rely on iterative, often sequential sub-calls, they can be significantly slower than a single base model call. The median RLM run might be efficient, but “tail” trajectories can involve hundreds of recursive steps.
  • Coding maturity: An RLM is only as good as its ability to write the Python code that explores its environment. Small models that struggle with coding generally fail to act as effective RLMs.
  • Non-optimal choices: Early trajectories show that models can be “liberal” with sub-calls, sometimes performing a separate LM call for every single line of a thousand-line context when a batch approach would have sufficed.

The RLM pattern suggests that we may be reaching the limits of what can be achieved through neural scaling alone. My feeling is that the next generation of “Deep Research” agents will look less like a single massive model and more like a programmatic scaffold that allows models to manage their own context dynamically.

By treating the prompt as a piece of data to be queried rather than a message to be read, we move closer to systems that can truly handle long-horizon tasks without getting lost in the noise. The bottleneck is no longer the “window,” but the model’s ability to architect its own search.

And life goes on …