Your Model Has a Million-Token Context Window. That Doesn't Mean You Should Delete Your RAG Pipeline.

The pitch that keeps showing up in vendor decks

Every few months someone pitches the same idea: now that context windows run into the hundreds of thousands or millions of tokens, why bother with retrieval-augmented generation at all? Just paste the whole knowledge base into the prompt and let the model sort it out. No chunking decisions, no embedding model to pick, no reranker to tune, no vector database to run and pay for. One fewer system to own.

It's a real option now in a way it wasn't three years ago, and for some workloads it's the right call. But "the context window is big enough to fit the document" and "the model will actually use everything in that window correctly" are two different claims, and the research on the second one is less flattering than the marketing suggests.

What the research actually shows

The relevant study here is "Lost in the Middle: How Language Models Use Long Contexts," by Nelson Liu and coauthors at Stanford and Samaya AI, published in TACL. The setup was simple: give a model a question along with several documents, only one of which actually contains the answer, and vary where that answer-bearing document sits in the input. The result was a consistent U-shaped curve — models were noticeably better at using information placed at the very start or the very end of the context than information buried in the middle. The effect held even for models explicitly built and marketed for long-context use, and accuracy degraded further as the overall input got longer, not just as the needle moved toward the middle.

That finding matters more than the headline context-window number, because it describes a failure mode that doesn't show up in the demo. If you ask a long-context model to find one needle in a mostly-empty haystack, it usually will — needle-in-a-haystack benchmarks are specifically designed to be easy. Production queries aren't like that. They need the model to weigh several relevant passages scattered across a large, messy input, and that's exactly the condition under which the U-shaped curve bites: the model systematically underweights information that happens to land mid-document, regardless of how relevant it is.

RAG sidesteps this by construction. A retrieval step narrows thousands of pages down to the handful of passages actually relevant to the query, so the model is reasoning over a short, high-signal context rather than searching for a needle in a haystack it has to hold in its head. The cost of that narrowing is a pipeline you have to build and maintain. The benefit is that the model's attention isn't spread across material it was never going to need to answer the question.

The cost and latency problem nobody puts in the demo

There's a second issue that's architectural, not statistical: processing a long context isn't free, and it isn't flat. Every token in the input has to be attended to before the model produces a single output token, so prefill time and the memory needed to hold the attention cache both grow with context length. A query answered by retrieving three relevant paragraphs and a query answered by re-processing an entire 200-page manual cost and latency look nothing alike, even when both eventually land on the same answer. If you're running that query once, the difference is a rounding error. If you're running it ten thousand times a day because it's the support bot embedded in your product, the difference is your hosting bill.

Context windows also don't scale the way the headline number implies. Providers set a maximum, but that's an architectural ceiling, not a performance guarantee — publishing "supports a million tokens" is a different claim than "performs equally well at a million tokens as at ten thousand," and the Lost in the Middle results are exactly the gap between those two claims made concrete.

Where long context genuinely wins

None of this means RAG should always win by default. Long context is the better tool when the task requires holding the whole document in view at once rather than retrieving a relevant slice of it — summarizing an entire contract, reviewing a full codebase for a cross-cutting bug, or reasoning about how an argument develops across an entire deposition transcript. In those cases, retrieval would chop up exactly the continuity the task depends on, and you want the model seeing everything, degraded recall in the middle and all. Long context also removes real engineering overhead for smaller corpora — if your knowledge base is twenty pages, not twenty thousand, building a retrieval pipeline to serve it is solving a problem you don't have.

The actual decision

The honest framing isn't "RAG vs. long context" as a one-time architecture choice for your whole product. It's a routing decision made per query type, because most real systems have both kinds of questions. "What does section 4.3 of this specific contract say?" is a retrieval problem. "Does this 80-page contract contain any clause that conflicts with the one we signed last year?" is a long-context problem, because the conflict could be anywhere and retrieval might miss the comparison entirely. A system that always stuffs full documents into context will overpay and underperform on the first kind of question. A system that only ever retrieves top-k chunks will miss the second kind entirely, no matter how good the reranker is.

In practice that means instrumenting your system to know which kind of question it's answering, keeping both paths available, and resisting the vendor pitch that one of them makes the other obsolete. It doesn't, yet, and the research explaining why is specific enough to design around rather than just take on faith.

Takeaway: a bigger context window buys you the option to skip retrieval for a narrower set of tasks than it looks like — it doesn't buy you reliable recall across everything you put in it. Decide per query type, not per product.