RAG vs long context
Large context windows did not retire retrieval. The choice is a budget and freshness decision, and most production systems end up doing both.
Short answer
Put the whole corpus in the window when it fits comfortably, is stable between requests and is small enough to cache. Retrieve when the corpus is larger than the cost you are willing to pay on every call, when it changes faster than you can rebuild a prompt, or when different users must see different subsets of it. Neither is a default. The window size tells you what is possible; the cost per request, the freshness requirement and the access rules tell you what to do.
When context windows were small, retrieval was not a design choice. You could not fit a handbook into a request, so you fetched the paragraphs you needed. Windows are now large enough that the whole handbook often does fit, and the conclusion many teams draw is that retrieval was scaffolding they can now remove.
That conclusion is right for some systems and expensive for others, and the discriminator is rarely the window size. It is what you pay per request, how often the material changes, and whether every user is allowed to see all of it. A corpus that fits is not the same as a corpus you want to send on every call, and a prompt that is cheap because it is cached stops being cheap the moment the material behind it changes.
There is also a quality argument that cuts against the naive reading of a large window. A model spends a finite attention budget across the tokens it is given, and recall degrades as the count grows, an effect Anthropic's engineering writing calls context rot. Filling a window because you can is not free even when the tokens are affordable.
Retrieval vs Long context, side by side
| Retrieval | Long context | |
|---|---|---|
| What you pay per request | The retrieved slice, plus the cost of the retrieval hop | The whole corpus, every time, though caching makes repeats far cheaper |
| Latency | A retrieval round trip, then a smaller model call | One call, but a large one. A cache read is much faster than a cold send. |
| Freshness | Whatever the index holds. Re-indexing is a separate job with its own lag. | Whatever you assembled at request time, though a cached prefix is only as fresh as the cache. |
| Access control | Filter in the query, which is usually where per-user rules belong | Filter before assembly, or accept that everyone sees the same corpus |
| Typical failure | The retriever misses the relevant passage, so the model never sees it and answers confidently without it | The passage is present but recall degrades across a full window, so the answer misses it anyway |
| Auditability | You can log exactly which passages were used | Everything the model saw is in the request, which is simpler to reason about |
| Where it wins | A corpus too large, too fresh or too permissioned to ship whole | A stable, bounded set: one contract, a handbook, a small codebase |
It is a budget question first
Start with arithmetic rather than architecture. Take the corpus size in tokens, multiply by your request volume, and price it. If that number is comfortable and the material is stable enough to cache, sending it whole is the simpler system: no index to build, no embedding model to keep in step, no retriever to tune, and no class of bug where the right passage existed and was not fetched.
If the number is uncomfortable, retrieval is not a compromise, it is the correct engineering response to a cost you cannot pay. The mistake in both directions is treating the choice as a matter of sophistication. A retrieval pipeline built for a corpus that would have fit is an extra failure mode bought at the price of complexity.
A large window does not mean uniform attention
The second input to the decision is quality, and it works against filling the window on principle. The model has a finite attention budget spread across whatever it is given, and recall gets weaker as the token count grows. A relevant sentence buried in two hundred thousand tokens of loosely related material is not as good as the same sentence in a focused prompt.
This is why the layout guidance matters when you do send large inputs: put long documents at the top, above the query and instructions, wrap each in tags that separate source from content, and ask the model to quote the passages it is relying on before it reasons over them. That last instruction is worth more than it looks, because it turns an unverifiable answer into one you can check.
Just-in-time retrieval is the middle path
The framing of retrieval as a preprocessing step is dated. The alternative is to keep lightweight identifiers in context, such as file paths, table names or search queries, and let the model pull the content it needs during the run using tools.
That changes the trade meaningfully. The prompt stays small, the corpus can be arbitrarily large, freshness is whatever the source has right now rather than whatever the index has, and access control lives in the tool rather than in a copy of your permission model. The cost is turns: each fetch is another round trip, and a model that fetches badly wastes them. It is the right shape when the corpus is large and the relevant slice is genuinely unpredictable.
Caching changes the arithmetic, in one direction only
Prompt caching shifts the balance toward long context, but only for a specific shape of traffic: many requests that share the same long prefix within the cache lifetime. A stable document set at the top of the prompt, with the varying question at the end, is the canonical fit.
Two constraints decide whether it applies. There is a minimum cacheable prefix length, which varies by model, so short corpora do not qualify. And the cache is keyed on an exact prefix, so any change before the breakpoint invalidates it. A corpus that is regenerated on every request, or that has a timestamp near the top, will never be read from cache no matter how it is configured. If the material changes often, caching does not rescue the long-context option, and retrieval starts to look better again.
Hybrid is the normal end state
The two are not exclusive, and the strongest designs usually combine them: retrieve to narrow a large corpus down to a candidate set, then give the model all of what survived rather than a handful of fragments. You get retrieval's cost control and long context's coherence, and you avoid the worst failure of aggressive chunking, which is handing the model a passage stripped of the context that made it meaningful.
The practical rule is to retrieve at the coarsest unit that still fits your budget. Whole documents beat sections, sections beat paragraphs, and paragraphs beat sentences, right up to the point where the budget says otherwise.
How to choose
Reach for retrieval when
- The corpus is larger than you are willing to send on every request
- The material changes faster than you could rebuild and re-cache a prompt
- Different users are entitled to different subsets of the data
- You need a log of exactly which passages informed each answer
- Relevant material is a small, findable fraction of a large whole
Reach for long context when
- The corpus is bounded and comfortably within budget at your request volume
- The material is stable enough that a cached prefix stays valid
- Answers depend on relationships across the whole set, not on locating one passage
- Everyone using the system is entitled to see all of it
- You would rather not own an index, an embedding model and a retriever
Where candidates go wrong
Trap 1: the window is big enough now, so retrieval is obsolete
Capacity and economics are different questions. A corpus that fits may still be unaffordable at volume, may change too often to cache, and may not be something every user is allowed to see. None of those are solved by a larger window. There is a quality half to this too. Filling a window because it is available spends attention budget on material that is not relevant to the current question, and recall degrades as the count grows. Options that justify removing retrieval purely by citing window size are answering a capacity question that was not asked.
Trap 2: retrieval is the safe default
The opposite error, and it is more common among experienced engineers. Adding retrieval to a small, stable corpus buys a chunking strategy, an embedding model, an index to keep in step with the source, and a new failure mode in which the answer is confidently wrong because the right passage was never fetched. That failure is particularly unpleasant because it is invisible from the output. A missing passage does not announce itself; the model simply answers from what it was given. If the whole thing fits and rarely changes, sending it is both simpler and more auditable.
Trap 3: smaller chunks retrieve better
Smaller chunks do match queries more precisely, which is why the instinct is to shrink them. What they lose is the surrounding material that made the passage mean anything: the clause that qualifies it, the heading that scopes it, the sentence before that says which product it applies to. The result is retrieval that scores well and answers badly. Prefer the coarsest unit your budget allows, and where a fine-grained match is needed, match at the small unit but return the larger one around it. Options proposing ever-smaller chunks as a fix for poor answer quality usually have the diagnosis backwards.
Trap 4: we cache it, so sending everything is free
Caching reduces the cost of re-sending an identical prefix. It does not make the first send free, and a cache write costs more than sending the same tokens uncached, so a prefix used once is more expensive cached than not. It also does nothing for a prefix that changes. Anything varying before the breakpoint, including a timestamp, a user name or a freshly retrieved document set, produces a miss on every request. Teams reach this trap by assembling the corpus dynamically and then wondering why the cache never reads. Stable content first, varying content last, and if nothing is stable then caching is not the lever.
How the exam tests it
Prompt and Context Engineering is 11.0% of the CCDV-F blueprint, roughly 6 of the 53 scored questions, and Context Engineering is 3.8% of that. The exam treats context engineering as curation of the tokens present at inference rather than as prompt wording, so questions here tend to be about what should be in the window at all, which is exactly the decision on this page.
Common questions
Do large context windows make RAG unnecessary?
No. A larger window changes what is possible, not what is affordable, current or permitted. Retrieval still wins when the corpus is too large to send at your request volume, changes faster than a prompt can be rebuilt, or must be filtered per user. It also keeps a per-answer record of what was used.
When is long context clearly the better choice?
When the corpus is bounded, stable and comfortably affordable, and when answers depend on relationships across the whole of it rather than on finding one passage. A single contract, a policy handbook or a small codebase are the usual examples. You avoid an index, an embedding model and a class of retrieval misses.
What is just-in-time retrieval?
Keeping lightweight identifiers in context, such as file paths or queries, and letting the model fetch content with tools during the run rather than preloading it. The prompt stays small, freshness comes from the source rather than an index, and access control lives in the tool. The cost is extra turns.
Does prompt caching remove the cost argument for retrieval?
Only when traffic shares a long, stable prefix within the cache lifetime. There is a minimum cacheable prefix length that varies by model, and the cache is keyed on the exact prefix, so anything that changes before the breakpoint produces a miss. A corpus assembled fresh on each request will never read from cache.
Can I use both?
Yes, and it is usually the best answer. Retrieve to narrow a large corpus to a candidate set, then give the model all of what survived rather than fragments. That keeps cost under control while avoiding the main weakness of aggressive chunking, which is passages stripped of the context that made them meaningful.
Practise this
Prompt and Context Engineering is 11% of the CCDV-F blueprint, and the bank is weighted to match. Every option carries a written explanation, not just the correct one, so a distinction like this one is explained where you get it wrong.
See CCDV-FOther CCDV-F distinctions
Not affiliated with or endorsed by Anthropic. Domain names and weightings are taken from the published exam guide; always check the official guide before booking.