RAG vs. long context: which does your AI need?

There are two common ways to get an AI assistant to work with a large amount of material. One is to find the relevant parts and show the model only those. The other is to put as much as possible in front of it at once. They are called retrieval and long context, and each has a different failure mode.

What retrieval-augmented generation is

Retrieval-augmented generation, usually shortened to RAG, is a pattern in which the system looks something up before the model answers. The term comes from a 2020 research paper, and the idea has since become a standard building block.

A typical setup works like this:

• Prepare. Your documents are split into passages, and each passage is turned into a list of numbers, called an embedding, that reflects its meaning. These are stored in an index.
• Search. When you ask a question, the question is turned into numbers the same way, and the index returns the passages whose numbers are closest.
• Answer. The best passages are placed into the prompt next to your question, and the model writes its reply using them.

The model never sees your whole collection, only what the search returned for that question.

What long context means

Long context takes the opposite approach. The model has a large context window, and you place the whole document, or a large part of it, directly in the prompt. There is no search step to get wrong. For what the window is and why it has limits, see what an AI context window is.

Where retrieval is strong

• Collections larger than any window. A company's documents or years of notes will not fit in one prompt, but a search can still find the right page.
• Fresh or private material. You can add documents without changing the model.
• Cost per question. Sending a few passages is cheaper and faster than sending everything every time.
• Traceable answers. The system can show which passages it used, which makes checking easier.

Where retrieval struggles

• Missed passages. If the search does not return the passage that holds the answer, the model never sees it, and it may answer anyway from the passages it did get.
• Questions about the whole. "What are the main themes across all of these?" or "What changed between the first and last version?" need more than the few passages that match a query.
• Split-up context. A passage cut out of a document can lose what made it meaningful, such as a definition given pages earlier.
• Setup effort. The index has to be built and kept current, and the quality of the splitting and searching decides the quality of the answers.

Where long context is strong

• Whole-document reasoning. The model can relate a statement on page three to one on page ninety.
• Simplicity. There is no index to build and no search to tune.
• No retrieval miss. If the text is in the prompt, it is available to the model.

Where long context struggles

• Cost and speed. Processing a very long prompt generally takes more computation, so each question can be slower and more expensive.
• Uneven use of the text. Research on long inputs, including the 2023 paper "Lost in the Middle," found that models often did best with information at the beginning or end of the input and worse in the middle.
• A ceiling still exists. Whatever the window size, some collections are larger.
• Noise. More text can bury the passage that matters.

Choosing between them

• One document or a few, and you want reasoning across it: try long context first.
• A large, changing collection, and questions about specific facts: retrieval fits.
• Questions about the whole of a large collection: use summaries or a staged process, as in how to summarize a long document.
• Answers you must be able to check: prefer a setup that shows its sources, whichever method it uses.

Many real systems combine both: retrieval narrows a large collection to a handful of documents, and long context lets the model read those documents in full.

Quick answers

Is RAG the same as giving the AI a document? Not quite. Pasting a document puts all of it in the prompt. RAG searches a collection and puts in only the parts that matched your question.

What is an embedding? A list of numbers that represents the meaning of a piece of text, built so that passages with similar meanings end up close together. It is what makes searching by meaning possible.

Does RAG stop made-up answers? It reduces them when the right passages are found and the model is told to rely on them. It does not eliminate them, because the model can still misread or ignore a passage. See why AI makes things up.

Can I use both? Yes, and many systems do: retrieval picks a handful of documents, and long context lets the model read them in full.

Where NAMELESS stands

NAMELESS is an AI model that is still in training, built on our own architecture for unlimited context: it does not reread the conversation to continue it, so there is no window to fit things into. How an assistant remembers and finds things still matters, which is the subject of AI memory vs. context.

You can join the waitlist to follow progress.