← The blog

Explainer · · 6 min read

RAG is not a product. It is a plumbing decision

Retrieval sounds like a feature you buy. It is really a set of unglamorous choices about chunking, ranking and what to do when nothing matches.

By Manish

Retrieval-augmented generation gets sold as a product category. It is not one. It is the plumbing that decides which of your documents the model sees before it answers, and plumbing is exactly the right way to think about it.

The common story goes like this. A team buys a vector database, loads their documents, asks a question, gets a mediocre answer, and concludes the model is not good enough. The model is almost always fine. The retrieval is what failed, and retrieval failures are invisible unless you go looking.

What is actually happening

When someone asks a question, the system converts it into a numerical representation, finds document chunks whose representations are close, and pastes those chunks into the prompt alongside the question. The model then answers using what it was handed.

That last part is the bit people skip. The model does not search your documents. It reads whatever your retrieval step decided to give it, and it has no way of knowing what was left out.

So the quality ceiling is set entirely by retrieval. A frontier model handed the wrong three paragraphs will produce a fluent, confident, wrong answer, and it will do so faster than a weaker model would.

Decision one: how you split documents

Chunking sounds like an implementation detail and is the single largest driver of quality.

Chunk too small and you lose the context that made a passage meaningful. A sentence saying this limit does not apply to enterprise accounts is useless when separated from the limit it refers to. Chunk too large and the relevant sentence gets buried in four paragraphs of noise, diluting the match and wasting context.

What works better than any fixed size is splitting on structure. Break on headings, sections and natural boundaries rather than on character counts, and keep a small overlap so a thought spanning a boundary survives. Adding the document title and section heading to every chunk costs almost nothing and improves matching substantially, because it restores the context that splitting removed.

Decision two: how you rank results

Pure vector similarity finds things that are topically close. Topically close is not the same as correct, and the difference shows up constantly in production.

Ask about the refund policy for annual plans and vector search will happily return the refund policy for monthly plans, because those two documents are nearly identical in meaning space. It is a good match and the wrong answer.

Hybrid search plus reranking typically beats any amount of model upgrading, and costs considerably less. It is the highest-return change available in most RAG systems.

Decision three: what happens when nothing matches

This is the one nobody plans for and the one users notice immediately.

Vector search always returns something. It returns the closest chunks even when the closest chunks are irrelevant, because closest is a relative measure with no floor. If your system passes those to the model and the model answers anyway, you have built a confident liar.

A retrieval system that says I could not find that is worth more than one that always produces an answer.

Two fixes, and you want both. Set a relevance threshold so genuinely poor matches are discarded rather than passed along. And instruct the model explicitly that if the provided material does not contain the answer, it must say so rather than infer.

Test retrieval separately from generation

Before blaming the model, look at what it was handed. Log the retrieved chunks for twenty real questions and read them yourself.

If you could not have answered from those chunks, the model could not either, and no prompt engineering will fix it. This one habit catches the majority of RAG problems and costs an afternoon.

It is also the diagnostic almost nobody runs, because the interface shows you the answer rather than the evidence. Build the debug view that shows both. Every team that has done this reports the same reaction, which is surprise at how bad the retrieval was.

When not to build this at all

If your document set fits in a context window, put it in the context window. Modern models handle hundreds of pages, and no retrieval system beats simply showing the model everything.

One caveat, and it is important. Long-context research consistently finds that facts placed in the middle of very long inputs are recalled less reliably than facts near the edges. A large window is a capacity, not a promise, and dumping four hundred pages in does not guarantee the model attends to page two hundred.

So the honest threshold is lower than the advertised window. Below roughly a few dozen pages, skip retrieval. Above that, retrieve well and then give the model a generous window to reason over what you retrieved. Hybrid beats either extreme.

What RAG does not fix

It does not fix bad documents. If your internal wiki contradicts itself, retrieval will faithfully surface the contradiction and the model will pick one at random. Teams frequently discover their documentation problem while building a RAG system, which is genuinely useful and not what they budgeted for.

It does not fix stale content either. A retrieval system over a knowledge base nobody updates is an efficient way to distribute out-of-date answers. Freshness metadata and a way to prefer recent documents matter more than most people expect.

The order to build in

Start with the simplest thing that could work: structural chunking, hybrid search, a relevance threshold, and an explicit instruction about missing information. Measure it on twenty real questions.

Then add reranking, which is usually the biggest single improvement. Then metadata filtering. Only after all of that should you consider a larger model, because by then you will actually know whether the model was ever the constraint.

Most teams do this in reverse, upgrade the model first, see no improvement, and conclude the technology is overrated. The technology was fine. The pipe was blocked.

Evaluation is the part everyone skips

Most teams evaluate RAG by trying a few questions and forming an impression. That catches obvious breakage and misses everything else, particularly the slow degradation that happens as a document set grows.

A real evaluation needs a fixed set of questions with known correct answers, drawn from your actual documents. Thirty is enough to be useful. Write them once, and include the awkward cases: questions whose answer spans two documents, questions where the answer is genuinely absent, and questions where two documents disagree.

Then measure two things separately. Retrieval accuracy, meaning whether the right chunk was in the retrieved set at all. And answer accuracy, meaning whether the model used it correctly. Conflating those is why teams cannot tell which half to fix.

If retrieval accuracy is poor, no model upgrade helps. If retrieval is good and answers are wrong, then the prompt or the model is the problem. That single split saves weeks.

Permissions are a design problem, not a filter

The moment a retrieval system covers documents with different access levels, you have a security question that must be answered at retrieval time rather than afterwards.

Filtering results after generation is not sufficient. If a restricted document reached the model, its content is potentially in the answer, and stripping the citation does not remove the leak.

The correct pattern is filtering before search, so the user's permissions constrain which chunks are searchable at all. This has to be built in from the start, because retrofitting it means re-architecting the index.

Cost grows in ways people do not anticipate

RAG looks cheap in a prototype and gets expensive in three specific places.

That last one is usually the largest and the least noticed, because it is buried in ordinary model spend rather than appearing as a RAG line item.

Find the tools to build it

Browse the infrastructure tools developers use to build AI apps.

Browse tools for building AI apps

See AI model rankings

Tools mentioned