Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation, or RAG, connects a language model to a search over your own documents. Before answering, the system retrieves relevant passages and passes them to the model. The answer then comes from those passages rather than from the model's training memory.

How does RAG work?

RAG runs in two phases. The first happens once, in the background: your documents are split into sections, each section is turned into an embedding, and those vectors go into a vector database.

The second phase happens on every question. The question is turned into an embedding too, the database returns the closest matching sections, and those sections go to the model alongside the question. The instruction is effectively: answer this using these sources.

The effect is twofold. The model can work with content it never saw in training, your handbook or your price list. And the answer can be traced, because you know which passages went into it.

Why does RAG matter for companies?

Most of the genuinely useful answers inside a company live in documents no public model has ever read. Without a connection to those documents, a language model stays a well-read conversational partner with no knowledge of your business.

RAG is also the option with the fewest side effects. The documents stay where they are. Change one, and the next answer changes with it, without anyone retraining a model. And because the sources travel with the answer, every hallucination becomes easier to spot.

RAG or fine-tuning?

The two solve different problems, though they get set against each other constantly.

RAG gives a model knowledge. Reach for it when content changes, when sources have to be verifiable, or when access rights need to apply.

Fine-tuning gives a model behaviour. Reach for it when a format, a tone or one very specific task has to land consistently.

Both together is possible and rarely necessary. There is a longer treatment in RAG vs fine-tuning.

Where RAG fails in practice

Almost always in the retrieval, not the model. Sections that are too large bury the actual statement in noise. Sections that are too small lose the context that made them meaningful. Tables and forms tend to shatter into useless fragments when split.

The second common failure is permissions. If the search can reach everything in the file store, the system will happily answer questions the person asking has no right to ask. Access rights belong in the retrieval layer, not in the prompt.