Every second discovery call we run includes some version of the sentence "we want to train the model on our data." It is a reasonable instinct and almost always the wrong plan. The confusion is understandable: RAG vs fine-tuning gets framed as a rivalry between two ways of making an LLM "know your business," when they are actually two different tools for two different failure modes — and they differ by orders of magnitude in what they cost to build, run, and keep correct.
This is the decision guide we wish clients had before that call, with real numbers from current provider pricing pages rather than vibes.
Two techniques, two different failure modes
Retrieval-augmented generation (RAG) keeps your knowledge outside the model. Documents are chunked, embedded, and stored in a searchable index; at question time the system retrieves the relevant passages and hands them to the model as context. The idea goes back to Lewis et al. 2020, which showed that pairing a generator with a retriever produces more specific, more factual output than relying on what is baked into the weights — and, crucially, gives you provenance for every answer.
Fine-tuning changes the weights themselves by training on examples. It is excellent at shifting behavior — tone, output format, adherence to a narrow task — and surprisingly bad at adding facts. Ovadia et al. 2023 tested exactly this question ("Fine-Tuning or Retrieval?") and found that RAG consistently outperformed unsupervised fine-tuning for knowledge injection — both for knowledge the base model had seen during training and for entirely new facts. Models struggle to absorb new factual information through fine-tuning at all unless the same fact is drilled in many paraphrased variations.
So the first sorting question is not "which technique is better" but "is our gap missing knowledge or wrong behavior?" Missing knowledge — current prices, policies, product specs, contracts — is a retrieval problem. Wrong behavior — verbose answers, broken JSON, off-brand tone — is a prompting problem first and a fine-tuning problem only after prompting measurably plateaus.
Why retrieval wins the knowledge problem
Four properties make RAG the default for a business knowledge base, and none of them are subtle.
Freshness is an index update, not a training run. When a policy changes, you re-index one document and the next answer reflects it. A fine-tuned model reflects the world as of its last training run — and "just retrain" means re-paying the training cost and re-validating the model every time your content moves.
Answers come with receipts. Because the model is generating from retrieved passages, the system can cite the exact source paragraph. For anything customer-facing or compliance-adjacent, that audit trail is the difference between a tool people trust and a tool legal shuts down. A fine-tuned model can tell you an answer; it cannot tell you where the answer came from.
Grounded generation fails less, and more visibly. Hallucination doesn't disappear with RAG, but it becomes small and measurable: Vectara's hallucination leaderboard, which measures how often models invent facts when summarizing a provided document — the grounded, RAG-like setting — puts today's better models in the low single digits, with the best around 2%. And when retrieval finds nothing, a well-built system can say "not in the knowledge base" instead of confabulating. There is no equivalent confidence signal for facts recalled from weights.
Your data stays deletable. Under GDPR, the
right to erasure (Art. 17) is
straightforward to honor in a RAG architecture: delete the document, re-index,
done. Data that has been trained into model weights cannot be selectively
removed — there is no DELETE FROM weights WHERE customer_id = …. For
European SMEs handling personal data, that asymmetry alone should settle most
arguments.
The cost mechanics, honestly
RAG's costs sit in three places: a one-time indexing bill, a small storage line, and a per-query context-token bill that scales with usage.
Indexing is almost embarrassingly cheap. OpenAI's
current pricing lists
text-embedding-3-small at $0.02 per million tokens and
text-embedding-3-large at $0.13 — embedding an entire corporate archive
costs less than lunch. Even the fancier preprocessing has a published price:
Anthropic's contextual retrieval
technique, which prepends an LLM-generated context sentence to every chunk
and cuts top-20 retrieval failure rates by 49% (67% with reranking), costs
about $1.02 per million document tokens when run through prompt caching.
Storage is a rounding error: at typical chunk sizes a mid-sized corpus is a
few tens of thousands of vectors, which fits comfortably in the Postgres +
pgvector instance you probably already run.
The recurring cost is context tokens: every retrieved chunk is input the model must read on every request. On Anthropic's price list, Claude Sonnet 4.6 runs $3 per million input tokens / $15 per million output, and Haiku 4.5 $1 / $5. Two levers cut this hard: prompt caching bills repeated prefix content at 0.1× the input price (writes cost 1.25×), and batch processing halves everything that isn't latency-sensitive. One threshold worth knowing before you build anything: Anthropic's own guidance is that a knowledge base under ~200,000 tokens (roughly 500 pages) doesn't need RAG at all — put the whole thing in the prompt and let caching absorb the cost.
Fine-tuning's sticker prices look comparable until you read the structure.
On OpenAI's pricing page,
training runs $5 per million tokens for gpt-4.1-mini and $25 for
gpt-4.1, with fine-tuned inference at $0.80/$3.20 and $3/$12 respectively.
Three costs hide behind those numbers. First, training is paid per run, and
every substantive content change is a new run plus a new validation pass.
Second, the dominant cost isn't compute at all — it is building and
maintaining the labeled dataset and the eval set that tells you whether the
fine-tune helped. Third, generation lock-in: the fine-tunable models on
today's price list are the gpt-4o / gpt-4.1 / o4-mini generation, while
the current flagships are the gpt-5.6 family. A RAG system inherits every
new model the day it ships; a fine-tune is welded to a snapshot that is
already a generation behind.
A worked example: the 10,000-document assistant
Take a concrete SME case: an internal assistant over 10,000 documents — manuals, contracts, wiki pages — averaging ~2,500 tokens each, so a 25-million-token corpus, answering 5,000 questions a month.
Build (one-time):
- Embeddings: 25M tokens × $0.02/MTok = $0.50 ($3.25 with
text-embedding-3-large) - Contextual-retrieval preprocessing: 25M × ~$1.02/MTok ≈ $26
- Vector store: ~31,000 chunks of 800 tokens — pgvector on existing infrastructure, effectively $0
Serve (monthly): each answer retrieves ten 800-token chunks plus system prompt and question, call it 10,000 input tokens, with a ~500-token answer. On Sonnet 4.6 that is $0.030 + $0.0075 ≈ 4 cents per answer, or about $190/month at 5,000 questions; on Haiku 4.5 it is ~$65/month. Caching the system prompt and tool definitions shaves it further.
The fine-tuning "alternative" for the same corpus: 25M training tokens on
gpt-4.1-mini at $5/MTok is $125 per epoch — $375 for a typical
three-epoch run, repeated on every meaningful content update. And per the
Ovadia et al. results, the resulting
model still won't reliably reproduce those facts, and can never cite them.
The entire RAG indexing bill costs less than a quarter of one training epoch,
and the monthly serving bill is less than a day of anyone's engineering time.
That is the honest economics: for the knowledge problem, retrieval is not
just better — it is cheaper at every line item.
Where fine-tuning actually earns its keep
None of this makes fine-tuning useless. It has four legitimate jobs, all of them behavior-shaped:
- Format and style at scale. When you need thousands of outputs a day in an exact schema or a specific voice, a fine-tune bakes the behavior in and lets you delete the pages of few-shot examples from every prompt.
- Distillation into small models. Teach a cheap model a narrow task using
a frontier model's outputs as training data: fine-tuning
gpt-4o-minicosts $3 per million training tokens and serves at $0.30/$1.20 — an order of magnitude below flagship rates, which matters at real volume. - Narrow classification and extraction where prompting has measurably plateaued on your eval set — not where someone suspects it might.
- Latency. Shorter prompts on smaller models respond faster; for interactive products this is sometimes the whole business case.
The mature architecture is usually hybrid, and not in the hand-wavy sense: RAG carries the facts, while a small fine-tuned model handles one narrow, high-volume subtask like query routing or field extraction. The techniques compose; they don't compete.
How to decide, in order
Work through these in sequence and stop at the first answer:
- Is the whole knowledge base under ~200K tokens? Skip RAG. Whole corpus in the prompt, prompt caching on, ship it this week.
- Is the gap knowledge? RAG. Spend the saved money on document hygiene and a retrieval eval — chunking and search quality will dominate your answer quality far more than model choice.
- Is the gap behavior? Prompt engineering with a real eval set first. Only when the eval shows prompting has plateaued and volume justifies the pipeline, fine-tune the narrowest possible model for that one task.
- Do you need both? Build RAG first. It ships in weeks, produces the logged question-answer pairs that become your fine-tuning dataset later, and nothing about it blocks adding a fine-tuned component when step 3 genuinely triggers.
The pattern behind all four steps: at SME scale, API costs are noise compared to engineering time. The scarce resources are a clean LLM knowledge base and an eval set built from real user questions — and both of those investments pay off regardless of which technique you end up running.