Decision matrix
RAG or fine-tuning: weigh it for your project
Set how much each criterion matters to you. The bars show where your weights lean across four approaches, including the baseline of prompting alone.
1 · Set what matters to you
2 · Where your weights lean
- Prompting onlyGood fit
- RAG (retrieval)Strong fit · leads
- Fine-tuningWeak fit
- RAG + fine-tuningGood fit
Your weights lean towards RAG (retrieval). Your priorities point at retrieval: answers from your current documents, with sources, filtered per user. Prompting carries the tone.
3 · The matrix, cell by cell
| Criterion · your weight | Prompting only | RAG (retrieval) | Fine-tuning | RAG + fine-tuning |
|---|---|---|---|---|
| Facts change oftenWeight: High | Good fit Fine while the content fits in one prompt; an edit is live on the next request. | Strong fit Re-index a changed document and the next answer uses it. | Poor fit A new fact means a new training run, and old facts linger in the weights. | Strong fit Retrieval carries the facts that change. |
| Answers must cite sourcesWeight: High | Partial fit The model can quote what it was given, but nothing checks which passage it used. | Strong fit Each answer can link the passages it was built from. | Poor fit Knowledge stored in weights has no source to point to. | Strong fit Citations come from the retrieval layer. |
| Consistent format, tone or labelsWeight: Low | Good fit Instructions plus a few worked examples get most of the way. | Partial fit Retrieval adds facts, not a house style; the prompt still carries the style. | Strong fit Its real strength: a consistent format, tone or label set learned from examples. | Strong fit The tuned model carries format and tone. |
| Different users see different contentWeight: Medium | Poor fit Everyone gets the same prompt unless you build and route separate ones. | Strong fit Retrieval is filtered by the user’s permissions before the model sees anything. | Poor fit A model cannot forget a document for one user, so per-user access is out of reach. | Strong fit Permissions are enforced at retrieval, as with RAG alone. |
| Time to a first working versionWeight: Medium | Strong fit Days: write the prompt, test it against real questions, ship. | Good fit Weeks: ingestion, chunking, retrieval tuning and an evaluation set. | Partial fit Longest single path: collect and clean examples, train, evaluate, repeat. | Poor fit Two systems to build, evaluate and keep in step. |
| Cost and speed per requestWeight: Low | Partial fit Every request pays for the whole pasted context, so cost grows with content. | Good fit Adds retrieved passages to each request, but only the few that matter. | Strong fit Short prompts, and a smaller tuned model can stand in for a larger one on a narrow task. | Good fit Shorter prompts than RAG alone, plus the upkeep of a tuned model. |
| Training examples on handWeight: Medium | Strong fit Needs only a handful of written examples. | Strong fit Needs your documents, not labelled examples. | Poor fit Needs a large, clean set of input and output pairs you trust. | Partial fit Needs both documents and a training set. |
| Freedom to switch models laterWeight: Medium | Strong fit A prompt moves to another model after re-testing. | Good fit Index and prompts move to a new model; re-embed only if the embedding model changes. | Poor fit Tied to one base model and one tuning service; a retirement means retraining. | Partial fit The retrieval half ports to a new model; the tuned half does not. |
Ratings are our judgement, written next to the reason in each cell. They are a starting point for a conversation, not a recommendation for your situation; the discovery questions further down decide it.
What each option actually is
Retrieval-augmented generation keeps your knowledge outside the model. When a question arrives, the system searches an index of your documents, hands the most relevant passages to the model and asks it to answer from them. The research paper that named the approach paired a language model with a searchable index and pointed out that updating a model’s knowledge, and showing where an answer came from, were unsolved problems. Retrieval addresses both, because the knowledge lives in documents you can edit and cite.
Fine-tuning changes the model itself. You train it further on examples of the input you will send and the output you want back, and it learns the pattern: a format, a tone, a label set, a way of handling one narrow task. Parameter-efficient methods train a small set of extra weights instead of the whole model, which makes tuning cheaper to run and store. The result is still a model of your own that you have to evaluate, version and eventually retire.
Prompting alone is the baseline both are measured against: clear instructions, a few worked examples and, when it fits, the relevant content pasted straight into the request.
The most common misconception about fine-tuning is that it teaches a model your facts. It is the wrong tool for that job: facts held in weights cannot be cited, cannot be hidden from one user and stay stale until the next training run.
The criteria, one by one
How fast your knowledge changes
If a price, a returns policy or a product spec changes on Tuesday, a retrieval system answers correctly on Tuesday afternoon, after one re-index. A fine-tuned model answers with whatever it learned at training time until someone retrains it, and even then the old version can surface. For knowledge that moves, retrieval is the only option of the three that keeps up without a training pipeline.
Whether answers must show their sources
Support, compliance and internal-policy assistants usually need an answer a person can check. Retrieval can attach the passages behind each answer, and a confidence gate can decline when nothing relevant was found. A tuned model produces fluent text with no trail back to a document, so a reviewer has nothing to verify it against.
Who is allowed to see what
When a branch manager and a franchise partner must see different documents, the filter has to run before the model reads anything. Retrieval does that by tagging each chunk with who may see it and filtering the search. There is no way to make a tuned model forget a fact for one user, so anything sensitive that went into training is available to everyone who uses the model.
Behaviour versus knowledge
This is where fine-tuning earns its place. A classifier that must always return one of a fixed set of labels, an extraction step that must emit the same structure every time, or a writing task that must match a house voice across thousands of outputs can all improve with tuning. Even here, try a tighter prompt with good examples first: it is faster to change and it moves with you to the next model.
The data you actually have
Retrieval needs the documents you already keep, cleaned up and current. Fine-tuning needs a training set: real inputs paired with outputs your team has checked and would sign off. Most organisations have the first and not the second, and building a trustworthy training set is often the largest hidden cost of a tuning project.
Vendor availability and the freedom to switch
Tuning services are vendor products, and they change. As of the access date in the sources, OpenAI’s model optimization guide says it is winding down its fine-tuning platform: new users can no longer reach it, and existing tuned models stay available until their base models are deprecated. Whichever vendor you tune with, check that the service and the base model will outlive your project. A retrieval index carries no such tie: documents, chunks and prompts move to another model, with re-embedding only if the embedding model changes.
What drives the cost
We do not quote vendor prices here because they change and depend on your volume. What does not change is where the money goes.
| Approach | Build cost comes from | Running cost comes from | Upkeep cost comes from |
|---|---|---|---|
| Prompting only | Writing and testing the prompt and examples | Model calls that carry the full pasted context | Re-testing when content or model changes |
| RAG | Ingestion, chunking, retrieval tuning, evaluation set | Embedding, search and model calls with a few passages each | Re-indexing, watching retrieval quality, handling new sources |
| Fine-tuning | Building and reviewing the training set, training runs | Calls to the tuned model, often with short prompts | Retraining when behaviour drifts or the base model retires |
| RAG + fine-tuning | Both of the above, plus testing them together | Retrieval plus tuned-model calls | Both upkeep loops, kept in step |
Where each approach fits, and where it does not
| Approach | Fits | Does not fit |
|---|---|---|
| Prompting only | A short policy or FAQ set, a first prototype, a narrow writing task | Large or changing document sets, per-user permissions |
| RAG | Support and sales assistants, internal knowledge search, anything that must cite | Pure style or label problems with no knowledge gap |
| Fine-tuning | Fixed-label classification, strict extraction formats, a consistent voice at volume | Teaching facts, fast-moving content, anything that needs sources |
| RAG + fine-tuning | A mature assistant where retrieval works and a measured style gap remains | A first version, or a team without a training set |
The questions we ask in discovery
- Which questions will people actually ask? We ask for real ones from your inbox, chat logs or sales calls, because they show whether the gap is knowledge or behaviour.
- Where does the answer to each one live today, and who edits it? That tells us what to index and how often.
- What happens when the answer is wrong? A wrong delivery date and a wrong dosage instruction need very different safeguards.
- Must the person asking see a source, or must a reviewer be able to check later?
- Do different people have the right to different answers?
- Is there a format the output must always follow, and do you have reviewed examples of it?
- How many requests a day do you expect at the start, and how fast must each one return?
- Which model vendors are you allowed to use, and under whose account?
How we decide
We decide with a test set, not a preference. The steps are the same on every project, and each one is cheap compared with building the wrong system.
- Write the test set first: real questions, each with the answer your team would accept and the source it should come from.
- Run prompting alone against it and sort the failures into missing facts, stale facts, wrong format and wrong tone.
- If the failures are missing or stale facts, add retrieval and re-run. If they are format or tone, improve the instructions and examples and re-run.
- Only when prompt-level fixes stop improving a behaviour failure do we test a tuned model, on the same set, and keep it only if it wins.
- Keep the set and run it on every change of prompt, content or model, so a regression shows up before your users see it.
When MiniAI Labs is the wrong choice
If you need a model trained from scratch, or a large research programme tuning open-weight models on your own hardware, hire a machine-learning research team or a specialist ML consultancy. We build retrieval systems and tune a model only where an evaluation justifies it; we do not run large training programmes.
If your knowledge is one short document that rarely changes, you may not need a builder. A carefully written prompt in a hosted assistant could be enough, and we will tell you so on the scoping call.
If the open question is whether your data may be used for training at all under your sector’s rules, take it to your counsel before you take it to any vendor.
Limits of this comparison
- Fit ratings are editorial judgement from the reasons written in each cell, not benchmark results.
- Vendor tuning services change often; the availability notes hold as of the access dates in the sources.
- Larger context windows keep moving the line between prompting alone and retrieval, so re-test that line whenever you change models.
- The comparison assumes text knowledge and text answers; image, audio and numeric prediction tasks have trade-offs of their own.
Sources
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.) (opens in a new tab) · arXiv · accessed
- Model optimization (opens in a new tab) · OpenAI API docs · accessed
- PEFT: Parameter-Efficient Fine-Tuning (opens in a new tab) · Hugging Face docs · accessed