AI Engineering

RAG vs Fine-Tuning: Which Does Your Use Case Need?

Wooden library card catalogue drawers with paper labels Teach it facts or habits?

RAG changes what a model knows at the moment it answers. Fine-tuning changes how the model behaves. If your problem is company knowledge that changes, such as policies, prices, contracts or product documents, use RAG, or simply put the documents in the prompt. Fine-tune only when you need a consistent format or style that prompting cannot give you, and in late 2026 fewer hosted models offer it.

This guide is for owners, operations leads and CTOs who have been told they need one or the other and want to check. It covers the cheapest option first, then when each approach is the right call, where fine-tuning is being withdrawn, and five questions that settle most cases. For the mechanics of retrieval itself, see our guide to what RAG is and how it works. This post is about the choice.

RAG vs fine-tuning at a glance

The two approaches solve different problems, so they are rarely true substitutes. RAG is like handing someone the right pages before they answer. Fine-tuning is like training them until a way of working becomes habit.

AspectRAGFine-tuning
What it changesThe facts the model sees per questionThe model’s behaviour: format, tone, labels
Data neededYour documents, kept tidy and currentMany example inputs with the outputs you want
How updates workChange a document, next answer reflects itPrepare new examples and train again
What goes wrongWrong or stale passage retrievedBenefits aren’t guaranteed, and the model can be retired

Cost also has a different shape. RAG adds to the cost of every question, because each answer carries retrieved text. Fine-tuning front-loads effort into preparing examples and testing the result. We cover running costs below.

Start with the cheapest option: put it in the prompt

Before building retrieval or training anything, check whether your documents fit in a single prompt. Today’s models read a lot. Anthropic lists Claude Sonnet 5.5 with a 1M-token context window, roughly 555,000 words. OpenAI’s gpt-6-astra takes about 1.05M tokens, and Google lists Gemini 4 Argon at 1M.

In 2024, Anthropic suggested that if your knowledge base is under 200,000 tokens, you can include the whole thing in the prompt instead of building retrieval. That advice came in its Contextual Retrieval announcement and was a threshold for its models at the time. With 1M-token windows today, the set of cases where a plain prompt is enough has grown.

There is a cost caveat. A long prompt is paid for on every call. On gpt-6-astra, OpenAI charges 2x for input and 1.5x for output once a request goes above 272K input tokens. A handbook pasted into every question can become the most expensive part of your bill.

A simple test helps. If the documents fit, rarely change and are used by a handful of people on a handful of questions, start with the prompt. You can build retrieval later if volume or size grows, and you will have working answers to measure it against.

When RAG is the right answer

RAG is the right answer when the knowledge changes, is large, differs by user, or must be cited. Four situations push you there.

  • The knowledge changes. A returns policy updated last week should answer questions this week. With RAG you replace the document and you are done.
  • There are more documents than fit in a prompt. Or they fit, but sending them all each time costs too much.
  • People have different permissions. The search step can filter by who is asking, before any text reaches the model. A prompt stuffed with everything cannot do that.
  • You need to show your sources. Retrieval knows which passage it used, so the answer can link to it and a person can check.

The idea was named in a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis and colleagues. The principle has not changed: pair a model with a searchable index of documents.

Retrieval quality is where the work is

Many RAG failures start in the search step, so retrieval quality often matters more than model choice. Anthropic reported that its Contextual Retrieval method, which adds explanatory context to each passage before it is indexed, combined with keyword search, cut failed retrievals by 49%, and by 67% when reranking was added (19 September 2024). Those are one vendor’s test results on its own data, so read them as a sign that search deserves engineering time, and do not expect to reproduce the exact figures.

The embedding step is cheap. At the time of writing (November 2026), OpenAI’s pricing page lists text-embedding-3-small at $0.02 per 1M tokens and text-embedding-3-large at $0.13. Prices change, so check the page before you budget. The real cost of RAG is the engineering around search, permissions and testing, plus keeping the documents current. That last part is usually a data engineering job.

When fine-tuning is the right answer

Fine-tuning is the right answer when the problem is behaviour, not knowledge. OpenAI’s model optimisation guide lists the uses it has in mind:

  • getting a consistent output format
  • handling novel inputs the base model struggles with
  • using more examples than fit in a prompt
  • shortening prompts
  • running a smaller, cheaper model

A realistic example is a back-office team that needs every extracted invoice returned in one exact structure, or a classifier that must apply your own internal labels the same way thousands of times. These are habits, and examples teach habits well.

Two cautions apply. First, fine-tuning does not keep facts current. Microsoft’s fine-tuning overview for Foundry says it “doesn’t replace retrieval for current information”. If your price list changes monthly, training it into a model creates a stale copy you cannot easily edit. Second, the same page warns that benefits aren’t guaranteed. You need a set of test cases and a baseline from prompting first, so you can tell whether training helped.

One related benefit is worth knowing. Google notes that tuning a Gemini model can reduce context length by removing few-shot examples from the prompt. If you are pasting the same twenty examples into every request, training them in can shorten every call.

Fine-tuning availability has shrunk

OpenAI is withdrawing fine-tuning and other hosted providers offer it on a limited set of models, which changes the risk of choosing it.

OpenAI is winding down fine-tuning. Its supervised fine-tuning guide says new users cannot access it, and eligible existing users can create jobs until 6 January 2027, unless the base model shuts down earlier. Fine-tuned gpt-4.1-nano models shut down on 23 October 2026, and fine-tuned gpt-4.1 and gpt-4.1-mini models shut down on 14 January 2027.

The picture elsewhere is mixed:

A fine-tuned model is tied to the base model it was trained on, and base models get retired. If you build on RAG and good prompting, you can move to a newer model and rerun your tests. If you build on a fine-tune, retirement can mean repeating the work. Where you do fine-tune, plan for it to be redone, and keep your training examples and test set so that repeating it is a rerun, not a restart.

Can you combine them?

Yes, and sometimes you should. Microsoft’s overview page lists using retrieved context effectively as a goal of fine-tuning, which means you can train a model to handle retrieved passages better, for instance by following a citation format or answering in a set structure. Retrieval supplies the facts and the fine-tune shapes the response.

Most businesses should not start there. Get RAG working and measured first. If the answers are correct but the format or tone keeps drifting, and prompting has not fixed it, that is the evidence that justifies a fine-tune. Without it, you are adding a second moving part to a system you cannot yet test.

How to decide: five questions

Five questions settle most cases. Answer them in order and stop when one gives you a clear direction.

  1. Does the knowledge change? If yes, use RAG or the prompt. Do not train facts that will go out of date.
  2. Does it fit in a prompt? If it fits and does not cost too much per call, start with the prompt. It is the simplest thing that works.
  3. Do users need different permissions? If yes, retrieval with permission checks in the search step. A shared prompt cannot separate who sees what.
  4. Do you need citations? If yes, RAG. A fine-tuned model cannot point to where an answer came from.
  5. Is the problem format or style, not facts? If yes, and a well-written prompt with examples has failed on your test cases, consider fine-tuning, and check that your provider still offers it.

If you answered yes to any of the first four, retrieval or the prompt is your path. Only a clear yes to the fifth, after prompting has been tried and measured, points to fine-tuning.

What it costs

At CodeLeap, a first AI agent or assistant starts from £5,000 (about $6,500), with a fixed price agreed upfront. Support starts from £400 (about $520) a month. What moves the figure is how many sources the assistant reads and how tidy they are, how strict permissions need to be, and how much human sign-off the work needs.

Running costs are separate: model calls for each question, hosting for the search index, and the time to keep documents current. Our guide to what an AI agent costs breaks those down. New to agents? Start with what an AI agent is.

A sensible next step

Write down the ten questions you most want an AI to answer. For each, note where the answer lives, how often it changes and who is allowed to see it. If the answers sit in documents that change, you want retrieval or a long prompt. If the answers are really about a fixed format or labelling, test a good prompt first and fine-tune only if it falls short.

If you would like a second opinion, our AI agents and automation service starts with exactly this question, and an AI readiness assessment can map where your data stands before you spend on a build. Book a free 30-minute call and we will tell you which route fits.

Frequently asked questions

What is the difference between RAG and fine-tuning?

RAG searches your documents when a question arrives and hands the best passages to the model, so it changes what the model knows at answer time. Fine-tuning trains the model on examples, so it changes how the model behaves, such as its format, tone or labelling. RAG suits facts that change. Fine-tuning suits consistent habits.

Is RAG better than fine-tuning?

For company knowledge, usually yes. Policies, prices and contracts change, and with RAG you update a document and the next answer reflects it. Fine-tuning does not do that, and Microsoft's documentation says it doesn't replace retrieval for current information. Fine-tuning is better when the problem is a fixed output style or format that prompting cannot deliver.

Can you fine-tune Claude or ChatGPT models?

It depends on the provider and the date. OpenAI is winding down fine-tuning: new users cannot access it, and eligible existing users can create jobs only until 6 January 2027. On Amazon Bedrock, the only Anthropic model listed as fine-tunable is the older Claude 3 Haiku. Check the current documentation before planning around it.

Can you use RAG and fine-tuning together?

Yes. Microsoft lists teaching a model to use retrieved context effectively as a fine-tuning goal, so the two can work as a pair. Most businesses should get retrieval working and measured first, because it fixes the most common problem, missing or outdated facts, and it does not tie you to one model version.

Do I need RAG if the model has a 1M token context window?

Not always. If your whole knowledge base fits in the prompt and rarely changes, pasting it in is the simplest option. RAG earns its place when you have far more documents than fit, when long prompts make each call too expensive, when different users may see different files, or when answers must cite a specific source.

Not sure which one fits?

Book a free 30-minute consultation and we'll tell you whether you need retrieval, fine-tuning or just a better prompt.

Book a consultation Get a free quote

Keep exploring

Back to all insights

Book a free call with Vivian

COO and co-founder · London

Request a callback

Leave your name and number and we'll call you back.