Skip to content
How AI agents work· Guide 8 of 9Intermediate8 min readUpdated

RAG vs fine-tuning ​

Retrieval-augmented generation (RAG)RAG (retrieval-augmented generation): A pattern that first retrieves documents relevant to a question and puts them in the prompt, so the model answers from that text instead of from memory alone. and fine-tuning solve different problems. RAG looks up documents when a question arrives and puts the relevant passages in the prompt, so the model answers from text you control and can cite. Fine-tuning trains an existing model further on your examples, so its weights change and it behaves differently on every later call: a fixed output format, a house tone, a narrow classification task done reliably by a smaller model. Use RAG when the model needs to know something that changes, is private, or must be traceable to a source. Use fine-tuning when the model needs to do something in a consistent way that prompting alone does not get right. Many teams need neither, and some need both.

Two places to put knowledge ​

The original RAG paper (Lewis et al., 2020) frames the two kinds of memory well. A pre-trained model stores facts in its weights, its "parametric" memory. RAG adds a second, "non-parametric" memory: in the paper, "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever". (In the paper, retriever and generator are fine-tuned together; RAG as most teams build it today retrieves for a model they do not train.) The authors point at the weakness of weights alone: "providing provenance for their decisions and updating their world knowledge remain open research problems."

That sentence is the core of the comparison. Facts held in weights cannot point to where they came from, and changing them means training again. Facts held in an index can be cited, replaced and deleted one document at a time.

Where RAG and fine-tuning actTop row, at query time: a question goes to a retriever, which reads an index of your documents and adds the top passages to the prompt sent to the model. Bottom row, before deployment: training examples go into a training job that produces new model weights, which the model then uses on every call.At query time (RAG)QuestionRetrieverIndex of your docsPrompt + passagesModelBefore deployment (fine-tuning)Training examplesTraining jobNew weights
RAG changes what the model reads on each call. Fine-tuning changes the model itself, once, before it is used.

What fine-tuning is for ​

Provider documentation is consistent about what fine-tuning does well, and it is mostly about behaviour, not facts:

  • OpenAI's model optimization guide says fine-tuning can "enable a model to consistently format responses in a certain way", and that you can "train a smaller, cheaper, faster model to excel at a particular task". It also names shorter prompts as a benefit, since examples baked into the weights no longer need to be sent on each call.
  • Google's Vertex AI tuning guide recommends supervised fine-tuning for "Strict output formatting, classification, sentiment analysis, entity extraction, and domain-specific style or tone", and says it suits cases where "the output of your model isn't complex". Its typical dataset size is "Hundreds to thousands of labeled examples".

Both guides put prompting first, and OpenAI's starts with evals. OpenAI's describes optimization as "a combination of evals, prompt engineering, and fine-tuning" and starts with evals that establish a baseline; it also notes that "The prompt engineering process may be all you need". Provider offerings change, too; check what your provider offers before you plan around it.

Neither guide recommends fine-tuning to keep a body of facts current or citable. That is the job RAG was designed for.

The comparison ​

QuestionRAGFine-tuning
What changes?The prompt, on each callThe model's weights, once per training run
What it is good atFacts, documents, policies, anything you must citeFormat, tone, classification, a narrow task done by a smaller model
Updating a factRe-ingest or delete one document; live on the next queryBuild new examples, train again, re-test, redeploy
Showing the sourceEach passage carries its document id, so answers can citeWeights cannot say where an answer came from
Who may see whatRetrieval can be limited per audience; text never enters the weightsAnything trained in may be extractable by anyone who can call the model
Removing dataDelete the chunksRetrain without them; the old model may still be deployed
Up-front workIngest, chunk, embed, tune retrievalCollect and label hundreds of examples or more, run training
Cost per callExtra input tokens for the passages, plus a searchCan be lower: shorter prompts, possibly a smaller model
What to evaluateRetrieval (did the right passage come back?) and the answerThe answer, against a baseline of the untuned model
Failure modeThe right passage is not found, or the model ignores itThe model learns the wrong habit, or forgets a general skill

Two rows deserve more words.

Access control. In a RAG system, permissions are a property of the index and the query: one team's documents can live in a separate collection, or carry metadata that a filter excludes. A fine-tuned model has no such boundary. If HR tickets went into the training set, any caller who can reach the model can, in principle, coax that text back out. Train only on data every caller of the model is allowed to see.

Removing data. A request to delete a person's data, or a document that turns out to be wrong, is a delete in an index. In a fine-tuned model it is a new training run, and until the old model is retired, the old behaviour is still live.

When to use which ​

Work down this list and stop at the first answer that fits.

  1. Better instructions and a few examples in the prompt. If a clear system prompt and three examples fix the problem, you are done.
  2. RAG when the model lacks information: product documentation, a policy that changed last week, a customer's contract, the ticket history. Also when you must show the source, or when different users may see different documents.
  3. Fine-tuning when the model has the information but behaves wrongly in a consistent way you can show with examples: it will not keep to a JSON shape, it labels tickets inconsistently, or a large model does a narrow task well and you want a smaller, cheaper one to do the same.
  4. Neither when the answer must come from a live system. A stock level or an order status belongs in a tool the agent calls, not in an index that is a copy, and not in weights.

The last point matters most for agents. An AI agentAI agent: A system in which a language model decides, step by step, which tools to call and in what order, reading each result before it chooses the next step, until it can answer. can call tools, so it often does not need knowledge copied anywhere: it can ask the system that owns the data, with the caller's permissions, at the moment it needs it.

A worked example ​

A support team wants an assistant that answers questions about their product and tags each ticket with one of 40 categories.

  • Product answers change with every release, must link to the help article, and some articles are for enterprise customers only. That is RAG: index the help centre, keep enterprise articles in their own collection, and require citations.
  • Ticket tagging is a fixed classification task. The team has 6,000 tickets tagged by hand. If a prompt with the category list and examples tags poorly, a small fine-tuned classifier is a reasonable next step, judged against the hand-tagged set.
  • Order status is a live lookup. Neither technique applies; give the agent a read-only tool.

Using both ​

RAG and fine-tuning combine without conflict. A model fine-tuned to answer in your support format, cite passages as [1] and refuse when the context is empty can then be given retrieved passages at query time. The fine-tuning shapes how it uses the context; RAG supplies what it reads. Keep the two evaluations apart: when answers get worse, you want to know whether retrieval stopped finding the passage or the model stopped using it.

RAG in L2M (L2M does not fine-tune) ​

L2M runs RAG pipelines; it does not train or fine-tune models. A fine-tuned model that your provider serves under a model id or an Azure OpenAI deployment, or that an OpenAI-compatible server serves, is to L2M a model like any other: a chat model node calls it by that id or deployment name, and model allowlists can limit which models workflows may use.

For the retrieval side:

  • Knowledge bases are a built-in vector storeVector store: A database that keeps embeddings (lists of numbers that stand for the meaning of text chunks) and returns the chunks whose embeddings are nearest to a query's. in the L2M database. Upload text, Markdown, HTML, CSV or JSON; L2M chunks and embeds it. Deleting a source (DELETE /api/knowledge-bases/:id/sources/:sourceId) removes its chunks, so correcting a document means deleting its source (an admin action) and uploading the new version, not a training run.
  • RAG Retrieve searches a knowledge base by vector, keyword (BM25) or hybridHybrid search: Retrieval that runs a keyword search (such as BM25) and a vector search for the same query, then merges the two ranked lists into one. search, returns topK passages (3 by default) as documents and a numbered context, and can feed a Rerank node. External stores such as Qdrant, pgvector and Azure AI Search plug into the same node for vector search.
  • Citations. RAG Retrieve also emits citationInstructions for the prompt, and the Extract Citations node maps [N] markers in the answer back to chunk and source ids.
  • Access. A knowledge base that belongs to a project can only be used by workflows in that project (project access); one with no project is shared across the deployment. Within one knowledge base, a search covers every chunk; there is no per-chunk filter. Give each audience its own knowledge base, or use an external store whose filter you set on the node (Qdrant's filter).
  • Evaluation. RAG-aware scorers check retrieval and answers separately: context_precision and context_recall are programmatic, and faithfulness and answer_relevance use a judge model once judges are turned on (RAG scorers; see also agent evals).

Try it in L2M ​

The RAG knowledge base recipe builds a knowledge base and a retrieval workflow that run with no API key. The RAG with citations recipe adds numbered sources to the answer. When you are weighing whether to retrieve at all or to put whole documents in the prompt, read RAG vs long context.

Further reading ​

Last updated:

This website (not the L2M product) uses Google Analytics, only after you accept cookies. Cookie settings