Skip to content

RAG (retrieval-augmented generation) ​

A pattern that first retrieves documents relevant to a question and puts them in the prompt, so the model answers from that text instead of from memory alone.

A model only knows what was in its training data, and it cannot say where a fact came from. RAG adds a lookup step. The name comes from a 2020 paper by Lewis and others, which combined a generator with "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever". Today the term covers any pipeline that searches your own content and hands the results to the model.

The pipeline ​

  1. Ingest. Split documents into chunks and store them, usually with an embedding (a vector) per chunk in a vector storeVector store: A database that keeps embeddings (lists of numbers that stand for the meaning of text chunks) and returns the chunks whose embeddings are nearest to a query's..
  2. Retrieve. Turn the question into the same kind of vector and fetch the nearest chunks. Hybrid searchHybrid search: Retrieval that runs a keyword search (such as BM25) and a vector search for the same query, then merges the two ranked lists into one. also matches keywords, which catches names and error codes that vectors miss.
  3. Generate. Put the chunks in the prompt with an instruction to answer only from them, and ideally to cite them.

Example ​

A helpdesk bot is asked "How do I reset my password?". Retrieval returns the FAQ chunk about password resets; the prompt says "Use the context to answer"; the model replies with the steps from that chunk and marks it as its source.

Where it goes wrong ​

  • Bad retrieval. The right chunk is never found, so the model answers from nothing, or guesses.
  • Ignored context. The model answers from memory anyway. Citations make this visible.
  • Poisoned content. A retrieved document can carry instructions for the model, a form of indirect prompt injectionIndirect prompt injection: Prompt injection in which the attacker's instructions hide in content the model reads while doing its job, such as a web page, an email, a document or a tool result.. OWASP lists vector and embedding weaknesses as LLM08.

Common confusion ​

RAG is not fine-tuning. Fine-tuning changes the model's weights; RAG leaves the model alone and changes the prompt, so new documents count as soon as they are ingested.

In L2M ​

Knowledge bases give you a built-in vector store in the L2M database. The RAG Retrieve node searches one by vector, keyword (BM25) or hybrid, and can emit citation instructions; a Rerank node can reorder results. Try it with no API key in the RAG knowledge base recipe.

Further reading ​

This website (not the L2M product) uses Google Analytics, only after you accept cookies. Cookie settings