RAG (retrieval-augmented generation)
A pattern that first retrieves documents relevant to a question and puts them in the prompt, so the model answers from that text instead of from memory alone.
A model only knows what was in its training data, and it cannot say where a fact came from. RAG adds a lookup step. The name comes from a 2020 paper by Lewis and others, which combined a generator with "a dense vector index of Wikipedia, accessed with a pre-trained neural retriever". Today the term covers any pipeline that searches your own content and hands the results to the model.
The pipeline
- Ingest. Split documents into chunks and store them, usually with an embedding (a vector) per chunk in a vector storeVector store: A database that keeps embeddings (lists of numbers that stand for the meaning of text chunks) and returns the chunks whose embeddings are nearest to a query's..
- Retrieve. Turn the question into the same kind of vector and fetch the nearest chunks. Hybrid searchHybrid search: Retrieval that runs a keyword search (such as BM25) and a vector search for the same query, then merges the two ranked lists into one. also matches keywords, which catches names and error codes that vectors miss.
- Generate. Put the chunks in the prompt with an instruction to answer only from them, and ideally to cite them.
Example
A helpdesk bot is asked "How do I reset my password?". Retrieval returns the FAQ chunk about password resets; the prompt says "Use the context to answer"; the model replies with the steps from that chunk and marks it as its source.
Where it goes wrong
- Bad retrieval. The right chunk is never found, so the model answers from nothing, or guesses.
- Ignored context. The model answers from memory anyway. Citations make this visible.
- Poisoned content. A retrieved document can carry instructions for the model, a form of indirect prompt injectionIndirect prompt injection: Prompt injection in which the attacker's instructions hide in content the model reads while doing its job, such as a web page, an email, a document or a tool result.. OWASP lists vector and embedding weaknesses as LLM08.
Common confusion
RAG is not fine-tuning. Fine-tuning changes the model's weights; RAG leaves the model alone and changes the prompt, so new documents count as soon as they are ingested.
In L2M
Knowledge bases give you a built-in vector store in the L2M database. The RAG Retrieve node searches one by vector, keyword (BM25) or hybrid, and can emit citation instructions; a Rerank node can reorder results. Try it with no API key in the RAG knowledge base recipe.