RAG Design Patterns

Overview

Every RAG system uses one or more of five core patterns: Naive RAG (the baseline), Hybrid RAG (combining dense and sparse retrieval), Agentic RAG (multi-step reasoning with tool use), Graph RAG (entity-relationship traversal), and Memory-augmented RAG (conversation-aware retrieval). These patterns are not mutually exclusive — production systems often combine two or three — but each one solves a distinct retrieval problem, and adding complexity before it is needed creates maintenance burden without quality gains.

Pattern selection should be driven by the failure mode you observe, not by the sophistication of the pattern. Start with Naive RAG for every new use case. If keyword queries fail — users searching for exact product codes, error messages, or proper nouns — add the BM25 layer of Hybrid RAG. If questions require reasoning across multiple retrieval steps, introduce Agentic RAG. If your data is inherently relational — entities with typed connections — Graph RAG becomes appropriate. Memory augmentation is needed only when follow-up questions without explicit context ("does that apply to me?") are a significant portion of your query traffic.

The patterns also differ substantially in operational complexity. Naive and Hybrid RAG are stateless per query and easy to debug. Agentic RAG introduces state across retrieval iterations and is harder to trace. Graph RAG requires maintaining a knowledge graph in sync with source documents. Memory-augmented RAG requires conversation storage alongside document retrieval. Factor in the engineering cost of operation, not just the quality gain, when choosing a pattern for production.

1. Naive RAG (Baseline)

Simple: retrieve top-K docs, pass to LLM. Best for: MVP, simple Q&A, small docs.

Python

query → embed → search → LLM answer

Pros: Simple, fast
Cons: Low quality, hallucinations, no reasoning

2. Hybrid RAG

Combines dense (semantic) + sparse (keyword) search + reranking. Best for: Production, technical docs.

YAML

query → [dense search + BM25] → combine → rerank → LLM

Pros: Better quality, catches keywords
Cons: Slower, more complex

3. Agentic RAG (ReAct)

LLM decides when to retrieve, search web, use tools. Best for: Complex questions, multi-step reasoning.

YAML

LLM: "I need to search docs AND search web"
→ [retrieve + search_web]
→ Combine results
→ Answer

Pros: Flexible, can adapt
Cons: Slower, harder to control

4. Graph RAG

Structure docs as knowledge graph, traverse edges. Best for: Highly interconnected data, entity relationships.

Python

Query: "What treatments work for diabetes?"

Graph structure:
Diabetes → [Medications: Insulin, Metformin, ...]
        → [Complications: Heart disease, Kidney disease, ...]
        → [Lifestyle: Diet, Exercise]

Traverse graph → Retrieve relevant paths → Answer

5. Memory-Augmented RAG

Keep conversation history, retrieve relevant past context. Best for: Multi-turn conversations.

Python

Turn 1: User: "Tell me about taxes"
        RAG retrieves and answers

Turn 2: User: "Can I claim that as deduction?"
        RAG: Uses context from Turn 1 + new retrieval
        → Better understanding

Notes

HyDE: embed a hypothetical answer, not the query

Hypothetical Document Embeddings (HyDE) generate a plausible answer to the question using the LLM, embed that answer, and use the answer embedding for retrieval rather than the query embedding. This improves recall when query phrasing differs significantly from document language — e.g., a question phrased conversationally retrieving technical documentation. The trade-off is an extra LLM call per query before retrieval even begins.

CRAG adds a retrieval quality classifier as a fallback gate

Corrective RAG (CRAG) adds a lightweight binary classifier after retrieval: if the top-K results score below a relevance threshold, the system falls back to web search before generating. This is powerful for corpora that are intentionally narrow (a company's internal docs) where many user questions fall outside the index. The classifier can be a simple cross-encoder confidence score rather than a separate ML model.

Query routing reduces noise before retrieval starts

A routing layer classifies each incoming query and sends it to a specialized retrieval pool (billing index, product catalog, technical docs) before the main retrieval step. This narrows the embedding space for each pool, improving precision without any changes to the retrieval algorithm itself. A lightweight classifier (fine-tuned DistilBERT or a few-shot prompted LLM) can route queries with high accuracy at minimal latency cost.

Graph traversal depth must be bounded in Graph RAG

Multi-hop graph traversal can generate exponentially large subgraphs: at depth 3 with average degree 10, you are exploring up to 1,000 nodes. Always bound traversal depth (typically 2–3 hops) and prune by edge weight or semantic relevance at each hop. Without bounds, a single query can exhaust graph DB memory and produce a context window too large for the LLM to process effectively.

RAG Design Patterns FAQ

What is Agentic RAG?

Agentic RAG gives an LLM agent tools to perform multiple retrieval calls in a reasoning loop. The agent decides when to retrieve, what to query, and whether the results are sufficient before generating a final answer.

What is Graph RAG?

Graph RAG indexes documents into a knowledge graph (entities + relationships) in addition to a vector store. At query time it combines graph traversal with vector search, which is powerful for multi-hop questions that require connecting facts across documents.

What is multi-hop retrieval in RAG?

Multi-hop retrieval performs sequential retrievals where the result of the first retrieval informs the next query. It is used for questions like "Who founded the company that made X?" which require connecting two pieces of information.

When should I use HyDE (Hypothetical Document Embeddings)?

HyDE generates a hypothetical answer to the query using the LLM, embeds that answer, and uses its embedding for retrieval instead of the query embedding. It improves recall when query phrasing differs significantly from document phrasing.

What RAG design pattern is best for customer support bots?

A hybrid pattern works well: sparse (BM25) retrieval for exact product names and error codes, dense retrieval for general questions, and a reranker to merge results. Add a structured FAQ lookup as a fallback for common questions.