Advanced Techniques

Overview

Basic RAG — retrieve top-K chunks and pass them to an LLM — works well for simple factual Q&A over well-structured documents. It starts failing when questions are ambiguous, when answers span multiple documents, when the LLM needs to reason iteratively, or when your data changes faster than you can re-index. Advanced techniques address each of these failure modes individually.

Recursive retrieval handles long documents by indexing summaries first and drilling into relevant sections on demand. Agentic RAG replaces the fixed retrieve-once loop with a reasoning agent that can retrieve, analyze gaps in its knowledge, and query again before answering. Self-reflection adds a verification pass after generation so hallucinations are caught internally rather than reaching users. Knowledge graphs let the retriever follow entity relationships rather than relying purely on embedding similarity. Adaptive retrieval dynamically adjusts how many documents to fetch based on estimated query complexity.

When adopting these techniques, match the solution to the actual failure mode you observe. If retrieval precision is already high but hallucinations persist, self-reflection and stronger system prompts will help more than adding retrieval complexity. If the right documents are indexed but ranked poorly, reranking and hybrid search are the right levers. Start simple and add complexity only when the data shows you need it.

1. Recursive Retrieval & Summarization

When a single document is too long, retrieve its summary first, then retrieve relevant chunks within it.

Step 1: Index summaries (small)
summaries = [
  {"id": "doc1_summary", "summary": "About tax deductions...", "source_doc": "doc1"},
  ...
]

Step 2: Search summaries
top_summaries = vector_db.search(query_embedding, k=5)

Step 3: Retrieve chunks from relevant docs
relevant_chunks = []
for summary in top_summaries:
    source_doc = summary.source_doc
    chunks = db.query(source_doc, k=10)
    relevant_chunks.extend(chunks)

Result: Top 5 most relevant documents, with best chunks from each.

2. Agentic RAG (Tool Use & ReAct)

Instead of retrieving and answering, let the LLM decide: should I retrieve more docs? Search web? Use a tool?

agent = create_agent(
    tools=[
        {"name": "search_documents", "description": "Search internal docs"},
        {"name": "search_web", "description": "Search internet"},
        {"name": "query_database", "description": "Query structured data"}
    ],
    llm=gpt4
)

# LLM decides which tool to use
response = agent.run(
    "What's the latest COVID guidance AND historical data?"
)

# Agent logic:
# 1. User asks question
# 2. LLM decides to use "search_documents" + "search_web"
# 3. Executes both tools
# 4. Combines results
# 5. Generates final answer

Best for: Complex questions, multi-step reasoning, uncertain data freshness.

3. Self-Reflection & Verification

Have the LLM verify its own answer against retrieved docs. Catch hallucinations before returning.

Step 1: Generate answer
answer = llm.generate(query, retrieved_docs)

Step 2: Self-verify
verification_prompt = f"""
You just gave this answer: {answer}

Does it match the retrieved documents? If not, correct it.
Documents: {retrieved_docs}
"""

verified = llm.generate(verification_prompt)

Step 3: If confidence low, retrieve more
if verified.confidence < 0.7:
    more_docs = retrieve(query, k=10)
    answer = llm.generate(query, more_docs + retrieved_docs)

return answer

4. Knowledge Graphs for Structured Retrieval

For highly structured data, use knowledge graphs instead of pure semantic search.

5. Adaptive Retrieval (Knowing When to Stop)

Don't always retrieve fixed K documents. Stop when confident you have enough.

def adaptive_retrieve(query, max_docs=20):
    retrieved = []
    confidence = 0
    k = 5

    while confidence < 0.8 and len(retrieved) < max_docs:
        batch = vector_db.search(query, k=k)
        retrieved.extend(batch)

        # Compute confidence: diversity + relevance
        unique_topics = len(set(doc.topic for doc in retrieved))
        avg_score = sum(doc.score for doc in retrieved) / len(retrieved)
        confidence = (unique_topics / k) * avg_score

        k = min(k + 5, 20)  # Increase batch size

    return retrieved[:max_docs]

Benefit: Reduces costs, faster for simple queries, better UX for complex queries.

Notes

Advanced RAG Techniques FAQ