Limits of Naive RAG
Basic RAG — embed the query, find the top-k nearest chunks — works well for simple lookups but fails when:
Query-document mismatch
User questions are short and conversational; documents are long and technical. Their embedding spaces don't align well.
Chunk boundary artifacts
The answer spans multiple chunks, but only one is retrieved — leaving the LLM with incomplete context.
Noisy chunks
Retrieved chunks contain the right sentence buried in irrelevant text, wasting context window space.
Single-perspective retrieval
One query formulation may miss relevant documents phrased differently in the corpus.
RAG Parameter Tuning Guide
Before adding complexity with HyDE or re-ranking, tune the basic retrieval parameters. Small adjustments here often yield 20–30% quality improvements.
chunk_size (text splitting)
| Use Case | Recommended chunk_size | Why |
| Q&A (e.g., FAQ) | 256–512 | Answer fits in chunk, less noise |
| Semantic search | 512–1024 | Balanced precision & context |
| Summarization | 1024–2048 | More context needed for summary |
| Legal/technical docs | 2048+ | Dense text, answer spans paragraphs |
k (retrieval count)
Default: k=4 — retrieves 4 top chunks
Increase k if: Questions require multi-paragraph context, or your corpus is noisy. Try k=8 or k=16.
Decrease k if: Context window fills up quickly or LLM gets confused by too many options. Try k=2 or k=3.
Trade-off: Higher k = more context but slower retrieval + more LLM tokens. Test empirically for your use case.
chunk_overlap
Standard: chunk_overlap = 0.2 × chunk_size
Why overlap? Prevents answers split across boundaries from being missed.
Example: chunk_size=1024, chunk_overlap=200 means 80% new content per chunk.
Tuning: Increase if boundaries are causing misses. Decrease if cost/latency matters more than recall.
When to use which technique
↓ Try HyDE first if:
Questions are phrased differently from documents (e.g., user asks "how do I X" but docs describe "X process")
↓ Use Multi-Query if:
A single query formulation misses relevant documents in different phrasings
↓ Use Parent-Document Retriever if:
Chunks are too small to provide context, or related chunks should be grouped
↓ Add Re-ranking / Compression if:
Retrieved chunks contain the right info buried in noise, or you want to trim irrelevant sentences
Tuning Order
- Fix chunk_size for your use case
- Tune k (number of results) empirically
- Only then add HyDE, re-ranking, etc.
Quick Wins
- Increase k from 4 → 8
- Add chunk_overlap
- Use MMR search instead of pure similarity
HyDE — Hypothetical Document Embeddings
Instead of embedding the raw query, ask the LLM to generate a hypothetical answer, then embed that. The hypothetical answer lives in the same embedding space as real documents, dramatically improving retrieval.
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
llm = ChatOpenAI(model="gpt-4o-mini")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# Step 1: generate a hypothetical answer
hyde_prompt = ChatPromptTemplate.from_template(
"Write a short paragraph that directly answers this question:\n{question}"
)
hyde_chain = hyde_prompt | llm | StrOutputParser()
# Step 2: embed the hypothetical answer and retrieve
vectorstore = Chroma(embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
from langchain_core.runnables import RunnablePassthrough
rag_chain = (
{"question": RunnablePassthrough()}
| RunnablePassthrough.assign(hypothesis=hyde_chain)
| RunnablePassthrough.assign(
docs=lambda x: retriever.invoke(x["hypothesis"]) # search with hypothesis
)
)
result = rag_chain.invoke("How does LangGraph handle state?")Multi-Query Retrieval
Generate multiple phrasings of the user query, retrieve for each, and merge results. This catches relevant documents missed by a single query angle.
from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_openai import ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
vectorstore = Chroma(embedding_function=OpenAIEmbeddings())
base_retriever = vectorstore.as_retriever()
llm = ChatOpenAI(model="gpt-4o-mini")
multi_retriever = MultiQueryRetriever.from_llm(
retriever=base_retriever,
llm=llm,
# Default: generates 3 alternative query phrasings
)
# Under the hood: generates 3 queries, retrieves for each, deduplicates
docs = multi_retriever.invoke("How do I add memory to a LangChain agent?")The retriever de-duplicates results by document ID, so you get the union of all retrieved docs without duplicates.
Parent-Document Retriever
Index small chunks for precise retrieval, but return the full parent document (or a larger chunk) to the LLM. Small chunks match better; large chunks provide more context.
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
# Child splitter: small chunks for precise embedding search
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)
# Parent splitter: larger context returned to LLM
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
vectorstore = Chroma(embedding_function=OpenAIEmbeddings())
docstore = InMemoryStore() # stores parent documents
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=docstore,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
retriever.add_documents(docs)
# Retrieval: matches on small child chunks, returns large parent
results = retriever.invoke("How does LangChain memory work?")Re-Ranking with Cross-Encoders
A bi-encoder (embedding) retrieves fast but imprecisely. A cross-encoder re-ranks the top-k candidates by scoring query-document pairs jointly — much more accurate but too slow for full corpus search.
# Option 1: Cohere re-ranker (API)
from langchain.retrievers.contextual_compression import ContextualCompressionRetriever
from langchain_cohere import CohereRerank
from langchain_community.vectorstores import Chroma
base_retriever = Chroma(embedding_function=embeddings).as_retriever(
search_kwargs={"k": 20} # retrieve wide, then re-rank narrow
)
compressor = CohereRerank(model="rerank-english-v3.0", top_n=5)
retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=base_retriever,
)
docs = retriever.invoke("What is LCEL?")
# Option 2: Local cross-encoder (no API cost)
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
cross_encoder = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")
local_reranker = CrossEncoderReranker(model=cross_encoder, top_n=5)
retriever = ContextualCompressionRetriever(
base_compressor=local_reranker,
base_retriever=base_retriever,
)Contextual Compression
Strip irrelevant sentences from retrieved chunks before passing them to the LLM — reducing noise and token cost.
from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain.retrievers.contextual_compression import ContextualCompressionRetriever
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(
base_compressor=compressor,
base_retriever=base_retriever,
)
# Each chunk is filtered down to only the sentences relevant to the query
compressed_docs = compression_retriever.invoke("What are LangChain callbacks?")Stack them: combine re-ranking (remove irrelevant documents) with contextual compression (trim irrelevant sentences) for the best quality. The pattern is always ContextualCompressionRetriever(compressor=X, base_retriever=Y).
Adaptive & Self-Corrective RAG
Use LangGraph to build retrieval loops that check whether the retrieved context is sufficient before answering — and re-retrieve if not.
from langgraph.graph import StateGraph, END
from typing import TypedDict, List
class RAGState(TypedDict):
question: str
documents: List[str]
answer: str
attempts: int
def retrieve(state: RAGState) -> RAGState:
docs = retriever.invoke(state["question"])
return {"documents": [d.page_content for d in docs], "attempts": state.get("attempts", 0) + 1}
def grade_documents(state: RAGState) -> str:
"""LLM judges whether retrieved docs answer the question."""
grade = grading_chain.invoke({
"question": state["question"],
"documents": "\n".join(state["documents"]),
})
if grade == "sufficient" or state["attempts"] >= 3:
return "generate"
return "rewrite_query"
def rewrite_query(state: RAGState) -> RAGState:
better_q = rewrite_chain.invoke({"question": state["question"]})
return {"question": better_q}
def generate(state: RAGState) -> RAGState:
answer = rag_chain.invoke(state)
return {"answer": answer}
graph = StateGraph(RAGState)
graph.add_node("retrieve", retrieve)
graph.add_node("rewrite_query", rewrite_query)
graph.add_node("generate", generate)
graph.set_entry_point("retrieve")
graph.add_conditional_edges("retrieve", grade_documents, {"generate": "generate", "rewrite_query": "rewrite_query"})
graph.add_edge("rewrite_query", "retrieve")
graph.add_edge("generate", END)
app = graph.compile()Advanced RAG FAQ
New to retrieval? Start with the RAG Pipeline tutorial before layering on the advanced techniques below.
What is reranking in RAG?
Reranking is a second-stage retrieval step that reorders the candidate chunks returned by the vector store using a more accurate model. A fast bi-encoder embedding search retrieves a wide set of candidates, then a cross-encoder reranker scores each query-document pair jointly and keeps only the top few. This lifts the most relevant chunks to the top and removes near-misses, which is one of the highest-impact upgrades you can make to a LangChain RAG pipeline.
What is query expansion in RAG?
Query expansion rewrites or augments the original user query into several richer formulations before retrieval, so the search is not limited to one phrasing. In LangChain this is commonly done with MultiQueryRetriever, which uses an LLM to generate alternative phrasings, retrieves for each, and merges the de-duplicated results. It catches relevant documents that a single query angle would miss.
What is multi-query retrieval in LangChain?
Multi-query retrieval is LangChain's built-in form of query expansion. MultiQueryRetriever.from_llm wraps a base retriever and an LLM, generates three alternative phrasings of the question by default, runs retrieval for each phrasing, and returns the union of results de-duplicated by document ID. It is useful when users phrase questions differently from how the source documents are written.
What is hybrid search in RAG?
Hybrid search combines dense vector similarity with sparse keyword search such as BM25, then fuses the two ranked lists. Dense search captures semantic meaning while keyword search captures exact terms, identifiers, and rare tokens that embeddings often miss. In LangChain you can blend retrievers with EnsembleRetriever and weight each component to balance semantic recall against exact-match precision.
What is HyDE in RAG?
HyDE (Hypothetical Document Embeddings) asks the LLM to generate a hypothetical answer to the query first, then embeds that hypothetical answer instead of the raw question. Because the generated text lives in the same embedding space as real documents, it aligns far better with the corpus than a short conversational question, which improves retrieval when queries and documents are phrased very differently.
How do I improve RAG retrieval quality in LangChain?
Start by tuning the basics: pick a chunk_size suited to your use case, set chunk_overlap to about 20 percent, and tune k empirically before adding complexity. Then layer in advanced techniques where they help: HyDE for query-document mismatch, multi-query expansion for phrasing variety, parent-document retrieval for context, and reranking plus contextual compression to strip noise. Combine them with ContextualCompressionRetriever for the best production quality. See the RAG Pipeline tutorial for the foundations.