What is RAG?
RAG (Retrieval-Augmented Generation) is the technique of fetching relevant documents from your own data and injecting them into the prompt before asking the LLM to answer. Instead of the model hallucinating from training data, it answers from documents you provide at query time.
RAG is the foundation of most enterprise LLM applications — document Q&A, internal knowledge bases, customer support bots, and code search.
Step 1 — Load documents
LangChain provides loaders for dozens of file formats. They all return a list of Document objects with page_content and metadata.
from langchain_community.document_loaders import (
PyPDFLoader,
TextLoader,
WebBaseLoader,
)
# Load a PDF
loader = PyPDFLoader("report.pdf")
docs = loader.load() # list of Document objects, one per page
# Load a webpage
loader = WebBaseLoader("https://example.com/docs")
docs = loader.load()
# Load a plain text file
loader = TextLoader("notes.txt")
docs = loader.load()Each Document has .page_content (the text) and .metadata (source, page number, etc.).
Step 2 — Chunk the text
LLMs have context limits, and embeddings work best on focused passages — not entire documents. You split documents into smaller chunks before embedding.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=500, # characters per chunk
chunk_overlap=50, # overlap to preserve context across chunk boundaries
)
chunks = splitter.split_documents(docs)Chunking strategies compared
| Strategy | How it splits | Best for |
|---|---|---|
| RecursiveCharacter | Tries paragraphs → sentences → words → characters | General prose — the default choice |
| Character | Splits on a fixed separator (e.g. "\n\n") | Structured text with clear delimiters |
| Token | Splits by token count using the model's tokenizer | When exact token budgets matter |
| Markdown / HTML | Splits on headings, sections, tags | Docs, wikis, structured markup |
| Semantic | Splits where sentence embeddings shift topic | High-quality retrieval, slower preprocessing |
chunk_overlap prevents answers from being split across boundaries — 10–15% of chunk_size is a good starting point.
Step 3 — Embed and store
An embedding model converts each text chunk into a numeric vector. Similar chunks produce similar vectors — this is what enables semantic search.
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# embed all chunks and store in FAISS (in-memory, no server needed)
vectorstore = FAISS.from_documents(chunks, embeddings)
# save to disk so you don't re-embed every time
vectorstore.save_local("faiss_index")
# load later
vectorstore = FAISS.load_local("faiss_index", embeddings,
allow_dangerous_deserialization=True)Step 4 — Retrieve
A retriever takes a query string and returns the most relevant chunks. The default is similarity search (cosine distance between embeddings).
retriever = vectorstore.as_retriever(
search_type="similarity", # or "mmr"
search_kwargs={"k": 4}, # return top-4 chunks
)
docs = retriever.invoke("What is the refund policy?")MMR — Maximum Marginal Relevance
Instead of returning the top-k most similar chunks (which may all say the same thing), MMR balances relevance with diversity. Use search_type="mmr" when your chunks have redundant content.
Multi-query retrieval
Generate multiple phrasings of the user's question and retrieve for each. Useful when the user's phrasing doesn't match how the document was written.
from langchain.retrievers import MultiQueryRetriever
retriever = MultiQueryRetriever.from_llm(retriever=base_retriever, llm=llm)
Step 5 — Answer with a RAG chain
Combine the retriever and LLM into an LCEL chain that fetches context and answers in one call.
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
prompt = ChatPromptTemplate.from_template("""
You are an assistant. Answer the question using only the context below.
If the answer is not in the context, say "I don't know."
Context:
{context}
Question: {question}
""")
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
answer = rag_chain.invoke("What is the refund policy?")Common mistakes
Chunks that are too large
A 2,000-token chunk dilutes the signal. The embedding captures the average meaning of the whole chunk, so precise facts get lost. Start with 300–600 characters and tune from there.
Chunks that are too small
50-character chunks lose surrounding context. The retrieved chunk may contain the answer but not enough context for the LLM to interpret it correctly. Overlap helps but doesn't fix fundamentally tiny chunks.
Re-embedding on every startup
Save your FAISS index with .save_local() and reload it. Re-embedding a large document set on every restart is slow and wastes API credits.
Using k=1 (only one retrieved chunk)
The answer may span multiple chunks. Start with k=4 or k=5, then reduce if the context window is too full. More chunks = higher chance of finding the answer, at the cost of a larger prompt.
Not telling the model to say "I don't know"
Without explicit instruction, the model fills gaps with training data. Add "If the answer is not in the context, say I don't know" to your system prompt.
Quick summary
- RAG: embed your docs, find relevant chunks at query time, inject them into the prompt
- Use
RecursiveCharacterTextSplitterwith chunk_size 300–600 and 10–15% overlap as a starting point - FAISS is great for local dev; use Pinecone / Chroma for persistent production stores
- MMR retrieval adds diversity; multi-query retrieval handles phrasing mismatches
- Always tell the model to say "I don't know" when the answer isn't in the context
- Save your vector index to disk — never re-embed the same documents twice
LangChain RAG Pipeline FAQ
What is RAG (Retrieval-Augmented Generation)?
RAG is the technique of fetching relevant documents from your own data and injecting them into the prompt before the LLM answers. Instead of hallucinating from training data, the model answers from documents you provide at query time, which is why RAG underpins most enterprise document Q&A and knowledge-base applications.
What chunk size should I use for RAG?
Start with a chunk_size of about 300-600 characters and a chunk_overlap of 10-15%. Chunks that are too large dilute the embedding signal so precise facts get lost; chunks that are too small lose surrounding context. Tune from this starting point based on your retrieval quality.
Which vector store should I use for RAG in LangChain?
FAISS is great for local development because it runs in-memory with no server. Chroma is easy to set up with persistence. For production scale, use a hosted store such as Pinecone, Weaviate, or Qdrant. All expose the same retriever interface in LangChain, so you can swap them with minimal code changes.
What is the difference between similarity search and MMR retrieval?
Similarity search returns the top-k chunks with the closest embeddings, which can be redundant if several chunks say the same thing. MMR (Maximum Marginal Relevance) balances relevance with diversity, so you get varied supporting passages. Use search_type='mmr' when your chunks contain overlapping content.
How do I stop a RAG chatbot from hallucinating?
Explicitly instruct the model to answer only from the retrieved context and to say 'I don't know' when the answer is not present. Retrieve enough chunks (start with k=4 or 5) so the answer is actually in context, and keep chunks focused so the relevant facts are not diluted.