How to Build a Local RAG Pipeline with LangChain and Ollama

What you're building

A command-line chatbot that answers questions about your documents — PDFs, text files, anything. You load documents once, then query them instantly using a local LLM. No internet required, no API keys, complete privacy.

By the end, you'll have two Python scripts: ingest.py (loads and indexes documents) and rag.py (queries them). You'll run ingest once, rag as many times as you want.

Prerequisites

Project setup

mkdir local-rag && cd local-rag
pip install langchain-ollama langchain-community langchain-text-splitters chromadb pypdf

local-rag/
├── docs/          ← Put your PDFs here
├── ingest.py      ← Run once to index documents
├── rag.py         ← Run repeatedly to query
└── chroma_db/     ← Created automatically by ingest.py

Download the models you'll need

Before running any code, pull the two models that the pipeline uses: an embedding model (for converting text to numbers) and a chat model (for generating answers).

ollama pull nomic-embed-text    # 686 MB, 1-2 min
ollama pull llama3.1:8b         # 4.7 GB, 5-10 min depending on connection

Step 1: Create ingest.py (indexes your documents)

This script reads PDFs from the docs/ folder, splits them into chunks, embeds them with the local model, and saves the vector store to disk for reuse.

from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import Chroma

DOCS_DIR = "./docs"
CHROMA_DIR = "./chroma_db"
EMBED_MODEL = "nomic-embed-text"

def ingest():
    # Load all PDFs from docs/ directory
    loader = DirectoryLoader(DOCS_DIR, glob="**/*.pdf", loader_cls=PyPDFLoader)
    raw_docs = loader.load()
    print(f"Loaded {len(raw_docs)} pages from {DOCS_DIR}")

    # Split into overlapping chunks (512 tokens = good default)
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=512,
        chunk_overlap=64,
        separators=["\n\n", "\n", ".", " ", ""],
    )
    chunks = splitter.split_documents(raw_docs)
    print(f"Split into {len(chunks)} chunks")

    # Embed with local model
    embeddings = OllamaEmbeddings(model=EMBED_MODEL)

    # Save to disk — reusable across runs
    vectorstore = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=CHROMA_DIR,
    )
    print(f"Stored {len(chunks)} chunks in {CHROMA_DIR}")

if __name__ == "__main__":
    ingest()

Step 2: Create rag.py (query your documents)

This script loads the vector store and runs a loop that lets you ask questions. It retrieves relevant chunks and sends them to the local LLM, which generates answers.

from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

CHROMA_DIR = "./chroma_db"
EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "llama3.1:8b"

# Load the vector store from disk
embeddings = OllamaEmbeddings(model=EMBED_MODEL)
vectorstore = Chroma(persist_directory=CHROMA_DIR, embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# System prompt — tells the LLM to use only the provided context
prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a helpful assistant. Answer the question using ONLY the
context provided below. If the answer is not in the context, say so clearly.

Context:
{context}"""),
    ("human", "{question}"),
])

llm = ChatOllama(model=CHAT_MODEL, temperature=0)

def format_docs(docs):
    return "\n\n---\n\n".join(doc.page_content for doc in docs)

# RAG chain: retrieve → format → prompt → LLM → output
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

if __name__ == "__main__":
    while True:
        q = input("\nQuestion (or 'quit'): ").strip()
        if q.lower() == "quit":
            break
        answer = rag_chain.invoke(q)
        print(f"\nAnswer: {answer}")

Expected output and example interaction

$ python ingest.py
Loaded 15 pages from ./docs
Split into 127 chunks
Stored 127 chunks in ./chroma_db

$ python rag.py

Question (or 'quit'): What is the main topic of the document?
Answer: The document discusses machine learning fundamentals, including supervised and
unsupervised learning approaches, with emphasis on neural networks.

Question (or 'quit'): How does gradient descent work?
Answer: Gradient descent is an optimization algorithm that iteratively updates model
parameters in the direction of steepest descent to minimize the loss function...

Next: Enhance your pipeline (optional)

The basic setup above works. The sections below add optional improvements — use them only if you need better quality.

Enhancement 1: Optimize chunking for your documents

The default chunk size of 512 tokens is a safe starting point, but different documents benefit from different sizes. The key insight: there's a tradeoff between precision (small chunks) and context (large chunks).

Enhancement 2: Hybrid search (keyword + semantic)

Vector search excels at semantic similarity but misses exact keyword matches. Hybrid search runs both in parallel, then merges results with Reciprocal Rank Fusion (RRF). This usually outperforms either method alone and is especially effective for product names, error codes, and proper nouns.

How to enable: Replace only the retriever setup in rag.py. Everything else stays the same.

pip install rank-bm25

# Import at the top
from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever

# Load the vector store
embeddings = OllamaEmbeddings(model=EMBED_MODEL)
vectorstore = Chroma(persist_directory=CHROMA_DIR, embedding_function=embeddings)

# BM25 needs chunks (re-embed from your documents)
from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
loader = DirectoryLoader("./docs", glob="**/*.pdf", loader_cls=PyPDFLoader)
raw_docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_documents(raw_docs)

# Create both retrievers
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 4
vector_retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

# Ensemble: 40% keyword, 60% semantic (adjust weights for your use case)
retriever = EnsembleRetriever(
    retrievers=[bm25_retriever, vector_retriever],
    weights=[0.4, 0.6],
)

Enhancement 3: Reranking (improve answer quality)

Retrieval returns top-k candidates — but not all are equally relevant. A reranker uses a smaller, cross-encoder model to re-score results for higher precision. The pattern: retrieve many, rerank to few. This costs minimal overhead and significantly improves quality.

pip install sentence-transformers

from sentence_transformers import CrossEncoder
from langchain_core.documents import Document

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank(query: str, docs: list[Document], top_n: int = 3) -> list[Document]:
    pairs = [(query, doc.page_content) for doc in docs]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, docs), key=lambda x: x[0], reverse=True)
    return [doc for _, doc in ranked[:top_n]]

# Retrieve 8, rerank to top 3
broad_retriever = vectorstore.as_retriever(search_kwargs={"k": 8})

def retrieve_and_rerank(query):
    docs = broad_retriever.invoke(query)
    return rerank(query, docs, top_n=3)

# Update rag_chain to use reranking
rag_chain = (
    {"context": (lambda q: format_docs(retrieve_and_rerank(q))),
     "question": RunnablePassthrough()}
    | prompt | llm | StrOutputParser()
)

Enhancement 4: Return source citations

Always show which document chunks were used. This builds trust and lets users verify answers — especially important for local models that may hallucinate more than cloud models.

from langchain_core.runnables import RunnableParallel

# Add this after the rag_chain definition
rag_with_sources = RunnableParallel(
    {"answer": rag_chain, "source_docs": retriever}
)

if __name__ == "__main__":
    while True:
        q = input("\nQuestion (or 'quit'): ").strip()
        if q.lower() == "quit":
            break
        result = rag_with_sources.invoke(q)

        print(f"\nAnswer: {result['answer']}")
        print("\nSources:")
        for doc in result["source_docs"]:
            src = doc.metadata.get("source", "unknown")
            page = doc.metadata.get("page", "?")
            print(f"  • {src} (page {page})")

Deployment: Docker (when to use)

Use Docker if: You want to share this with team members, run it on a server, or ensure reproducibility. Skip if: You're just testing locally on your laptop.

Docker packages your code + Python + all dependencies into a container. You run docker compose instead of managing Ollama and Python separately.

services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ~/.ollama:/root/.ollama   # persist models so you don't re-download
    restart: unless-stopped

  rag-app:
    build: .
    depends_on:
      - ollama
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./docs:/app/docs
      - ./chroma_db:/app/chroma_db

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip3 install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "rag.py"]

langchain-ollama
langchain-community
langchain-text-splitters
chromadb
pypdf

# Start Ollama and the app
docker compose up -d

# Pull models (run in one of the containers)
docker compose exec ollama ollama pull nomic-embed-text
docker compose exec ollama ollama pull llama3.1:8b

# Run ingestion
docker compose exec rag-app python ingest.py

# Run the chatbot
docker compose exec rag-app python rag.py

Troubleshooting

Production tips that move the needle

Once the default pipeline is running, these are the changes that most consistently improve answer quality and retrieval precision on real document sets. Apply in order — each one is independent.

Local RAG Pipeline FAQ

Related tutorials

This tutorial builds on the LangChain with local models guide — review it for ChatOllama and OllamaEmbeddings fundamentals. If your answers aren't reliable enough, understand why in the limitations of local models tutorial. New to Ollama? Start with the Ollama setup guide.