Local RAG Setup with Ollama & FAISS

Q: Can I run a RAG system entirely locally without any API keys?

Yes. Use Ollama to run a local LLM (Llama 3, Mistral, Phi-3), sentence-transformers for local embeddings, and FAISS as an in-process vector store. The entire pipeline runs on your machine with no external API calls.

Q: What is Ollama and how does it work for local RAG?

Ollama is a desktop application that runs open-source LLMs locally via a local REST API on port 11434. It handles model downloads, GPU/CPU inference, and exposes an OpenAI-compatible API, making it easy to swap into any RAG pipeline.

Q: How fast is local RAG compared to cloud-based RAG?

On Apple Silicon or a modern GPU, local RAG generates at 20–60 tokens/second — comparable to streaming from a cloud API. Retrieval (FAISS) is typically faster than cloud vector DBs since there's no network overhead.

Q: What hardware do I need for local RAG?

A Mac with Apple Silicon (M1+) handles 7B–13B parameter models smoothly with 16 GB RAM. For 70B models, you need 64 GB RAM or an NVIDIA GPU with 40+ GB VRAM. Smaller 3B–7B models run on most laptops.

Q: Is local RAG private and secure?

Yes — all data stays on your machine. No document content, queries, or embeddings are sent to any external API. This makes local RAG the right choice for sensitive legal, medical, or proprietary business data.

Full Offline Stack

Python

# 1. Install Ollama (LLM inference)
# Download from ollama.ai
ollama pull mistral  # ~7B fast model
# or llama2:70b for highest quality

# 2. Install dependencies
pip install sentence-transformers faiss-cpu langchain

# 3. Run full RAG locally
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitters import TokenTextSplitter
from sentence_transformers import SentenceTransformer
import faiss
import requests
import json

# Load docs
loader = PyPDFLoader("document.pdf")
docs = loader.load()

# Chunk
splitter = TokenTextSplitter(chunk_size=400)
chunks = splitter.split_documents(docs)

# Embed locally
model = SentenceTransformer('all-mpnet-base-v2')
embeddings = model.encode([c.page_content for c in chunks])

# Store in FAISS
dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)
index.add(embeddings.astype('float32'))

# Query
query = "How do I file taxes?"
query_emb = model.encode(query).reshape(1, -1)
distances, indices = index.search(query_emb, k=5)

retrieved = [chunks[i].page_content for i in indices[0]]

# Call local LLM (Ollama)
prompt = f"""Context: {retrieved}
Question: {query}
Answer:"""

response = requests.post('http://localhost:11434/api/generate',
    json={"model": "mistral", "prompt": prompt, "stream": False}
)
answer = response.json()['response']
print(answer)

Ollama for Local LLMs

Fast & Small (7B models)

Mistral, TinyLlama. ~4GB RAM, fast. Trade-off: lower quality.

Balanced (13B models)

Llama 2 13B, Neural Chat. ~8GB RAM. Good balance.

Powerful (70B models)

Llama 2 70B. ~40GB RAM (needs GPU). Highest quality.

Recommendation: Start with Mistral (7B). Upgrade to Llama 13B if quality matters.

FAISS for Vector Storage

Python

# FAISS usage
import faiss
import numpy as np

# Create index
dimension = 768  # embedding dimension
index = faiss.IndexFlatL2(dimension)  # L2 distance

# Add vectors
embeddings = np.random.random((10000, 768)).astype('float32')
index.add(embeddings)

# Search
query = np.random.random((1, 768)).astype('float32')
distances, indices = index.search(query, k=5)

# Save/load
faiss.write_index(index, "index.faiss")
index = faiss.read_index("index.faiss")

Pros: Blazingly fast, in-memory, no DB overhead. Cons: All data in RAM, no filtering.

Performance Expectations

On MacBook Pro M1 (16GB RAM)

Embedding: ~200 docs/sec

Search: ~5ms for 10K vectors

LLM: ~10 tokens/sec (Mistral 7B)

On GPU (RTX 4090)

Embedding: ~5000 docs/sec

Search: <1ms

LLM: ~100 tokens/sec (Llama 70B)

Notes

FAISS is in-process only — no persistence without extra steps

FAISS lives entirely in memory. If your process restarts, the index is gone unless you call faiss.write_index() before shutdown and faiss.read_index() on startup. For development this is fine, but in any persistent deployment replace FAISS with Chroma (local SQLite backend) or Qdrant (local mode with disk storage) to survive restarts without re-indexing.

Model size vs. RAM: plan before you pull

Ollama loads the full model into RAM (or VRAM). Llama 3 8B at 4-bit quantization needs ~5 GB; 70B needs ~40 GB. Running the embedding model simultaneously adds another 1–2 GB. On a 16 GB machine, stick to 7B–8B models and sentence-transformers for embeddings — don't try Llama 3 70B locally unless you have 48+ GB of memory or a capable GPU.

Ollama's API is OpenAI-compatible

Ollama exposes a REST API at http://localhost:11434 that is compatible with the OpenAI chat completions spec. You can swap base_url="http://localhost:11434/v1" and a dummy API key into any code that uses the OpenAI SDK — no other changes needed. This makes it trivial to switch between local and cloud LLMs with a single environment variable.

Local inference is slower; budget your chunk count accordingly

A local 8B model on CPU processes roughly 5–15 tokens per second versus 60–100 tokens/sec from hosted APIs. Retrieving 10 chunks and stuffing all of them into the context window produces very long prompts that take 30+ seconds to answer. On local setups, limit to 3–4 retrieved chunks max, and prefer shorter, denser chunks over long verbose ones.

Local RAG Setup FAQ

Can I run a RAG system entirely locally without any API keys?