Hyperparameter Tuning
Parameters to Tune:
Chunk Size (200-800 tokens)
Smaller = more retrievals needed, faster, noisier. Larger = context-heavy, slower, better semantics.
Retrieval K (3-20 documents)
More K = higher recall but costs more, context overflow risk. Less K = faster, lower cost, miss answers.
Embedding Model (768D vs 1536D)
Larger = better quality but 2x storage, slower. Smaller = fast but quality drops.
Reranker Threshold (0.0-1.0)
Higher = only return highly relevant docs. Lower = more retrieval but noise.
def find_optimal_chunk_size(eval_dataset):
"""Grid search for chunk size."""
results = {}
for chunk_size in [200, 300, 400, 500, 600]:
rag = RAGSystem(chunk_size=chunk_size)
metrics = rag.evaluate(eval_dataset)
results[chunk_size] = metrics
best = max(results.items(), key=lambda x: x[1]['f1_score'])
return best[0] # optimal sizeLatency Optimization
Caching (Biggest impact)
Cache query embeddings and LLM responses. If same question asked twice, return cached answer immediately.
Async Retrieval
Retrieve docs while streaming LLM response. Don't wait for retrieval to start generation.
Batch Embeddings
Never embed one doc at a time. Batch 100+ for 10x speedup.
Vector DB Selection
HNSW (Qdrant) is faster than IVF (Milvus). Choose based on scale.
Typical latency breakdown: Retrieval 50ms + LLM 1000ms + overhead 50ms = 1.1s. Focus on LLM speed (streaming, smaller model).
Cost Optimization
Cost breakdown per query:
- Embedding (query): $0.0002 (text-embedding-3-small)
- Vector DB search: $0.0001 (Pinecone)
- LLM (GPT-4): $0.03 (worst case, 2K output)
Total per query: ~$0.032
For 1M queries/month: $32,000
Cost reduction strategies:
1. Reduce K from 10 to 5 → -50% vector DB cost
2. Use smaller LLM (GPT-3.5) → -75% inference cost
3. Batch embeddings → -30% API cost
4. Cache results → -60% repeat queries cost
5. Local embeddings → $0 embedding costA/B Testing Changes
def ab_test_retrieval(control_rag, treatment_rag, test_queries):
"""Compare two RAG systems."""
results = {
"control": {"quality": 0, "latency": 0, "cost": 0},
"treatment": {"quality": 0, "latency": 0, "cost": 0}
}
for i, query in enumerate(test_queries):
if i % 2 == 0:
# Control group
start = time.time()
answer, docs = control_rag.query(query)
results["control"]["latency"] += time.time() - start
results["control"]["quality"] += evaluate_answer(answer)
else:
# Treatment group
start = time.time()
answer, docs = treatment_rag.query(query)
results["treatment"]["latency"] += time.time() - start
results["treatment"]["quality"] += evaluate_answer(answer)
# Analyze results
print(f"Control Quality: {results['control']['quality']/len(test_queries)}")
print(f"Treatment Quality: {results['treatment']['quality']/len(test_queries)}")
# Statistically significant?
if results["treatment"]["quality"] > results["control"]["quality"] * 1.05:
print("Treatment is significantly better!")Notes
Change one variable at a time
It's tempting to switch chunk size, embedding model, and retrieval algorithm simultaneously. Resist — you'll have no idea which change moved the metric. Run a proper A/B test: hold your evaluation set fixed, change exactly one parameter, measure RAGAS scores before and after, then decide. Multi-variable changes can cancel each other out or create misleading improvements that don't hold on new queries.
Reranking often gives more ROI than larger embeddings
Switching from a 768-dim to a 1536-dim embedding model adds cost and latency for marginal retrieval gains. Adding a cross-encoder reranker (Cohere Rerank, BGE Reranker, or FlashRank locally) to re-score the top-20 retrieved chunks and keep only top-3 typically yields larger precision improvements at a fraction of the cost. Try reranking before upgrading embedding models.
Chunk overlap trades storage for coherence
A 10–15% overlap between adjacent chunks ensures that sentences split across chunk boundaries aren't lost during retrieval. However, overlap also inflates your vector index size — 15% overlap on 1M chunks adds ~150k extra vectors. For large corpora, use smaller overlap (50 tokens) and rely on sentence-aware splitters rather than fixed character counts to preserve natural sentence boundaries.
Benchmark on queries your users actually ask
Synthetic evaluation sets generated by GPT-4 tend to be cleaner and more direct than real user queries. Users ask ambiguously, use abbreviations, and make spelling mistakes. Log a sample of real production queries from day one and build your golden dataset from those — optimization against synthetic queries can silently make real-user performance worse.
RAG Optimization and Tuning FAQ
What is the biggest factor in RAG quality?
Retrieval quality — specifically whether the correct chunks are in the top-K results — has the largest impact on answer quality. Improving chunk size, embedding model, and hybrid search typically yields bigger gains than prompting the LLM differently.
How do I tune chunk size for my RAG system?
Run offline experiments: index your documents at 256, 512, and 1024 tokens, then evaluate each with RAGAS on your golden test set. Pick the chunk size with the best context recall and precision score for your specific content type.
How do I A/B test RAG configurations?
Route a fraction of production queries through a variant pipeline (different chunk size, embedding model, or retrieval strategy). Compare RAGAS metrics and user feedback rates over 1–2 weeks before promoting the winner.
Does adding a reranker always improve RAG quality?
In most benchmarks, adding a cross-encoder reranker improves answer quality by 5–15% at the cost of 50–200ms added latency. The improvement is largest when the initial retriever has low precision.
How can I optimize RAG for faster responses?
Cache embeddings for frequently asked queries, reduce top-K from 10 to 5 if precision is acceptable, use a smaller embedding model, and enable LLM streaming so users see the first tokens immediately while generation continues.