Why Benchmarking RAG Systems
Published benchmarks for vector databases and embedding models give you a useful shortlist, but they rarely predict production performance for your specific workload. Dataset distribution, query length, index parameters, hardware, and network topology all interact in ways that make "Qdrant is 3× faster than Pinecone" statements true in one setup and false in another.
The right approach is to use external benchmarks to narrow your options to two or three candidates, then evaluate on a representative sample of your own queries and documents. BEIR (Benchmarking Information Retrieval) is the closest thing to a neutral standard — it evaluates retrieval across 18 diverse tasks — but BEIR results diverge substantially from domain-specific corpora like legal contracts, medical notes, or source code. NDCG@10 and Recall@100 are the BEIR metrics most commonly reported; for RAG specifically, Recall@5 is more predictive of answer quality because the LLM only sees the top few retrieved chunks.
Measure each pipeline stage independently. In early prototypes, retrieval latency typically dominates. In production under real query volume, LLM generation almost always dominates. Understanding this shift tells you where to focus optimization: retrieval tuning (indexing algorithms, chunk size, reranking) vs. LLM cost reduction (smaller models, prompt compression, caching). The data in this lesson provides a starting point; always validate on your stack before making infrastructure commitments.
Vector DB Benchmarks
| DB | Query Latency (1M vecs) | Memory | Cost/month |
|---|---|---|---|
| Pinecone | 45-60ms | N/A (managed) | $100-500 |
| Qdrant | 10-20ms | ~8GB | $0 (OSS) |
| Milvus | 5-15ms | ~6GB | $0 (OSS) |
| FAISS | <1ms | ~10GB | $0 |
Test on YOUR data: Speed varies by dataset, indexing algorithm, and hardware.
Embedding Model Comparison
| Model | Dims | Cost | Quality | Speed (CPU) |
|---|---|---|---|---|
| OpenAI 3-small | 1536 | $0.02/1M | ⭐⭐⭐⭐⭐ | API |
| all-mpnet-v2 | 768 | Free | ⭐⭐⭐⭐ | ~500 docs/s |
| bge-large | 1024 | Free | ⭐⭐⭐⭐⭐ | ~200 docs/s |
Recommendation: Start with OpenAI for quality. Switch to local if privacy/cost critical.
Chunk Size Impact
Benchmark: 10K test queries on tax docs
Chunk Size: 200 300 400 500 600
Recall@5: 0.73 0.79 0.82 0.81 0.78
MRR: 0.68 0.75 0.78 0.77 0.74
Latency: 45ms 48ms 52ms 55ms 60ms
Cost/1M: $180 $150 $130 $125 $120
🎯 Winner: 400-500 tokens (best quality/speed/cost balance)Full System Benchmarks
Setup A (Cost-optimized):
GPT-3.5 + Chroma + local embeddings
Latency: 1.2s | Quality: 0.75 | Cost: $5K/month
Setup B (Balanced):
GPT-4 + Pinecone + OpenAI embeddings + reranker
Latency: 2.1s | Quality: 0.88 | Cost: $15K/month
Setup C (High-quality):
Claude 3 + Qdrant + fine-tuned embeddings + agentic RAG
Latency: 3.5s | Quality: 0.96 | Cost: $30K/month
Notes
P99 latency matters more than median
Median (P50) latency looks good in most benchmarks. Users perceive slowness at the tail — P95 and P99. A vector DB with 15 ms median but 800 ms P99 spikes feels sluggish in practice. Always measure and report P95/P99 when comparing candidates, especially if your traffic has burst patterns or cold-start behavior after idle periods.
HNSW parameters require per-dataset tuning
For HNSW indexes (used by Qdrant, FAISS, and most modern vector DBs), the ef_construction and M parameters control the recall vs. speed vs. memory trade-off. Higher M improves recall but increases memory by roughly 8 bytes per edge per vector. Library defaults are intentionally conservative; tuning M and ef_search on your actual dataset can yield 30–50% throughput improvements with no quality loss.
Embedding model pricing changes frequently
OpenAI has cut embedding prices multiple times. The per-token costs in the table above may be outdated by the time you read this. Always check the current pricing page before building a cost model. More importantly, quality rankings between models change with each new release — re-evaluate open-source alternatives (BGE, E5, Nomic Embed) before assuming OpenAI is still the best quality-per-dollar option for your domain.
Recall@5 predicts RAG answer quality better than NDCG@10
NDCG@10 grades a ranked list of 10 results. In RAG, the LLM typically sees only 3–5 chunks. Optimizing a retriever on NDCG@10 can produce systems where the most relevant document ranks at position 7 — technically a high NDCG score, but the LLM never sees it. Recall@5 (was at least one correct chunk in the top 5?) better predicts whether the LLM will have the information it needs to answer correctly.
RAG Benchmarking FAQ
Which vector database is fastest for RAG?
In benchmarks, Qdrant and Milvus consistently lead on query latency at scale. FAISS is fastest for in-memory single-node setups. Pinecone is competitive but adds network overhead as a hosted service.
How do I benchmark embedding models for RAG?
Use BEIR (Benchmarking Information Retrieval) to evaluate retrieval quality across domains. Measure NDCG@10 and recall@100 on a representative sample of your queries and documents.
What chunk size performs best in RAG benchmarks?
Most benchmarks find 256–512 tokens per chunk optimal for retrieval precision. Larger chunks (1024+) help when questions require broader context but hurt precision. Always tune on your own data.
How do I measure RAG latency end-to-end?
Instrument each stage separately: embedding query, vector search, prompt construction, and LLM generation. P50/P95/P99 latencies reveal where to optimize — retrieval is usually sub-100 ms; LLM generation dominates.
What is a realistic throughput for a RAG system?
A single-node Qdrant or FAISS instance typically handles 1,000–5,000 QPS for retrieval. Throughput bottlenecks almost always come from the LLM, not the vector store, for typical chat-style workloads.