Why Evaluation is Critical
You can build a RAG system in hours. But how do you know it's actually good? Without evaluation:
❌ You ship hallucinations
Your LLM confidently invents facts from incomplete retrieval. Users discover the bugs.
❌ You tune blindly
You change chunk size, embedding model, or parameters without knowing if they help or hurt.
❌ You miss regressions
You update documents or embeddings, and quality silently drops until a user complains.
✓ Good evaluation means: You measure quality objectively, catch regressions early, and know exactly which changes improve results.
Building Ground Truth Datasets
Evaluation requires labeled data: queries paired with correct documents and answers.
Dataset Structure:
{
"id": "test_001",
"query": "How do I file taxes if self-employed?",
"relevant_doc_ids": ["tax_guide_p3", "deduction_rules_p1"],
"ground_truth_answer": "Self-employed people must...",
"difficulty": "medium"
}How to Build Ground Truth:
1. Manual Annotation
Subject matter experts label queries with correct documents and answers. Gold standard but slow ($10-50 per example).
2. Synthetic Generation
Generate test queries from your documents using an LLM.
from llama_index.core.evaluation import generate_question_context_pairs
from llama_index.llms import OpenAI
# From your indexed documents
dataset = generate_question_context_pairs(
documents,
llm=OpenAI(model="gpt-4"),
num_questions_per_context=2
)3. Hybrid Approach
Generate synthetically, then have humans validate/edit a sample. Faster and cheaper than pure manual.
📊 Starting point: 50-100 test examples for initial evaluation. 500+ for production monitoring.
Retrieval Evaluation Metrics
How many relevant documents did you retrieve? How highly did you rank them?
Recall@K
Of all relevant documents, how many did you retrieve in the top-K?
Recall@5 = (# of relevant docs in top-5) / (total # of relevant docs)
Example:
Query: "How do I file taxes?"
Relevant docs: [doc_1, doc_2, doc_3, doc_4, doc_5, doc_6, doc_7] (7 total)
Retrieved top-5: [doc_2, doc_4, random_doc, doc_1, another_random]
Recall@5 = 3/7 = 0.43 (missed 4 relevant docs)Higher is better. Typical threshold: Recall@5 ≥ 0.8 for good systems.
Mean Reciprocal Rank (MRR)
How high in the ranking is the first relevant document?
MRR = average(1/rank of first relevant doc)
Example:
Query 1: First relevant doc at rank 1 → 1/1 = 1.0
Query 2: First relevant doc at rank 3 → 1/3 = 0.33
Query 3: First relevant doc at rank 1 → 1/1 = 1.0
MRR = (1.0 + 0.33 + 1.0) / 3 = 0.78Range: 0-1. Higher is better. Penalizes wrong answers at top.
nDCG (Normalized Discounted Cumulative Gain)
Combines position and relevance. More nuanced than recall/precision.
nDCG rewards highly-relevant docs at top, less-relevant docs lower.
Range: 0-1. nDCG@10 typical threshold: ≥ 0.7 for good systems.Precision@K
Of the top-K retrieved, how many are actually relevant?
Precision@5 = (# relevant in top-5) / 5
Retrieved top-5: [doc_2✓, doc_4✓, doc_random, doc_1✓, doc_other]
Precision@5 = 3/5 = 0.6 (60% of what you returned is good)🎯 Quick recommendation: Monitor Recall@5 and MRR. Start with target Recall@5 ≥ 0.8 and MRR ≥ 0.6.
Answer Quality Evaluation
Even if retrieval is perfect, the LLM might still hallucinate. Measure answer quality directly.
Faithfulness
Does the answer use only retrieved documents, or does it invent facts?
Query: "How do I file taxes if self-employed?"
Retrieved: "Self-employed people must file quarterly estimated taxes..."
Answer: "Self-employed people must file quarterly estimated taxes AND must
use Form 1234-X which was introduced in 2023."
↑ "Form 1234-X" not in retrieved docs → hallucination!Tools like RAGAS measure this automatically.
Relevance
Does the answer actually address the question?
Query: "How do I deduct medical expenses?"
Answer: "Medical expenses are growing..." ← Off-topic, doesn't explain deduction process.
Semantic Similarity
How similar is the generated answer to the ground truth?
Compare embeddings of answer vs ground truth. Score: 0-1.
Evaluation Tools
RAGAS (Retrieval-Augmented Generation Assessment)
Purpose: Measure retrieval and answer quality with LLM-based judges. No manual annotations needed.
pip install ragas
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_recall,
context_precision
)
result = evaluate(
dataset=test_dataset, # [query, ground_truth_answer, context, answer]
metrics=[faithfulness, answer_relevancy, context_recall, context_precision]
)
print(result) # Shows scores for each metric✓ Best for quick evaluation. ⚠️ Depends on LLM quality (uses GPT-4 by default).
TruLens
Purpose: Real-time monitoring of RAG systems in production. Track feedback, quality metrics, costs.
pip install trulens-eval
from trulens_eval import TruChain, Feedback, Tru
# Wrap your chain
chain = TruChain(your_rag_chain, app_id="my-rag")
# Define quality feedback
from trulens_eval.feedback import Groundedness
groundedness = Groundedness(...)
# Run and track
response = chain.invoke(query)
# View dashboard at http://localhost:8501✓ Best for production. Tracks metrics over time. Catches regressions.
DeepEval
Purpose: Framework for defining custom LLM evaluation metrics. Great for domain-specific quality.
pip install deepeval
from deepeval import evaluate
from deepeval.metrics import Faithfulness, AnswerRelevancy
test_cases = [
{"query": "...", "answer": "...", "context": "..."}
]
result = evaluate(test_cases, [Faithfulness(), AnswerRelevancy()])
print(f"Pass rate: {result.pass_rate}")✓ Most flexible. Works with any LLM. Great for custom metrics.
Human Evaluation Workflows
Automated metrics have blindspots. For critical systems, involve humans.
Workflow:
1. Run batch evaluation (RAGAS, automated metrics)
2. Flag low-scoring examples
3. Send to humans for review (Label Studio, Argilla)
4. Aggregate ratings (multiple reviewers per example)
5. Update test dataset with corrections
6. Re-evaluate to establish new baseline
Human Rating Scale (example):
☑ Excellent - Answer is accurate, complete, well-sourced
☑ Good - Answer is mostly correct but missing nuance
☑ Fair - Answer partially correct but has gaps
☐ Poor - Answer is inaccurate or hallucinatedRetrieval metrics: Tracking Recall@5, MRR, nDCG@10
Answer quality: Measuring faithfulness, relevance via RAGAS or TruLens
Regression testing: Re-running evaluation after every change
Production monitoring: Tracking real user queries and feedback
Notes
RAGAS evaluation costs real money
Each RAGAS metric invokes an LLM judge (GPT-4 by default) to score individual samples. On a 100-question evaluation set, expect 400–600 LLM calls per full suite run. Use gpt-4o-mini as the judge for development cycles; switch to gpt-4o only for final benchmarks. Running evaluation on every commit will drain your API budget fast.
Don't optimize a single metric in isolation
Faithfulness and answer relevancy trade off against each other. A system that retrieves many chunks and quotes them verbatim scores high on faithfulness but low on relevancy (the answer is too long and literal). Tune your retrieval k, chunk size, and prompt template together, and check all four RAGAS metrics before declaring an improvement.
Human evaluation catches what automated metrics miss
RAGAS can't reliably detect tone errors, unnecessary hedging, or answers that are technically correct but practically unhelpful. Reserve 5–10% of your evaluation budget for a human review of the trickiest cases — adversarial queries, multi-hop reasoning, and domain-specific jargon that the judge LLM may not handle well.
Your evaluation set is only as good as its coverage
A test set built from easy, obvious questions will make your system look better than it is. Intentionally include: questions that span multiple documents, questions with no good answer in the corpus, and questions that match documents superficially but require different reasoning. Cover these in your golden dataset before going to production.
RAG Evaluation Framework FAQ
What is RAGAS and how does it evaluate RAG systems?
RAGAS is an open-source framework that evaluates RAG pipelines on four metrics: faithfulness (does the answer stick to retrieved context?), answer relevancy, context precision, and context recall. It uses LLMs as judges.
What is faithfulness in RAG evaluation?
Faithfulness measures whether every claim in the generated answer is supported by the retrieved context. A faithfulness score of 1.0 means no hallucination; 0.0 means the answer is entirely fabricated.
What is context precision in RAG evaluation?
Context precision measures what fraction of the retrieved chunks are actually relevant to the question. High context precision means your retriever returns mostly useful chunks; low precision means lots of noise in the LLM prompt.
How many test cases do I need to evaluate a RAG system?
A minimum of 50 question-answer pairs is recommended for statistical significance. 200+ gives reliable estimates across edge cases. Use diverse queries that cover simple factual lookups, multi-hop questions, and out-of-scope queries.
What is the difference between retrieval quality and answer quality in RAG?
Retrieval quality measures whether the right chunks were found (context recall and precision). Answer quality measures whether the LLM used them correctly (faithfulness and answer relevancy). You need both to diagnose problems.