Overview
RAG failures are hard to diagnose without structured observability because the problem can originate in any of five places: the query embedding (semantic drift from what was indexed), the retrieval step (wrong candidates surfaced), the reranker (correct candidates demoted), the prompt construction (context truncated or ordered poorly), or the LLM generation (hallucination despite good context). Each failure looks the same to the user — a wrong or unhelpful answer — but requires a different fix.
The correct debugging workflow is always top-down: first establish whether the correct document is in the index at all, then whether it was retrieved in the candidate set, then whether it ranked highly enough to appear in the LLM context, then whether the LLM ignored or misread it. Skipping steps in this process leads to fixing the wrong layer — adding a reranker when the actual problem is a missing document, or tuning the system prompt when the retriever never returns the right chunk.
Production observability tools like LangSmith, Langfuse, and Arize Phoenix surface each step of the pipeline with latency, token counts, and retrieved content. Integrating one of these before you have a problem in production is far easier than adding instrumentation during an incident. The debugging code in this lesson gives you the building blocks to roll your own observability if you prefer to avoid the dependency.
Diagnosing Bad Answers
❌ Problem: "I don't know"
Check: No relevant docs retrieved? Confidence low? Fix: Rerank, expand query, lower threshold.
❌ Problem: Hallucinated answer
Check: Retrieved docs don't support answer. Fix: Add prompt guardrails, use reranker, reduce temperature.
❌ Problem: Wrong answer
Check: Bad retrieval or bad LLM? Debug retrieval first with ground truth. Then check LLM prompt.
Debugging Retrieval
def debug_retrieval(query, expected_doc_ids):
"""Diagnose why docs aren't retrieved."""
# Get all results with scores
results = vector_db.search(query, k=100)
print(f"Query: {query}")
print(f"Expected docs: {expected_doc_ids}\n")
for rank, (doc_id, score) in enumerate(results, 1):
found = "✓" if doc_id in expected_doc_ids else " "
print(f"{found} Rank {rank}: {doc_id} (score: {score:.3f})")
# Analyze
retrieved_ids = [id for id, _ in results[:10]]
found = len(set(retrieved_ids) & set(expected_doc_ids))
print(f"\nFound {found}/{len(expected_doc_ids)} expected docs in top-10")
Debugging steps:
1. Check embedding quality (similar queries = similar embeddings?)
2. Check vector DB indexing (is index built?)
3. Check metadata filtering (are filters excluding docs?)
4. Use reranker to re-score (is ranking wrong?)
Token Usage Debugging
import tiktoken
def analyze_tokens(query, docs, answer):
enc = tiktoken.encoding_for_model("gpt-4")
tokens = {
"query": len(enc.encode(query)),
"system": len(enc.encode(SYSTEM_PROMPT)),
"docs": sum(len(enc.encode(doc.text)) for doc in docs),
"answer": len(enc.encode(answer))
}
total = sum(tokens.values())
print(f"Query: {tokens['query']} tokens")
print(f"System: {tokens['system']} tokens")
print(f"Docs: {tokens['docs']} tokens")
print(f"Answer: {tokens['answer']} tokens")
print(f"Total: {total} tokens (${total * 0.003 / 1000:.4f})")
# Check if exceeding limit
if total > 4096:
print("⚠️ EXCEEDING CONTEXT LIMIT!")Monitoring Dashboard
Key metrics to track:
- Query latency (p50, p95, p99)
- Retrieval quality (Recall@5, MRR)
- Cache hit rate
- Error rate & types
- Token usage per query
- Cost per query
- User satisfaction (from feedback)
Tools: Prometheus + Grafana, CloudWatch, DataDog, or custom dashboard
Grafana dashboard shows:
- Request rate (queries/sec)
- Latency histogram
- Error rate trend
- Cache performance
- Cost trajectoryNotes
Langfuse is the self-hostable alternative to LangSmith
LangSmith stores traces in LangChain's cloud, which is problematic for privacy-sensitive deployments involving PII or confidential documents. Langfuse is open-source, fully self-hostable via Docker, and provides equivalent trace visualization for retrieval steps, LLM calls, token usage, and latency. Arize Phoenix runs entirely locally with no backend required and is ideal for local debugging sessions.
Similarity score distributions reveal index health
Log the top-1 similarity score for every query. If 90% of queries return scores below 0.5, your embedding model is likely mismatched to your document domain — the vectors are too dissimilar for reliable retrieval. A healthy index typically shows median top-1 scores above 0.7 for in-domain queries. Consistently low scores are a signal to evaluate domain-specific embedding models or to inspect whether documents were correctly pre-processed before embedding.
Token usage logging doubles as an anomaly detector
Unexpected spikes in tokens per query are usually a symptom of a retrieval bug: top-K accidentally doubled, chunk size changed, or a new document type with very long sections started being indexed. Token count per query is cheap to log and easy to alert on. Set a P99 baseline and alert when it increases by more than 30% — this catches retrieval regressions before they appear in quality metrics.
Cold-start latency is 2–5× slower than steady-state
The first embedding API call after an idle period (connection pool expiry, cold Lambda, sleeping container) is significantly slower than subsequent calls. P99 latency measurements that include cold-start events will be misleadingly high if traffic is bursty. Measure P99 separately for cold and warm states, and use connection pre-warming (a lightweight health-check ping before serving real traffic) to eliminate cold-start latency for user-facing deployments.
Debugging RAG Systems FAQ
How do I debug a RAG system that gives wrong answers?
Add logging at each stage: log the query embedding, retrieved chunk IDs and scores, the exact prompt sent to the LLM, and the LLM response. Compare what was retrieved vs what the correct answer requires.
What metrics should I monitor in a production RAG system?
Monitor: retrieval latency (P50/P95), LLM latency, answer faithfulness (RAGAS), user thumbs-up/down rate, empty retrieval rate, and token usage per query. Alert on spikes in latency or drop in faithfulness.
How do I trace RAG hallucinations back to their source?
Check whether the correct answer is present in any retrieved chunk. If yes, the LLM ignored the context (tune the system prompt). If no, the retrieval failed (tune embeddings, chunk size, or top-K).
Why does my RAG system sometimes return "I don't know" incorrectly?
The retriever failed to surface relevant chunks. Check similarity scores — if all scores are below your threshold, lower it or switch to hybrid search. Also verify the document containing the answer was actually ingested.
What tools can I use to monitor a RAG pipeline?
LangSmith (LangChain's tracing platform), Langfuse (open-source), and Arize Phoenix are purpose-built for RAG observability. They trace each retrieval and generation step with latency and token counts.