Why Trace & Evaluate?
LLM applications have non-deterministic outputs. Without observability you cannot answer: which retrieval step failed? why did the agent loop? did the last model change improve or hurt quality? Tracing gives you visibility; evaluation gives you a score.
Debugging
See exact prompts, retrieved chunks, and tool outputs for every run.
Regression Testing
Run an eval suite before every deploy — catch quality drops automatically.
Continuous Improvement
Add failing production examples to your dataset to close the feedback loop.
LangSmith Tracing Setup
LangSmith is LangChain's built-in observability platform. Enable it with four environment variables — no code changes required.
# .env
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=ls__your_api_key
LANGCHAIN_PROJECT=my-rag-app # groups traces into a project
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com # default
# --- That's it. All LangChain calls are now traced automatically. ---
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
llm = ChatOpenAI(model="gpt-4o-mini")
chain = ChatPromptTemplate.from_template("Tell me about {topic}") | llm
# This call appears in LangSmith with full prompt/response/latency/tokens
result = chain.invoke({"topic": "LangGraph"})Every invoke creates a trace tree showing each node: the prompt template, the LLM call (with exact prompt text, model parameters, token counts), and the output parser.
Manual Tracing with @traceable
Use the @traceable decorator to wrap any Python function — including non-LangChain code — so it appears in the trace tree.
from langsmith import traceable
@traceable(name="fetch_user_profile", run_type="tool")
def fetch_user_profile(user_id: str) -> dict:
# This appears as a tool span inside the parent trace
return db.query("SELECT * FROM users WHERE id = ?", user_id)
@traceable(name="rag_pipeline", run_type="chain")
def answer_question(question: str, user_id: str) -> str:
profile = fetch_user_profile(user_id) # appears as child span
docs = retriever.invoke(question)
return rag_chain.invoke({"question": question, "docs": docs, "profile": profile})
result = answer_question("What is my subscription tier?", user_id="u-123")Building Evaluation Datasets
A dataset is a collection of (input, expected output) pairs. Create one in LangSmith UI or via the SDK.
from langsmith import Client
client = Client()
# Create a dataset
dataset = client.create_dataset(
dataset_name="rag-qa-v1",
description="Ground-truth Q&A pairs for RAG evaluation",
)
# Add examples
examples = [
{
"inputs": {"question": "What is LCEL?"},
"outputs": {"answer": "LCEL is LangChain Expression Language, a declarative way to compose chains."},
},
{
"inputs": {"question": "How do I add memory to a chain?"},
"outputs": {"answer": "Use RunnableWithMessageHistory and pass a session_id in the config."},
},
]
client.create_examples(dataset_id=dataset.id, examples=examples)
# Or: add examples directly from production traces
# client.create_examples_from_runs(run_ids=[...], dataset_id=dataset.id)Running Evaluations
Use evaluate() to run your application against the dataset and score outputs with one or more evaluators.
from langsmith.evaluation import evaluate, LangChainStringEvaluator
# Target: the function we want to evaluate
def run_rag(inputs: dict) -> dict:
answer = rag_chain.invoke(inputs["question"])
return {"answer": answer}
# Evaluator 1: exact match (fast, deterministic)
def exact_match(run, example) -> dict:
predicted = run.outputs["answer"].strip().lower()
expected = example.outputs["answer"].strip().lower()
return {"key": "exact_match", "score": int(predicted == expected)}
# Evaluator 2: LLM-as-judge (semantic correctness)
correctness_evaluator = LangChainStringEvaluator(
"labeled_criteria",
config={
"criteria": "correctness",
"llm": ChatOpenAI(model="gpt-4o", temperature=0),
},
prepare_data=lambda run, example: {
"prediction": run.outputs["answer"],
"reference": example.outputs["answer"],
"input": example.inputs["question"],
},
)
results = evaluate(
run_rag,
data="rag-qa-v1",
evaluators=[exact_match, correctness_evaluator],
experiment_prefix="gpt-4o-mini-baseline",
max_concurrency=4,
)RAGAS Metrics
RAGAS provides reference-free RAG metrics that don't need ground-truth answers — just the question, retrieved context, and generated answer.
| Metric | What It Measures | Needs Reference? |
|---|---|---|
| faithfulness | Answer supported by context (hallucination detection) | No |
| answer_relevancy | Answer addresses the question | No |
| context_precision | Retrieved context is useful | Yes |
| context_recall | All relevant info was retrieved | Yes |
| answer_correctness | Factual accuracy vs. ground truth | Yes |
from ragas import evaluate as ragas_evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
from datasets import Dataset
data = {
"question": ["What is LangGraph?"],
"answer": ["LangGraph is a library for building stateful multi-actor applications."],
"contexts": [["LangGraph is built on top of LangChain and enables cyclic graphs..."]],
"ground_truth": ["LangGraph enables building stateful, multi-actor LLM applications with cycles."],
}
score = ragas_evaluate(
Dataset.from_dict(data),
metrics=[faithfulness, answer_relevancy, context_recall],
)
print(score.to_pandas())CI/CD Integration
Run evaluations in GitHub Actions on every PR to catch regressions before merging.
# .github/workflows/eval.yml
name: LLM Evaluation
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install langsmith langchain-openai ragas
- name: Run evaluation
env:
OPENAI_API_KEY: $
LANGCHAIN_API_KEY: $
LANGCHAIN_TRACING_V2: "true"
run: python scripts/run_eval.py --fail-below 0.80
# Exit code 1 if average score < 0.80, blocking the mergeTip: Start with a small golden dataset of 20–50 examples. A focused dataset with clear pass/fail criteria is more actionable than 500 noisy examples. From here you can wire eval gates into your deployment pipeline.
LangChain Tracing and Evaluation FAQ
What is LangSmith?
LangSmith is LangChain's built-in observability and evaluation platform. It captures a trace tree for every run, showing each prompt, LLM call with token counts and latency, retrieved chunks, and tool outputs. It also lets you build datasets, run evaluations, and compare experiments, so you can debug and measure LLM applications instead of guessing.
How do I enable LangSmith tracing in LangChain?
Set four environment variables: LANGCHAIN_TRACING_V2=true, LANGCHAIN_API_KEY with your LangSmith key, LANGCHAIN_PROJECT to group traces, and optionally LANGCHAIN_ENDPOINT. No code changes are required; every LangChain invoke is then traced automatically. To trace plain Python functions, wrap them with the @traceable decorator so they appear as spans in the trace tree.
How do I evaluate a LangChain LLM application?
Build a dataset of input and expected-output pairs in LangSmith, then call evaluate() with your application as the target and one or more evaluators. Combine fast deterministic checks like exact match with LLM-as-judge evaluators that score semantic correctness, and use experiment_prefix to label and compare runs across model or prompt changes.
What metrics should I track when evaluating a RAG application?
Use RAGAS metrics: faithfulness checks the answer is supported by the retrieved context to catch hallucinations, answer_relevancy checks the answer addresses the question, context_precision and context_recall measure retrieval quality, and answer_correctness compares against ground truth. Faithfulness and answer_relevancy are reference-free, so you can run them without labeled answers.
How do I debug a LangChain chain that returns bad answers?
Open the LangSmith trace for the failing run and walk the trace tree from the top. Inspect the exact prompt text sent to the model, the retrieved chunks, tool inputs and outputs, and the raw model response at each node. This shows whether the problem is bad retrieval, a weak prompt, or the model itself, so you can fix the right step instead of rewriting the whole chain.
How do I run LangChain evaluations in CI/CD?
Run your evaluation script in GitHub Actions on every pull request. Install langsmith and your dependencies, pass OPENAI_API_KEY and LANGCHAIN_API_KEY as secrets, and have the script exit with a non-zero code when the average score falls below a threshold such as 0.80, which blocks the merge. Start with a focused golden dataset of 20 to 50 examples for actionable signal.