Evaluating LLM Chatbots and RAG Pipelines: A 2026 Crash Course

Introduction

Shipping a chatbot or a RAG application is the easy part now. The hard part — the part most teams skip and regret later — is answering one question with numbers instead of vibes: is this actually any good? Evaluation is what separates a demo that wins a Slack screenshot from a system that survives real users, model swaps, and prompt tweaks across a quarter.

This crash course walks through a working evaluation setup for both LLM chatbots and RAG pipelines, using LangSmith as the observability layer and LLM-as-a-judge as the grading mechanism. You’ll see how to build a golden dataset, define custom evaluators, score correctness, groundedness, answer relevance, and retrieval relevance, and compare multiple models against the same test set — the same loop most teams adopt once they outgrow eyeballing outputs.

Table of contents

Why LLM evaluation matters more than benchmarks

Public benchmarks tell you how a model behaves on the average internet question. They tell you almost nothing about how it behaves on your data, with your prompt, in your product. A model that tops MMLU can still hallucinate a refund policy or misread a Hinglish complaint. That gap is what application evaluation closes.

The shorthand: an eval suite is a unit-test suite for non-deterministic systems. You can’t assert equality on an LLM output, so you measure properties of it instead — correctness, conciseness, groundedness — and watch those numbers across runs.

The four pillars of LLM evaluation

Most production teams blend four evaluation styles. None of them is enough on its own; together they form a layered safety net.

Chatbot evaluation, step by step

A chatbot evaluation pipeline has four moving parts: a dataset of question-answer pairs, the chatbot itself, one or more evaluator functions, and a runner that ties them together. LangSmith handles the dataset storage and experiment tracking; the rest is plain Python.

Get the environment ready first. The two libraries that do the heavy lifting are langsmith and openai. Tracing is enabled by an environment variable.

import os
from dotenv import load_dotenv

load_dotenv()

os.environ["LANGSMITH_API_KEY"] = os.getenv("LANGSMITH_API_KEY")
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
os.environ["LANGSMITH_TRACING"] = "true"

Building a golden dataset in LangSmith

The dataset is the contract. Every example is a tuple of input (what the user types) and reference output (what a correct answer looks like). Quality beats quantity here — fifty well-chosen examples that cover edge cases will teach you more than five hundred lazy ones.

from langsmith import Client

client = Client()

dataset_name = "chatbot_evaluation"
dataset = client.create_dataset(dataset_name=dataset_name)

examples = [
    {"inputs": {"question": "What is LangChain?"},
     "outputs": {"answer": "A framework for building LLM applications."}},
    {"inputs": {"question": "What is LangSmith?"},
     "outputs": {"answer": "A platform for observing, testing, and evaluating LLM apps."}},
    {"inputs": {"question": "What is RAG?"},
     "outputs": {"answer": "Retrieval-augmented generation — LLMs grounded on retrieved context."}},
]

client.create_examples(dataset_id=dataset.id, examples=examples)

Once the dataset lands in LangSmith, it shows up under Datasets & Experiments. Every experiment you run later attaches to this dataset, which is how you compare runs side by side over time. Plenty of teams seed datasets from a CSV that annotators maintain in a spreadsheet — same shape, just a different ingestion path.

LLM-as-a-judge: writing a correctness evaluator

An evaluator is just a Python function that takes inputs, outputs, and reference_outputs and returns a boolean or a score. To keep judge calls traceable in LangSmith, wrap the OpenAI client with wrappers.wrap_openai.

import openai
from langsmith import wrappers

openai_client = wrappers.wrap_openai(openai.OpenAI())

EVAL_INSTRUCTIONS = (
    "You are an expert professor grading a student's answer to a question. "
    "Respond with 'correct' or 'incorrect' only."
)

def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    user_content = (
        f"Question: {inputs['question']}\n"
        f"Reference answer: {reference_outputs['answer']}\n"
        f"Student answer: {outputs['answer']}\n"
        "Respond with 'correct' or 'incorrect'."
    )
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": EVAL_INSTRUCTIONS},
            {"role": "user", "content": user_content},
        ],
    )
    return response.choices[0].message.content.strip().lower() == "correct"

A judge prompt is its own piece of engineering. Two patterns that pay off:

• Constrain the output to a short label or a structured JSON schema so parsing is reliable.
• Spell out the criteria — what counts as “correct,” what factual deviations are tolerated, when more detail than the reference is fine.

A useful companion to correctness is conciseness — a deterministic check that the model didn’t wander off into a five-paragraph answer to a one-line question:

def conciseness(outputs: dict, reference_outputs: dict) -> bool:
    return len(outputs["answer"]) < 2 * len(reference_outputs["answer"])

Running evaluations and comparing models

With a dataset and a couple of evaluators in hand, the runner pulls it together. Each example is fed to the chatbot, the output is scored, and LangSmith aggregates the run as one experiment.

DEFAULT_INSTRUCTIONS = (
    "Respond to the user question in a short, concise manner. "
    "One short sentence."
)

def my_app(question: str, model: str = "gpt-4o-mini",
           instructions: str = DEFAULT_INSTRUCTIONS) -> str:
    response = openai_client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": instructions},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

def ls_target(inputs: dict) -> dict:
    return {"answer": my_app(inputs["question"])}

experiment_results = client.evaluate(
    ls_target,
    data=dataset_name,
    evaluators=[correctness, conciseness],
    experiment_prefix="gpt-4o-mini-chatbot",
)

Open the experiment in LangSmith and you get a row-by-row breakdown: input, reference output, model output, and each evaluator’s score. Aggregate metrics sit at the top — correctness 0.60, conciseness 0.40, and so on. Swap the model and re-run, and the comparison appears side by side.

def ls_target_turbo(inputs: dict) -> dict:
    return {"answer": my_app(inputs["question"], model="gpt-4-turbo")}

client.evaluate(
    ls_target_turbo,
    data=dataset_name,
    evaluators=[correctness, conciseness],
    experiment_prefix="gpt-4-turbo-chatbot",
)

In the original walkthrough, GPT-4o mini ended up beating GPT-4 turbo on this particular dataset — a useful reminder that “bigger” doesn’t automatically mean “better for your task.” Picking a model without an eval suite is picking with a coin flip.

RAG evaluation: the four core metrics

RAG adds a retriever to the pipeline, and the retriever is its own source of failure. The same dataset-plus-judge pattern works, but the metrics expand to cover what the retriever returned and what the generator did with it.

These four cover most production needs. RAGAS and similar frameworks add finer-grained metrics like context recall (did retrieval pull in everything it should have?) and noise sensitivity, which are worth adding once the basics are in place.

End-to-end RAG evaluation pipeline

The pipeline mirrors the chatbot version with two additions: a real retriever, and evaluators that look at retrieved documents alongside the answer. Start by building the RAG itself.

from langchain_community.document_loaders import WebBaseLoader
from langchain_community.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

urls = [
    "https://example.com/blog/agents",
    "https://example.com/blog/prompt-engineering",
    "https://example.com/blog/adversarial-attacks",
]

docs = [d for url in urls for d in WebBaseLoader(url).load()]
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

vector_store = InMemoryVectorStore.from_documents(chunks, OpenAIEmbeddings())
retriever = vector_store.as_retriever()

Wrap generation in a @traceable function so each invocation lands in LangSmith’s trace view:

from langchain.chat_models import init_chat_model
from langsmith import traceable

llm = init_chat_model("openai:gpt-4o-mini")

@traceable
def rag_bot(question: str) -> dict:
    docs = retriever.invoke(question)
    context = "\n\n".join(d.page_content for d in docs)
    prompt = (
        "You are a helpful assistant. Use the context below to answer the question. "
        "Keep the answer to three sentences max.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    ai_message = llm.invoke([
        {"role": "system", "content": prompt},
        {"role": "user", "content": question},
    ])
    return {"answer": ai_message.content, "documents": docs}

Build a small RAG-specific dataset — questions whose answers come from the indexed documents — and upload it the same way as before. Then define four evaluators, one per metric, each using a structured-output LLM judge.

from typing import Annotated, TypedDict
from langchain_openai import ChatOpenAI

class CorrectnessGrade(TypedDict):
    explanation: Annotated[str, "Reasoning behind the grade."]
    correct: Annotated[bool, "True if the answer is correct, else False."]

CORRECTNESS_PROMPT = """You are a teacher grading a quiz.
You will be given a QUESTION, a GROUND TRUTH answer, and a STUDENT answer.

Grade the student answer on factual accuracy relative to the ground truth.
It is fine for the student answer to contain extra information,
as long as it does not conflict with the ground truth.

Return correct=true only if all factual claims align.
Explain your reasoning step by step."""

grader_llm = ChatOpenAI(model="gpt-4o-mini", temperature=0).with_structured_output(
    CorrectnessGrade, method="json_schema", strict=True
)

def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    user = (
        f"QUESTION: {inputs['question']}\n"
        f"GROUND TRUTH: {reference_outputs['answer']}\n"
        f"STUDENT: {outputs['answer']}"
    )
    grade = grader_llm.invoke([
        {"role": "system", "content": CORRECTNESS_PROMPT},
        {"role": "user", "content": user},
    ])
    return grade["correct"]

The other three evaluators follow the same pattern — a typed schema, a prompt that spells out the criteria, a judge LLM with structured output. The differences are just which fields are passed in:

• Answer relevance — takes question and answer; no reference needed.
• Groundedness — takes retrieved documents and answer; checks for unsupported claims.
• Retrieval relevance — takes question and retrieved documents; flags off-topic chunks.

Wire it all into a single evaluation run:

def target(inputs: dict) -> dict:
    return rag_bot(inputs["question"])

results = client.evaluate(
    target,
    data="rag_test_evaluation",
    evaluators=[correctness, answer_relevance, groundedness, retrieval_relevance],
    experiment_prefix="rag-gpt-4o-mini",
)

results.to_pandas()

LangSmith renders the run with a per-example breakdown of all four scores plus latency, token cost, and the full trace tree. That’s your scoreboard. Tune the retriever, swap the embedding model, edit the prompt — rerun, watch the numbers move.

Tools beyond LangSmith

LangSmith is the path of least resistance if you’re already in the LangChain ecosystem, but it’s not the only option. Pick based on where your stack lives and what you need to measure.

Eval-driven development as a workflow

The point of an eval suite isn’t the dashboard — it’s the feedback loop it unlocks. Once metrics are wired up, the iteration cycle changes shape: every prompt edit, model swap, or retriever tweak becomes a measurable experiment instead of a hunch.

Treat eval results like a regression test report. A merge that drops correctness from 0.85 to 0.72 is the AI-app equivalent of breaking a hundred unit tests — it shouldn’t reach main without a conversation.

Production observability: keeping evals honest after launch

Offline evals tell you how the system performs on a known dataset. Production observability tells you how it behaves on the dataset you didn’t design — real users. A mature pipeline runs both.

• Online sampling. Pick a small percentage of live traffic each day, run the same evaluators against it, and chart the metrics over time. Catches drift the offline suite can’t see.
• User feedback signals. Thumbs-up, thumbs-down, post-conversation surveys. Treat them as another evaluator output and correlate them with your LLM-judge scores.
• Failure mining. Cluster low-scoring or downvoted interactions, pull the hardest ones into the golden dataset, and re-run offline evals. That’s how the suite evolves.
• Cost and latency tracking. Quality isn’t the only metric. A model that’s 2 percent more accurate but 4× slower or 10× more expensive is rarely the right ship.

Best practices for shipping evaluated systems

Common mistakes to avoid

• Confusing “the model is good” with “the system is good.” A strong base model with a vague prompt and a bad retriever scores worse than a weaker model with a tight setup. Evaluate the whole pipeline, not just the LLM.
• Picking metrics that are easy to score, not metrics that matter. BLEU and ROUGE are tempting because they’re deterministic, but they correlate poorly with user-perceived quality. Spend the effort on judge-based metrics that align with what users actually care about.
• Tiny datasets that never grow. An eval suite that hasn’t added an example in three months is a suite that’s no longer testing anything new. Treat every customer-reported failure as a dataset PR.
• No baseline. “Correctness is 0.78” means nothing without a previous number to compare against. Always run the new experiment alongside the old one.
• Skipping retrieval-only evals. If retrieval relevance is 0.4, no generator on Earth will produce a grounded answer. Measure each stage independently before blaming the LLM.

Conclusion

LLM applications don’t fail loudly. They fail quietly — a slightly wrong refund policy here, a hallucinated source there, a model version that changed under your feet. The way out is the same way every other software discipline solved this problem: measurable, repeatable, automated checks.

Build a small dataset, write a judge, define a few metrics that map to what your product actually needs, and run the suite on every change. For RAG, add the four pillars — correctness, answer relevance, groundedness, retrieval relevance — and watch each stage of the pipeline independently. Tools like LangSmith, RAGAS, DeepEval, and TruLens make the plumbing cheap; the real work is curating data and writing judge prompts that match what humans would say. Do that, and “is this any good?” stops being a vibe check and starts being a number you can move on purpose.