DS DevShelfHub Projects · AI tools
Tutorials / Prompt Engineering / Prompt Testing & Engineering
Prompt Engineering Intermediate · 13 min read Page 10 of 10

Prompt Testing & Engineering: Test Prompts Like Code

By DevShelfHub

How to treat prompts like code — test cases, golden datasets, regression suites, and metrics for accuracy, consistency, latency, and cost.

Series progress10 / 10
Prompt testing and engineering tutorial — QA for LLM prompts

Treat prompts like code

Prompts drift. A change that improves one case breaks another. Without a test suite, you won't know until production. The engineering mindset: every prompt change is a code change — it needs tests before it ships.

Without testing

  • You "vibe check" a prompt with 2–3 examples
  • It looks good, you ship it
  • Edge cases fail silently in production
  • You can't safely improve the prompt later

With testing

  • Changes run against a golden dataset
  • Regressions are caught before shipping
  • You can improve the prompt confidently
  • Quality is measurable, not subjective

Writing prompt test cases

A test case has three parts: an input, the expected output (or criteria), and a pass/fail assertion.

python 
test_cases = [
    {
        "input":    "The product is amazing, highly recommend!",
        "expected": "Positive",
        "type":     "exact_match",
    },
    {
        "input":    "It works but nothing special.",
        "expected": "Neutral",
        "type":     "exact_match",
    },
    {
        "input":    "Terrible quality, broke on day one.",
        "expected": "Negative",
        "type":     "exact_match",
    },
    {
        "input":    "The packaging was fine but the product was disappointing.",
        "expected": "Negative",
        "type":     "exact_match",
        "note":     "edge case — mixed but negative overall",
    },
]

Cover: happy path, edge cases, ambiguous inputs, known failure modes, and adversarial inputs. Aim for 20–50 cases per task — enough to catch regressions, small enough to run cheaply.

Golden datasets

A golden dataset is a fixed set of (input, expected output) pairs that represents the full distribution of your task. It's your ground truth.

How to build one

  • Start with 20–50 real examples from your production data
  • Label the expected outputs manually (human ground truth beats LLM-labelled)
  • Include equal representation of all output classes / edge cases
  • Version it in git — it's as important as your code
  • Expand it whenever a new failure mode is discovered in production

Regression testing

Run the golden dataset against the new prompt before shipping. Compare scores against the previous version. Only ship if the new version doesn't regress.

python 
def run_eval(prompt_v, test_cases, llm_fn):
    results = []
    for tc in test_cases:
        output = llm_fn(prompt_v, tc["input"])
        passed = output.strip() == tc["expected"]
        results.append({"input": tc["input"], "passed": passed,
                         "output": output, "expected": tc["expected"]})
    accuracy = sum(r["passed"] for r in results) / len(results)
    return accuracy, results

acc_v1, _ = run_eval(prompt_v1, test_cases, call_llm)
acc_v2, _ = run_eval(prompt_v2, test_cases, call_llm)

print(f"v1 accuracy: {acc_v1:.1%}")
print(f"v2 accuracy: {acc_v2:.1%}")
if acc_v2 < acc_v1:
    print("REGRESSION — do not ship v2")

The four key metrics

Accuracy

% of test cases that produce the expected output. For classification tasks use exact match. For open-ended tasks use LLM-as-judge (see below) or human eval.

Consistency

Run the same prompt 5× at temperature > 0 and measure variance. High variance = the prompt is ambiguous or the task is under-specified. Target: <10% output variation for production prompts.

python 
outputs = [call_llm(prompt, input) for _ in range(5)]
unique = len(set(outputs))
print(f"{unique}/5 unique answers")

Latency

Time-to-first-token (TTFT) and total completion time. CoT and long prompts increase latency. Measure p50 and p95 — p95 is what users experience on a bad day.

python 
import time
start = time.perf_counter()
result = call_llm(prompt, input)
latency_ms = (time.perf_counter() - start) * 1000

Cost

Track input + output tokens per call. A longer system prompt means higher input cost on every call. A verbose output means higher output cost. Calculate cost per 1k calls, not per call — the math looks very different at scale.

python 
usage = response.usage
cost = usage.prompt_tokens * 0.15/1e6    # gpt-4o-mini input
     + usage.completion_tokens * 0.60/1e6  # output

LLM-as-judge

For open-ended tasks (summaries, emails, code explanations) there's no single correct answer. Use a second LLM call to score the output against your criteria.

python 
judge_prompt = """
You are evaluating a customer support response.
Score it 1–5 on each dimension:
- Accuracy: does it answer the question correctly?
- Tone: is it professional and empathetic?
- Brevity: is it under 3 sentences?

Response to evaluate:
"{response}"

Return JSON: {{"accuracy": int, "tone": int, "brevity": int}}
"""

score = call_llm(judge_prompt.format(response=output), temp=0)

LLM-as-judge scales to thousands of examples automatically. It's less accurate than human eval but fast enough to catch obvious regressions. Use GPT-4o or Claude 3.5 as the judge for best results.

Tooling

LangSmith

Full tracing for LangChain apps. Set LANGCHAIN_TRACING_V2=true — every prompt, output, token count, and latency is logged automatically. Run datasets and compare prompt versions in the UI.

PromptLayer

Version control for prompts. Track which prompt version is in production, compare outputs side-by-side, and roll back if a new version regresses.

Braintrust / Evals

Purpose-built eval platforms. Define datasets and scoring functions, run evals in CI, track quality trends over time. Good for teams with many prompts in production.

Series cheat sheet

Patterns

  • ICTF — Instruction + Context + Constraints + Format
  • ReAct — Thought → Action → Observation loop
  • Critic-Refine — Generate → Critique → Rewrite
  • Planner→Executor — Plan first, execute in steps
  • Role+Task+Format — 3-line system prompt template

Temperature guide

  • 0 — extraction, classification, structured output
  • 0.3–0.7 — summarisation, analysis, code generation
  • 0.8–1.0 — creative writing, brainstorming

When to use tools vs prompting

  • Tools — live data, exact math, side effects, external APIs
  • Prompting — reasoning, writing, analysis over provided text

Common interview questions

What is prompt injection and how do you defend against it?

Malicious user input that overrides your system prompt. Defend with: input sanitisation + delimiters, system prompt hardening, output validation, and content moderation. No single defence is sufficient — layer them.

When would you use RAG vs fine-tuning?

RAG when: knowledge changes frequently, data is proprietary, you need citations, or budget is limited. Fine-tuning when: consistent style/format is more important than knowledge, latency matters, or the task is highly specialised and your prompt engineering has hit a ceiling.

How do you test a prompt systematically?

Build a golden dataset (20–50 labelled examples), run both prompt versions against it, compare accuracy scores. Measure consistency (variance across 5 runs), latency, and cost. Use LLM-as-judge for open-ended outputs. Only ship if the new version doesn't regress on any metric.

When does Chain-of-Thought hurt?

For simple classification and extraction tasks — it adds latency and cost with no accuracy gain. For high-volume pipelines — output tokens are billed; CoT can 10× your cost. CoT also doesn't guarantee correctness — it can reason fluently to a wrong answer.

What is "lost in the middle" and how do you handle it?

LLMs attend less to content in the middle of long contexts. Mitigation: place critical instructions at the start (system prompt) and end (just before the user turn). For RAG, use reranking to surface the most relevant chunks rather than relying on position.

Notes

LLM-as-judge inherits the judge model's biases

Using GPT-4o as judge for GPT-4o outputs creates a systematic preference for GPT-4o-style responses. Use a different provider as judge when possible (e.g., Claude evaluating GPT outputs) to reduce model-preference bias. For high-stakes evaluations, ensemble multiple judge models and average scores.

Golden datasets decay and need scheduled review

The input distribution of your production traffic shifts as user behaviour evolves. A golden dataset built six months ago may no longer represent current traffic patterns. Schedule quarterly golden-dataset review sessions to add failure modes discovered in production and remove examples that no longer reflect real use cases.

Top-line accuracy is a misleading single metric

A prompt that scores 90% accuracy by nailing easy cases but failing every edge case is worse in production than one that scores 80% uniformly. Always break accuracy down by case type — easy, ambiguous, adversarial — to understand exactly where a new prompt regresses before shipping.

PromptFoo supports CI/CD integration out of the box

Add promptfoo eval --ci to your GitHub Actions or GitLab CI pipeline so every pull request that touches a prompt file automatically runs the golden dataset evaluation. This prevents prompt regressions from reaching production undetected and makes prompt quality a first-class CI gate.

Prompt Testing FAQ

Why should I test prompts like code?

Prompts drift — a change that improves one case can break another. Without a test suite, regressions reach production silently. Treating each prompt change like a code change (with test cases and a golden dataset) lets you improve prompts safely and measure the impact of every edit.

What is a golden dataset for prompt testing?

A golden dataset is a curated set of 20–50 labelled input/output pairs that represent the cases your prompt must handle correctly. You run both the old and new prompt versions against it and compare scores. It should cover typical cases, edge cases, and known failure modes.

What is LLM-as-judge for prompt evaluation?

LLM-as-judge uses a separate LLM call to score prompt outputs on criteria like accuracy, relevance, or tone — useful for open-ended outputs where exact string matching is impractical. The judge prompt describes the scoring criteria and returns a structured score. Use GPT-4 or Claude as judge for best reliability.

What are the key metrics for evaluating prompt quality?

The four key metrics are: accuracy (does the output match the expected answer?), consistency (does the same prompt produce the same answer across 5 runs?), latency (time to first token and total response time), and cost (input + output tokens × price per token). Track all four — a prompt that improves accuracy but doubles cost may not be worth shipping.

What tools are available for prompt testing and evaluation?

Popular prompt testing tools include: PromptFoo (open-source, CI-ready, supports multiple providers), LangSmith (LangChain's testing and tracing platform), Braintrust (evaluation with LLM-as-judge), and Weights & Biases Prompts. For simple cases, a Python script that runs your prompt against a golden dataset and computes accuracy is often enough.

Series complete — what you now know

  • Prompt anatomy: system/user/assistant roles, temperature, context window management
  • Zero-shot vs few-shot: when to use each, how to write good examples
  • Chain-of-Thought: when it helps, when it hurts, and smarter alternatives
  • Role and persona: how to frame expertise, tone, and audience targeting
  • Structured output: schema-in-prompt, JSON mode, Pydantic validation
  • 5 reusable patterns: ICTF, ReAct, Critic-Refine, Planner→Executor, Role+Task+Format
  • Function calling: tool schemas, the tool loop, tools vs prompting
  • Security: injection, jailbreaks, data leakage, and a layered defence strategy
  • Testing: golden datasets, regression suites, accuracy/consistency/latency/cost metrics
← Security & Guardrails All Tutorials →