DS DevShelfHub Projects · AI tools
Tutorials / CrewAI / Reference / Methods / test()
Method Crew

test(): Reference Guide

By DevShelfHub

Runs the crew in evaluation mode with auto-scoring, returning structured test results.

See the CrewAI methods catalog, CrewAI introduction, kickoff() reference, and core concepts for surrounding context.

What is test()?

Crew.test(n_iterations, filename=None) executes the same task graph as a normal kickoff but routes outputs through CrewAI's evaluation harness: each iteration is scored by a dedicated eval LLM (or configured scorer) so you get numeric or rubric-style signal instead of eyeballing logs. That makes test() the right primitive for CI — gate merges when scores drop, store iteration traces under filename for post-mortems, and compare branches by diffing score distributions rather than prose.

The eval model should be strictly stronger or more literal than the agents under test; reusing the same cheap chat model for both roles yields inflated scores that miss hallucinations. Expect roughly twice the token spend per iteration versus kickoff() alone because scoring adds another completion pass.

test() complements train(): training collects human preference rows while testing automates acceptance criteria. After you change tools, memory backends, or prompts, run test() before shipping so regressions surface in GitHub Actions instead of customer tickets.

Use Cases

  • • CI quality gates
  • • A/B testing agent variants

Key Features

  • ✓ Auto-evaluation
  • ✓ Configurable n_iterations

When NOT to Use

Latency-sensitive paths — test() runs an extra eval model.

Notes

Flaky scores and nondeterminism

Temperature on the eval model, nondeterministic tools, and live web search all inject variance. Run enough iterations for confidence intervals to tighten, and freeze tool inputs in CI when possible.

Cost and quota planning

Each scored iteration bills both the crew and the evaluator. Multiply n_iterations by your PR frequency before enabling test() on every commit — nightly or main-branch gates are common compromises.

Separate prod and eval credentials

Point the eval LLM at a key with its own budget and rate limits so a runaway test job cannot exhaust production inference quotas.

When automated scoring lies

Rubrics miss tone, safety, and domain nuance. Keep a small human spot-check set for releases even when test() is green.

Parameters

Parameter Type Required Purpose
n_iterations int No How many test passes.
filename str | None No Optional results output file.

Code Examples

CLI

python 
crewai test -n 3

Capture structured results to disk

python 
from crewai import Crew

def gate_crew(crew: Crew) -> None:
    report = crew.test(n_iterations=2, filename='artifacts/last_eval.json')
    print(report)

Compare two prompt variants

python 
reports = {}
for label in ('v1', 'v2'):
    crew = build_crew(prompt_pack=label)
    reports[label] = crew.test(n_iterations=5)
print(reports)

When to Use

Quality gating in CI; comparing agent variants.

Common Mistakes

❌ Running test() with the same model as the crew

✅ Configure a separate stronger eval model.

Related: @tool decorator reference, Agent class reference, and the first Crew tutorial.

test() FAQ

What is test() in CrewAI?

Runs the crew in evaluation mode with auto-scoring, returning structured test results. Crew.test(n_iterations, filename=None) executes the same task graph as a normal kickoff but routes outputs through CrewAI's evaluation harness: each iteration is scored by a dedicated eval LLM (or configured scorer) so you get numeric or rubric-style signal instead of eyeballing logs. That makes test() the right primitive for CI — gate merges when scores drop, store iteration traces under filename for post-mortems, and compare branches by diffing score distributions rather th…

Which CrewAI types expose the method test()?

DevShelfHub documents test() on Crew. The reference maps it to Python module crewai.Crew — pin your installed crewai version and match imports to the snippet on this page.

When should I use test()?

Quality gating in CI; comparing agent variants.

When should I avoid test()?

Latency-sensitive paths — test() runs an extra eval model.

How do I call test() from Python?

crew.test(n_iterations=3)

Where can I explore more CrewAI API reference pages?

Open the CrewAI API reference index on DevShelfHub to search classes, methods, and decorators, each with runnable examples, parameters, common mistakes, and cross-links.