Training
Run interactively to collect supervised feedback. Each iteration runs kickoff() on the crew, prompts for ratings, and persists the labeled data to a pickle for future runs to replay.
crewai train -n 5 -f trained.pklAPI: Crew.train().
Testing
Auto-eval mode: scores n_iterations using a separate (typically stronger) eval LLM. Wire into CI to detect quality regressions in PRs.
crewai test -n 3API: Crew.test().
Replay
Re-run from a specific task ID, reusing earlier task outputs. Useful when the last step fails and re-running the whole crew is expensive.
crewai replay -t abc-123API: Crew.replay().
Resetting Memories Between Runs
Use reset_memories with the right scope flags:
crewai reset-memories --all
crewai reset-memories --short
crewai reset-memories --knowledge --agent-knowledge
crewai reset-memories -k # latest kickoff task outputs onlyBest Practices
- ✓Train offline; deploy the resulting pickle. Don't train in production.
- ✓Test with a stronger eval model than the crew's working model.
- ✓Replay only when upstream data hasn't changed.
CrewAI train, test, and replay FAQ
What does CrewAI train() do?
train() runs an interactive loop that collects human ratings after each kickoff and persists labeled data to a pickle so future runs can replay the feedback and bias outputs toward higher quality.
How is CrewAI test() different from a normal kickoff?
test() executes the crew in evaluation mode with an auto-scoring eval LLM across multiple iterations so you can gate regressions in CI instead of eyeballing prose manually.
When should I use CrewAI replay()?
Use replay() when a late task fails but earlier outputs are still valid so you can resume from a task_id without paying the full upstream cost again.
What does crewai reset-memories control?
CLI flags choose which memory scopes to wipe such as short-term buffers, entity memory, knowledge stores, or everything so tests and migrations start from a clean slate.
Should I run CrewAI training in production?
No. Train offline, ship the resulting artifact, and keep production on deterministic kickoff paths with observability instead of interactive rating prompts.
Where can I read CrewAI method signatures for train, test, and replay?
Open the DevShelfHub CrewAI API reference entries for train(), test(), replay(), and reset-memories alongside this tutorial for parameters, CLI equivalents, and pitfalls.