The prompting ladder
Start at the bottom. Only climb higher if the step below doesn't solve your problem.
Step 1: Prompt Engineering
Try detailed prompts and system messages first. Often enough for one-off tasks. Zero infra, instant. Cost: cheap (pay per API call).
Step 2: RAG (Retrieval-Augmented Generation)
Add knowledge from documents. Use when the model needs fresh or proprietary data. Cost: moderate (retrieval + API calls).
Step 3: Fine-tuning
Adapt the model's behavior, style, and reasoning. Use when Steps 1 and 2 aren't enough. Cost: upfront (training) + moderate (inference).
Signs you SHOULD fine-tune
- ✓ Consistent format needed — Always return JSON, XML, or specific schema. Fine-tune to enforce it.
- ✓ Domain vocabulary — Model doesn't know your jargon. Medical, legal, or specialized language.
- ✓ Base model fails with prompting — You've tried complex prompts; they still don't work well.
- ✓ High volume / low latency — Inference speed and cost matter. A fine-tuned smaller model can be cheaper than calling GPT-4.
- ✓ Offline requirements — Model runs on-device or offline. Can't rely on API.
- ✓ You have good examples — 100+ high-quality (input, output) pairs covering your use case.
Signs you should NOT fine-tune
- ✗ Need up-to-date knowledge — Use RAG. Fine-tuning is static; knowledge in the training data is frozen.
- ✗ Base model already works — Prompting + RAG solve your problem well. Don't add complexity.
- ✗ Fewer than 50 examples — Too little data. Risk of overfitting. Try prompt engineering first.
- ✗ Don't want to manage infra — Fine-tuning requires training, versioning, deployment. Use API fine-tuning (OpenAI, Together) if you must.
- ✗ Budget is tight — Fine-tuning has upfront costs. Prompting is pay-per-call and cheaper for low volume.
Cost and effort comparison
| Approach | Upfront Effort | Inference Cost | Freshness | Performance |
|---|---|---|---|---|
| Prompting | Low (write prompt) | $ (per token) | Instant | Good baseline |
| RAG | Moderate (build DB + retriever) | $$ (retrieval + API) | Always current | Very good |
| Fine-tuning | High (collect data + train) | $-$$ (or free, if self-hosted) | Static (retrain for updates) | Excellent (specialized) |
Real examples
Code Review Bot — Fine-tuning Won
Requirements: Review PRs using your code style, automatically catch your team's patterns, return structured feedback. Prompting failed because the model wouldn't consistently follow your style. RAG didn't help (no new knowledge needed). Fine-tuned on 200 example reviews. Result: consistent, fast, low cost at scale.
Customer FAQ Bot — RAG Won
Requirements: Answer questions using your help docs. Docs change monthly. Prompting alone was too generic. Fine-tuning would be overkill (data changes, not behavior). Built RAG over your help center. Result: always current, high accuracy, minimal maintenance.
Decision rule
Start with prompting. Add RAG if you need fresh knowledge. Only fine-tune if you need behavior change AND have the data. When in doubt, try steps 1 and 2 first.
Making the decision in real projects
The prompting-RAG-fine-tuning ladder is a useful mental model, but real projects often blend all three. A production system might use a fine-tuned model for consistent JSON output formatting, RAG for injecting fresh product catalog data, and carefully crafted system prompts for tone control. The question is not which single approach to use, but which combination minimizes total cost of ownership while meeting quality requirements.
One common anti-pattern is jumping to fine-tuning before exhausting simpler options. If you spend a week on prompt engineering and achieve 85% accuracy, fine-tuning might push you to 95% — but you should quantify whether that gap justifies the ongoing maintenance cost of a training pipeline, versioned datasets, and model deployment infrastructure. Conversely, if prompt engineering plateaus at 60% and the task requires consistent structured output, fine-tuning is almost certainly the right move. For teams that need to gather domain-specific training data from the web, our Firecrawl RAG integration guide demonstrates how to build an automated data collection pipeline.
Fine-tuning Decision FAQ
When should I fine-tune instead of using prompts?
Fine-tune when prompt engineering hits a ceiling: the model consistently misses your desired format, tone, or domain knowledge despite detailed instructions.
Is RAG better than fine-tuning?
RAG is better for factual accuracy with frequently changing data. Fine-tuning is better for learning new behaviors, styles, or domain-specific patterns. Many production systems combine both.
What are signs I should not fine-tune?
Avoid fine-tuning if you have fewer than 50 quality examples, your task can be solved with better prompts, or you need the model to access real-time information.
Can I combine RAG and fine-tuning?
Yes, and many production systems do. Fine-tune for style and format, then use RAG to inject current facts. This gives you both behavioral control and factual accuracy.
How long does fine-tuning take?
A LoRA fine-tune on a 7B model with 1,000 examples typically takes 30 minutes to 2 hours on a single GPU. Larger models and datasets scale linearly.
Related tutorials
Continue learning with our RAG pipeline tutorial and choosing a base model.