How much data do you need?
Quality beats quantity. A small dataset of excellent examples beats a large dataset of mediocre ones. General guidance:
- 50–100 examples — Minimum. Risky but possible if examples are perfect.
- 100–500 examples — Sweet spot for most tasks. Enough to learn behavior, low risk of overfitting.
- 500–1000 examples — Excellent. Very confident results, handles edge cases.
- 1000+ examples — Overkill for small tasks. Use if you have them, but diminishing returns.
Dataset formats
Chat messages format (Recommended)
List of message dictionaries with roles (system/user/assistant) and content. Best for instruction-following models. Used by SFTTrainer.
Instruction format
Objects with "instruction" and "output" keys. Simple, works well. Less flexible than chat messages.
Plain text (line by line)
One training example per line. Simplest. Good for character-level or simple tasks.
JSONL (Recommended for APIs)
JSON Lines: one JSON object per line. Compact, easy to parse, standard for OpenAI and others.
Writing good examples
- ✓ Be specific in instructions. Bad: "Write about AI". Good: "Explain the difference between prompt engineering and fine-tuning in 2 sentences".
- ✓ Include context in input. For a customer support bot, include customer tone, product name, issue details.
- ✓ Make outputs complete and correct. Don't skip steps. Write the full response you want the model to produce.
- ✓ Vary examples. Cover different scenarios, edge cases, tones. Don't repeat the same example.
- ✓ Use consistent formatting. If you want JSON output, format all outputs as JSON. Same for markdown, XML, etc.
What to avoid
- ✗ Vague instructions. Model learns to produce vague outputs.
- ✗ Inconsistent formatting. Sometimes JSON, sometimes plain text. Model gets confused.
- ✗ Copying bad outputs. Don't use real customer complaints or buggy code as examples without fixing them.
- ✗ Too much data redundancy. Remove near-duplicates. Dedup before training.
Data cleaning checklist
- □ Remove duplicate or near-duplicate examples
- □ Fix encoding issues (weird Unicode, mojibake)
- □ Remove examples with empty inputs or outputs
- □ Validate format (all examples follow the same structure)
- □ Check for PII (names, emails, passwords) and redact if needed
- □ Remove examples that are too short or too long (outliers)
Train/eval split
Always split your data: 80% training, 20% evaluation. Use the eval set to measure overfitting and choose when to stop training.
Eval set should include edge cases: Unusual inputs, boundary conditions, less common scenarios. This reveals where the model struggles.
Code example: building a dataset
import json
examples = [
{
"messages": [
{"role": "system", "content": "You are a customer support agent for TechCorp. Be concise and helpful."},
{"role": "user", "content": "How do I reset my password?"},
{"role": "assistant", "content": "Go to Login page → 'Forgot password' → enter your email. You'll receive a reset link within 2 minutes."}
]
},
{
"messages": [
{"role": "system", "content": "You are a customer support agent for TechCorp. Be concise and helpful."},
{"role": "user", "content": "My payment failed but I was charged."},
{"role": "assistant", "content": "Sorry about that. Please email billing@techcorp.com with your order ID. Refunds typically process in 3–5 business days."}
]
}
]
with open("dataset.jsonl", "w") as f:
for example in examples:
f.write(json.dumps(example) + "\n")
print(f"Wrote {len(examples)} examples to dataset.jsonl")Key takeaway
100 high-quality examples beat 1,000 mediocre ones. Spend time writing clear instructions and correct outputs. Your training data is the blueprint your model will follow.
Building datasets from real sources
The hardest part of data preparation is usually sourcing the examples, not formatting them. For customer support fine-tuning, your existing ticket system is a gold mine — export resolved tickets, clean up the agent responses, and format as instruction-response pairs. For code generation, pull from your company's internal repositories. For domain-specific summarization, gather documents from your knowledge base and write summaries manually or use a stronger model like Claude or GPT-4 to draft them for human review.
Synthetic data generation is increasingly popular and effective. The pattern is straightforward: use a frontier model to generate training examples, have a human review and correct them, then use the cleaned data to fine-tune a smaller, cheaper model. This teacher-student approach can produce a 7B model that matches GPT-4 quality on your specific task at a fraction of the inference cost. If your training data lives on the web — product documentation, research papers, competitor analysis — tools like Firecrawl's advanced extraction features can automate the collection of structured, clean text ready for your training pipeline.
Data Preparation FAQ
What format should fine-tuning data be in?
Most frameworks expect JSONL with instruction-response pairs. The standard format is a list of messages with role and content fields, similar to the OpenAI chat format.
How do I clean training data for fine-tuning?
Remove duplicates, fix encoding issues, filter out low-quality examples, and ensure consistent formatting. Validate that each example actually demonstrates the behavior you want the model to learn.
How many training examples do I need?
Quality beats quantity. Start with 100 to 500 carefully curated examples. If results are poor, improve example quality before adding more data.
Should I include negative examples in my dataset?
Yes, but carefully. Include examples of what not to do only if the model consistently makes that specific mistake. Too many negative examples can confuse the training signal.
Can I use synthetic data for fine-tuning?
Yes. Using a stronger model like GPT-4 to generate training data for a smaller model is a common and effective strategy, especially when real data is scarce or expensive to label.
Related tutorials
Continue learning with our training with LoRA guide and evaluation tutorial.