The problem with full fine-tuning
In full fine-tuning, you update every weight in the model during training. For Llama 7B (7 billion parameters), this means:
- Huge GPU memory. 7B parameters × 4 bytes (float32) = 28GB base model + optimizer states = 80–100GB needed.
- Slow training. Update billions of weights each step.
- Expensive. Needs an A100 or H100 ($2–5 per hour on cloud).
- Hard to version. Every fine-tune produces a new 7B model file to store.
Result: full fine-tuning is practical only for large organizations with serious compute budgets.
PEFT: Parameter Efficient Fine-Tuning
PEFT is the umbrella term: instead of updating all weights, you freeze the base model and only train small additions. Common approaches:
LoRA (Low-Rank Adaptation) ⭐ Most popular
Add small matrices A and B to each layer. Learn ΔW = B × A where rank r is tiny. Trainable: 0.1% of base model. Amazing quality/size tradeoff.
QLoRA
LoRA + 4-bit quantization of base model. Fit a 7B model in 6GB VRAM instead of 40GB. Slightly lower quality but massive memory savings.
Prefix Tuning
Learn a small prompt prefix added to each layer. Less popular than LoRA.
Prompt Tuning
Only learn soft prompts (continuous vectors). Smaller models only.
How LoRA works (simplified)
Full fine-tuning modifies weights directly: W → W + ΔW. This is expensive. LoRA instead learns small matrices:
The rank r controls adapter size. Smaller r = fewer parameters but lower capacity. Common values:
- r=4 — Tiny (good for mobile or constrained devices)
- r=8 — Small (default for most tasks)
- r=16 — Medium (good quality, still efficient)
- r=32 — Large (near full fine-tuning quality but still 50x smaller)
QLoRA: LoRA + 4-bit quantization
Quantization: store model weights in fewer bits. QLoRA applies 4-bit quantization to the base model (not the LoRA adapters):
- Base model (7B) stored in 4-bit: ~2GB instead of 28GB
- LoRA adapters (0.1% of model): ~20MB
- Total VRAM needed: ~6GB (fits on consumer GPU!)
- Trade-off: slight quality loss, but minimal in practice
QLoRA is the sweet spot: maximum accessibility with minimal quality loss.
Comparison table: which method?
| Method | Trainable % | Min VRAM (7B) | Speed | Quality |
|---|---|---|---|---|
| Full fine-tuning | 100% | 80GB | Slow | Best |
| LoRA (r=8) | 0.1% | 40GB | Fast | Excellent |
| QLoRA (r=8) | 0.1% | 6–8GB | Very fast | Excellent |
| Prefix Tuning | ~1% | 50GB | Fast | Good |
When to use each
Use full fine-tuning when:
You have unlimited GPU budget, need highest quality, or are fine-tuning small models. Rare in practice.
Use LoRA when:
You have a 24–40GB GPU (A100 or RTX 6000). Good balance of quality and speed.
Use QLoRA when: ⭐ RECOMMENDED
You want to minimize cost and fit on consumer hardware. Works on RTX 4090, RTX 3060, even some laptop GPUs. Default choice for most people.
Key takeaway
LoRA and QLoRA are game-changers. They make fine-tuning accessible: high quality, low cost, fit on consumer hardware. For 99% of cases, use QLoRA.
LoRA in production systems
LoRA's real power emerges in production. Because adapters are tiny files (10-50MB), you can version them like code artifacts. A single base model can serve dozens of specialized tasks by hot-swapping adapters at inference time. Companies running multi-tenant systems use this pattern extensively: one Llama 8B base model in GPU memory, with different LoRA adapters loaded per customer request for customer support, code review, or content generation.
The rank parameter deserves experimentation. Start with r=8 and measure eval loss. If it plateaus early, try r=16 or r=32. For simple format-following tasks (always return JSON, match a template), r=4 often suffices. For complex behavior changes (domain-specific reasoning, multi-step problem solving), r=16 to r=32 yields meaningful gains. The alpha parameter (scaling factor) should typically equal the rank — this is a safe default that prevents training instability. If you are building a pipeline that collects training data from technical documentation, our Firecrawl web crawling tutorial demonstrates how to systematically extract content across entire documentation sites.
PEFT & LoRA FAQ
What is LoRA in fine-tuning?
LoRA (Low-Rank Adaptation) freezes the original model weights and trains small adapter matrices instead. This reduces memory usage by 10x or more while achieving comparable performance to full fine-tuning.
What is QLoRA and how is it different from LoRA?
QLoRA quantizes the base model to 4-bit precision before applying LoRA adapters. This cuts memory usage further, letting you fine-tune a 7B model on a GPU with just 6GB of VRAM.
What rank should I use for LoRA?
Start with rank 8 or 16 for most tasks. Higher ranks capture more complex patterns but use more memory. Increase the rank only if training loss plateaus and you have spare GPU memory.
Is LoRA as good as full fine-tuning?
For most practical tasks, LoRA matches or comes within 1-2% of full fine-tuning performance. The gap narrows with higher ranks and careful hyperparameter tuning.
Which layers should I target with LoRA?
Target the attention projection layers (q_proj, v_proj) as a baseline. Adding k_proj and output projections can improve results. Targeting all linear layers gives the best quality but uses more memory.
Related tutorials
Continue learning with our training with LoRA guide and choosing a base model.