When should I fine-tune an LLM instead of prompting?

Fine-tune when prompt engineering and RAG have hit their ceiling: the model consistently fails a specific format, domain vocabulary, tone, or task despite thorough few-shot examples. Fine-tuning excels at style consistency, domain jargon, and latency-sensitive deployments where you need a smaller, specialised model instead of a large general one.

What is the difference between LoRA and QLoRA?

LoRA (Low-Rank Adaptation) freezes the base model weights and trains small low-rank adapter matrices injected into the attention layers, reducing trainable parameters by 10 000x. QLoRA adds 4-bit NormalFloat quantisation of the frozen base model, cutting GPU memory by roughly 60 percent so you can fine-tune a 7B model on a single 24 GB consumer GPU.

What dataset format do I need for SFT fine-tuning?

Supervised fine-tuning (SFT) expects instruction-response pairs in a chat format. The most common schema is a list of messages with role and content fields, matching the ChatML or Llama chat template your base model uses. Aim for at least 1 000 high-quality examples; data quality matters far more than quantity for most narrow tasks.

What is DPO fine-tuning?

Direct Preference Optimisation (DPO) trains the model directly on pairs of chosen and rejected responses without needing a separate reward model. It is the most common replacement for RLHF in open-source workflows because it is stable, needs no PPO tuning, and typically requires only 500 to 2 000 preference pairs to steer tone, safety, or format.

Should I use a hosted fine-tuning service or run it myself?

Use hosted services (OpenAI, Cohere, Vertex AI) when you want zero infrastructure overhead, fast iteration, and can share data with the provider. Run it yourself with Hugging Face TRL or Axolotl when you need data privacy, control over the base model, cost efficiency at scale, or want to fine-tune an open-weight model like Llama or Mistral.

LLM Fine-Tuning Cheatsheet: LoRA, SFT and DPO

Start hereQuick start · 6 you’ll reach for daily

DatasetJSONL of messages / prefs

MethodSFT → LoRA → QLoRA

PreferenceDPO (no reward model)

TrainSFTTrainer / DPOTrainer

Hostedclient.fine_tuning.jobs.create()

ServevLLM / TGI / hosted endpoint

Target versions · paceVersions

Targets: transformers ≥ 4.45 trl ≥ 0.12 peft ≥ 0.13 bitsandbytes ≥ 0.43

Open-source snippets pin to the Hugging Face stack (transformers + trl + peft). Hosted snippets use OpenAI’s fine-tuning API. Anthropic does not currently offer base model fine-tuning — reach for prompt + tools + RAG there. Names current as of May 2026.

install · trackSetup

bash

# Open-source stack (LoRA / QLoRA on a single GPU)
pip install -U transformers trl peft datasets accelerate bitsandbytes evaluate

# Faster CUDA kernels (Ada / Hopper)
pip install -U flash-attn --no-build-isolation
pip install -U unsloth                    # 2-4x speedups for QLoRA, single-GPU

# Hosted (no GPU needed)
pip install -U openai                     # OpenAI fine-tuning API
pip install -U anthropic                  # Anthropic does not currently offer base fine-tuning
                                          # (use prompt + tools instead)

# Track + reproduce
pip install -U wandb tensorboard
huggingface-cli login                     # push checkpoints / pull base models
wandb login                               # optional, runs UI

decide firstWhen to fine-tune

Style / format mismatch	Strong signal. Model is capable but won’t output the shape you need.
Domain jargon / new tasks	Moderate signal. Try prompt + RAG first.
Throughput / latency cost	Fine-tune a smaller model to replace a big one.
Adding facts	Avoid Use RAG. Fine-tuning a model on facts is unreliable.
Tool-use repair	Moderate signal. Often a prompt + few-shot fix instead.
Safety / refusal behaviour	Strong signal for DPO with preference pairs.
Cold-start before data	Avoid Without 200+ examples, prompt engineering wins.

Ladder: prompt → few-shot → RAG → fine-tune. Climb only when the rung below stops paying off. Fine-tuning is the slowest, most expensive, hardest-to- iterate rung; arrive there with evidence.

SFT · PEFT · preferenceMethods

Full SFT	Update every weight. Best quality, biggest GPUs, slowest iteration.
LoRA	Preferred Update low-rank adapters; freeze base. 1-5% params, near-full quality.
QLoRA	4-bit base + LoRA adapters. Trains 7-13B on a single 24GB GPU.
DoRA	LoRA + magnitude decomposition. Marginal quality lift; same memory.
Prefix / prompt tuning	Train soft tokens only. Tiny artefacts, mediocre quality.
DPO	Preferred Preference alignment without a reward model.
KTO	Like DPO but accepts unpaired thumbs-up / thumbs-down data.
RLHF / PPO	Reward model + PPO. Powerful, brittle. Avoid unless you have a team.
Reinforcement fine-tuning (RFT)	Hosted-only path (OpenAI). Reward-signal-driven SFT.
Continued pre-training	More base pre-training on domain text. Use to inject vocabulary, not facts.

JSONL formats + hygieneDataset shape

Chat format	`{"messages": […]}`. Default for instruction-tuned chat models.
Prompt / completion	Legacy for base models. `{"prompt", "completion"}`.
Preference pairs	`{prompt, chosen, rejected}`. For DPO / IPO / KTO.
Min size	SFT: 200+ for a signal, 1k-10k typical. DPO: 500+ pairs.
Train / val split	90 / 10. Hold out a third “eval” set the trainer never sees.
De-duplication	Drop near-duplicates; they over-fit fast. Hash by normalised text.
Token budget	Most rows under model context. Truncate or split long ones.
Class balance	For classification-like tasks, balance positive / negative / edge.
PII scrub	Real-traffic data → mask emails, names, IDs before upload.

json

# JSONL — one example per line. Two shapes you'll meet:

# 1) Chat (preferred for instruction-tuning chat models)
{"messages": [
  {"role": "system", "content": "You are a triage bot."},
  {"role": "user",   "content": "My order #123 hasn't shipped."},
  {"role": "assistant", "content": "I'll check that order now..."}
]}

# 2) Prompt -> completion (legacy, base models)
{"prompt": "Translate to French: 'How are you?'", "completion": " Comment ca va ?"}

# 3) Preference pairs for DPO / RLHF (chosen vs rejected)
{
  "prompt":   "Suggest a one-line variable name for total revenue.",
  "chosen":   "total_revenue",
  "rejected": "tr"
}

# Quick dataset hygiene checks
python - <<'PY'
import json
from collections import Counter
rows = [json.loads(l) for l in open("train.jsonl")]
print("n =", len(rows))
print("avg msgs:", sum(len(r["messages"]) for r in rows) / len(rows))
print("dup prompts:", sum(c > 1 for c in Counter(r["messages"][1]["content"] for r in rows).values()))
PY

starting pointsHyperparameters

epochs: 1–3 (SFT)	More overfits fast. Stop at the validation knee.
learning_rate: 1e-4–2e-4 (LoRA)	10× the full-SFT rate.
learning_rate: 5e-7–1e-6 (DPO)	Tiny. Larger destabilises preferences.
warmup_ratio: 0.03	3% of steps. Prevents loss spike.
scheduler: cosine	Strong default for SFT + DPO.
batch_size: max that fits + grad accum	Effective batch = per-device × grad-accum × #GPUs.
precision: bf16 (or fp16)	bf16 on A100 / H100. fp16 on older.
LoRA r: 8–16, alpha: 16–32	Default. Higher r helps domain shift; rarely needed.
LoRA dropout: 0.05	Regularises small datasets.
target_modules	q, k, v, o projections at minimum. Add gate / up / down for harder tasks.
DPO beta: 0.1	Lower = closer to SFT model; higher = sharper preferences.

python

# QLoRA SFT on a single 24 GB GPU — open-source path.
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

base = "meta-llama/Llama-3.1-8B-Instruct"

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tok = AutoTokenizer.from_pretrained(base)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(
    base, quantization_config=bnb, device_map="auto",
    attn_implementation="flash_attention_2",
)

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)

ds = load_dataset("json", data_files="train.jsonl", split="train")

cfg = SFTConfig(
    output_dir="out", num_train_epochs=3,
    per_device_train_batch_size=2, gradient_accumulation_steps=8,
    learning_rate=2e-4, warmup_ratio=0.03, lr_scheduler_type="cosine",
    bf16=True, logging_steps=10, save_strategy="epoch",
)

trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds,
                     peft_config=lora, tokenizer=tok)
trainer.train()
trainer.save_model("out/lora-final")

trade-offsHosted vs open-source

OpenAI fine-tuning API	JSONL upload, automatic hyperparams, hosted serving. No GPU.
OpenAI RFT	Hosted-only reinforcement fine-tuning with a reward function.
Hugging Face AutoTrain	Managed UI + cluster on top of `transformers`.
Modal / Replicate / RunPod / Together	Pay-per-GPU. Bring your own training script.
Self-host single GPU	QLoRA on a 24GB GPU trains 7-13B models. Local dev loop.
Self-host multi-GPU	DeepSpeed / FSDP for > 30B. Real ops work.
Cost feel	Hosted: simple, opaque, “per token”. OSS: complex, transparent, “per GPU-hour”.

python

# Hosted SFT on OpenAI — no GPU, JSONL upload, polling for status.
from openai import OpenAI

client = OpenAI()

# 1 · Upload the JSONL (chat-format examples)
train = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
val   = client.files.create(file=open("val.jsonl",   "rb"), purpose="fine-tune")

# 2 · Create the job
job = client.fine_tuning.jobs.create(
    model="gpt-4o-mini-2024-07-18",
    training_file=train.id,
    validation_file=val.id,
    hyperparameters={"n_epochs": 3},        # or "auto"
    suffix="triage-v1",
)
print(job.id, job.status)

# 3 · Poll
while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    print(job.status, job.trained_tokens)
    if job.status in ("succeeded", "failed", "cancelled"):
        break

# 4 · Use the new model id
resp = client.chat.completions.create(
    model=job.fine_tuned_model,
    messages=[{"role": "user", "content": "Ship status for #123?"}],
)
print(resp.choices[0].message.content)

measure the liftEval

Held-out set	10-30% of data the trainer never sees. The single most important number.
Task-specific golden set	100–500 hand-curated cases representing real traffic.
LLM-as-judge	Rubric-scored grader. Cheap for fuzzy criteria; check for bias toward your model.
Pairwise A/B	New vs base side-by-side. Fastest signal for “is it better”.
Regression on capabilities	Hellaswag / MMLU subset. Catch general-capability loss after SFT.
Tool-use accuracy	For function-calling fine-tunes: schema-valid call rate, arg correctness.
Refusal rate	Watch for over-refusal after safety DPO. Common failure mode.
Cost / latency	Smaller fine-tuned model vs prompted big model: tokens, RPS, p95.

ship the artefactServing

Merge LoRA	`model.merge_and_unload()`. Produces a single fused checkpoint.
Push to Hub	`model.push_to_hub("org/name", private=True)`.
vLLM	Preferred for throughput. PagedAttention; multi-LoRA hot-swap.
TGI (Text Generation Inference)	Hugging Face server. Strong batching + streaming.
Ollama / llama.cpp	CPU + Apple silicon. Convert to GGUF.
SGLang	High-throughput, fast prefill. RadixAttention cache.
Multi-LoRA in one process	vLLM lets you serve N adapters over one base. Big cost win.
Versioning	Tag commit + dataset hash. Required for rollback.

SFT → DPO → mergeEnd-to-end · SFT + DPO with LoRA

SFT on instruction data first, then DPO on preference pairs starting from the SFT checkpoint, then merge and save. The shape almost every alignment recipe ends up in.

python

# Full path: clean -> SFT -> DPO -> eval -> push.
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer, DPOConfig, DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, PeftModel

base = "Qwen/Qwen2.5-7B-Instruct"
tok  = AutoTokenizer.from_pretrained(base)
tok.pad_token = tok.eos_token

# 1 · SFT on instruction data
sft_model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16",
                                                 device_map="auto")
sft_ds = load_dataset("json", data_files="sft.jsonl", split="train")
SFTTrainer(
    model=sft_model, tokenizer=tok, train_dataset=sft_ds,
    peft_config=LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM"),
    args=SFTConfig(output_dir="sft-out", num_train_epochs=2,
                   learning_rate=2e-4, bf16=True),
).train()

# 2 · DPO on preference pairs — start from SFT checkpoint
dpo_ds = load_dataset("json", data_files="prefs.jsonl", split="train")
dpo = DPOTrainer(
    model=PeftModel.from_pretrained(sft_model, "sft-out/checkpoint-last"),
    ref_model=None, beta=0.1, tokenizer=tok, train_dataset=dpo_ds,
    args=DPOConfig(output_dir="dpo-out", num_train_epochs=1,
                   learning_rate=5e-7, bf16=True),
)
dpo.train()
dpo.model.merge_and_unload().save_pretrained("dpo-merged")
tok.save_pretrained("dpo-merged")

Best practiceGood to know

Start with QLoRA on a small base. 7-8B model, LoRA r=16, 1-2 epochs. Iteration time dominates outcome. You can scale up once the recipe shows lift on a held-out set.

DPO from an SFT checkpoint, not from base. Skipping SFT makes DPO chase noise. The standard recipe is: SFT first, then DPO from that.

Freeze your eval before training. Decide what “better” means — the held-out metric, the LLM-as-judge rubric — before kicking off the run. Otherwise you’ll grade after the fact and pick the run that supports your story.

Common trapsWatch out for

Catastrophic forgetting. Heavy SFT on a narrow task hollows out general capability. Sprinkle in 5-15% generic instruction data from a public set (Alpaca / OpenOrca / Tulu) as a regulariser.

Don’t fine-tune for facts. Models memorise inconsistently; updating one fact contaminates neighbouring ones. Put facts in retrieval (RAG), not weights.

Tokeniser mismatch breaks LoRA stacking. Two adapters on different bases — or even different tokeniser versions of the same model — cannot be merged. Always note the exact base + tokeniser hash with each checkpoint.