Deploying Fine-tuned Models

Step 1: Merge adapters

In training, you saved the tiny LoRA adapter weights separately. For deployment, merge the adapter back into the base model weights:

from peft import AutoPeftModelForCausalLM

# Load the checkpoint with LoRA adapters
model = AutoPeftModelForCausalLM.from_pretrained(
    "./fine_tuned_model",
    device_map="auto",
)

# Merge adapters into base weights
merged_model = model.merge_and_unload()

# Save merged model (single file, ready for deployment)
merged_model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")

Why merge? Faster inference (no adapter inference overhead). Easier deployment (single model, not base+adapter). Trade-off: larger file size.

Step 2: Inference

Use the merged model for inference:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_model_id = "mistralai/Mistral-7B-Instruct-v0.2"
adapter_path  = "./fine_tuned_model/adapter"

tokenizer  = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_path)

# Merge adapter into base weights for faster inference
model = model.merge_and_unload()
model.save_pretrained("./merged_model")
tokenizer.save_pretrained("./merged_model")

def chat(user_message: str) -> str:
    messages = [
        {"role": "system", "content": "You are a helpful customer support agent."},
        {"role": "user", "content": user_message},
    ]
    prompt  = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs  = tokenizer(prompt, return_tensors="pt").to(model.device)
    output  = model.generate(**inputs, max_new_tokens=200, do_sample=False)
    return tokenizer.decode(output[0][len(inputs["input_ids"][0]):], skip_special_tokens=True)

print(chat("How do I reset my password?"))

Serving options

Quantization for deployment

Quantization shrinks model size and speeds up inference. Two popular formats:

Cost comparison

FastAPI wrapper pattern

Wrap your inference function in a simple REST API for easy integration:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class QueryRequest(BaseModel):
    message: str

@app.post("/chat")
async def chat_endpoint(request: QueryRequest):
    response = chat(request.message)
    return {"response": response}

# Run: uvicorn script:app --reload

This gives you a REST API instantly. Deploy with Docker, scale with Kubernetes. Add auth, rate limiting, logging as needed.

Deployment architecture patterns

The choice between merging adapters and serving them separately has significant architectural implications. Merged models are simpler: one file, one deployment, no adapter management. But if you fine-tune frequently or serve multiple task variants, keeping adapters separate with a framework like vLLM (which supports runtime LoRA loading) reduces GPU memory costs dramatically — one base model in memory can serve ten different fine-tuned behaviors.

For production deployments, quantization is nearly always the right choice. A 4-bit quantized 7B model fits in 4GB of VRAM with negligible quality loss for most tasks. This means you can serve it on an RTX 3060 or even a laptop GPU. The path from training to deployment is: train with QLoRA, merge adapters, quantize to GGUF or GPTQ, deploy with Ollama (local) or vLLM (server). Each step reduces size and increases speed. For applications that combine fine-tuned models with live web data, pairing your deployed model with Firecrawl for real-time content extraction creates a powerful hybrid system.

Model Deployment FAQ

Related tutorials

Continue learning with our evaluation tutorial and all tutorials.