Will local models match GPT-4 or Claude?

Not at consumer hardware levels. 70B-class open models on serious GPUs come close on some tasks but still trail on long-context reasoning, coding, and multimodal work. For the 80% of prompts that are summarisation, classification, or short-form generation, they're close enough.

Can I run local models on Windows without WSL?

Yes for Ollama (native installer). Docker Model Runner runs inside Docker Desktop, which uses WSL2 under the hood — you don't manage WSL directly.

Do these work with LangChain / LlamaIndex?

Yes. LangChain has first-class Ollama integration. Both LangChain and LlamaIndex accept any OpenAI-compatible endpoint, so Docker Model Runner works through the OpenAI provider with a base-URL override.

How do I expose a local model to teammates?

Put it on a small server with a GPU, run Ollama or Docker Model Runner there, and either expose the port over a VPN/Tailscale, or wrap it in a thin auth proxy. Never expose raw inference ports to the open internet.

Can I fine-tune models locally?

Possible but heavy. Use a small base model and tools like Unsloth, LLaMA Factory, or Axolotl for LoRA fine-tuning. Production-grade fine-tuning usually runs on rented cloud GPUs and the result gets pulled back to Ollama for local serving.

What about Apple's MLX or LM Studio?

LM Studio is a GUI alternative to Ollama with a similar feature set. MLX is Apple's framework for fast inference on Apple Silicon — powerful but lower-level. For most developers, Ollama or Docker Model Runner get you 95% of the value with less setup.

Run LLMs Locally with Ollama and Docker

Introduction

Running LLMs locally is no longer the niche, expensive setup it was in 2023. Two tools have made it pip-install easy: Ollama and the newer Docker Model Runner. Both let you download a model, run it interactively, and call it from code over a local HTTP API — effectively a drop-in replacement for the OpenAI API when you care about speed, privacy, or cost.

This is a side-by-side developer’s guide. How to install each one, how to pull a model, how to call it from Python (either via raw HTTP or with a client SDK), and when one tool fits better than the other.

📚 Table of contents

Why run LLMs locally at all
Hardware reality check
Method 1: Ollama — install, pull, run
Calling Ollama from Python (HTTP and SDK)
Method 2: Docker Model Runner — setup and CLI
Calling Docker Model Runner from Python
Pointing the OpenAI SDK at your local model
Ollama vs Docker Model Runner: which to pick
Common mistakes
FAQs

Why run LLMs locally at all

Privacy — the prompt never leaves your machine. Useful for sensitive data, draft legal docs, internal company information.
Cost — no per-token fees. Once you’ve downloaded the model, inference is free.
Speed (sometimes) — small models running on a decent GPU beat round-tripping to OpenAI for many tasks. No network latency.
Offline access — planes, conferences, secure environments without internet.
Reproducibility — the model file is yours; it won’t get deprecated or quietly updated.

Hardware reality check

Local LLM size scales with available memory. Rough guide:

8 GB RAM / no GPU — tiny models only (135M–1B params). Good for completion and classification, not reasoning.
16 GB unified memory (Apple Silicon) — 7B–8B quantized models run well. Llama 3 8B, Gemma 7B, Qwen 7B.
24–32 GB GPU VRAM — 13B–30B quantized models. Approaches usable quality for code and chat.
64 GB+ unified memory or 48 GB+ VRAM — 70B class models. This is where local quality starts to rival small cloud models.

Not sure? Drop your specs into ChatGPT and ask which models will run. For first-time setup, pick the smallest viable model — you’re testing the plumbing, not benchmarking quality.

Method 1: Ollama — install, pull, run

Ollama is the most popular open-source tool for managing local LLMs. Single download for macOS, Windows, or Linux at ollama.com. After install, make sure the Ollama service is running (menu bar on Mac, system tray on Windows), then drop into a terminal.

Bash

# Verify install
ollama --version

# Pull a small model (~270 MB) to test the pipeline
ollama pull smollm2:135m

# List what you have
ollama list

# Interactive chat
ollama run smollm2:135m
> What is the capital of Canada?
# (very small models like this will hallucinate — that's expected)
> /bye    # exits

Browse ollama.com/library for the full model catalog with sizes. For real work, try llama3:8b, gemma2:9b, or qwen2.5-coder:7b on a machine with 16 GB+ memory.

Calling Ollama from Python (HTTP and SDK)

Ollama exposes a REST API on localhost:11434. Two ways to call it:

Python — raw HTTP

import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "smollm2:135m",
        "stream": False,
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Write 200 words on the fall of Rome."},
        ],
    },
    timeout=120,
)
print(response.json()["message"]["content"])

No SDK, no dependencies beyond requests. Works in any language that can POST JSON.

Python — ollama SDK

# pip install ollama
import ollama

response = ollama.chat(
    model="smollm2:135m",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write 200 words on the fall of Rome."},
    ],
)
print(response["message"]["content"])

Method 2: Docker Model Runner — setup and CLI

Docker Model Runner ships with recent Docker Desktop versions. The pitch: more efficient GPU acceleration, broader hardware support, and — the killer feature — containerized apps can talk to it natively without bundling the model.

Setup:

Install Docker Desktop (free) and update to the latest version.
Open Docker Desktop → Settings → AI → enable Docker Model Runner and enable host-side TCP support.
If you don’t see the AI panel, enable Docker MCP Toolkit under Beta Features first.

Once enabled you can manage models from the GUI (Docker Desktop → Models) or from the CLI:

Bash

docker model pull ai/smollm2
docker model list
docker model run ai/smollm2
> hello
> /bye

Models live on the Docker Hub at hub.docker.com under the AI namespace. The CLI surface mirrors Ollama’s, deliberately.

Calling Docker Model Runner from Python

Same shape as Ollama, different port. DMR runs on localhost:12434:

Python

import requests

response = requests.post(
    "http://localhost:12434/engines/llama.cpp/v1/chat/completions",
    json={
        "model": "ai/smollm2",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Explain transformers in 200 words."},
        ],
    },
    timeout=120,
)
print(response.json()["choices"][0]["message"]["content"])

Notice the path: DMR speaks the OpenAI chat-completions schema. That’s a big deal — you can point any OpenAI-compatible SDK at it without changing your code.

Pointing the OpenAI SDK at your local model

Both Ollama and DMR support the OpenAI chat-completions API surface. Override the base URL on the OpenAI client and you have a drop-in local backend — same code, different endpoint:

Python

# pip install openai
from openai import OpenAI

# Docker Model Runner
client = OpenAI(
    base_url="http://localhost:12434/engines/llama.cpp/v1",
    api_key="not-used",   # required by SDK, ignored by DMR
)
res = client.chat.completions.create(
    model="ai/smollm2",
    messages=[{"role": "user", "content": "Explain transformers in 200 words."}],
)
print(res.choices[0].message.content)

Switching between local development and production OpenAI becomes a single environment-variable change. Same code path, same SDK, just different base_url and model name.

Ollama vs Docker Model Runner: which to pick

Pick Ollama if…

You want the simplest single-binary install.
You’re not already using Docker on this machine.
You like the broad model catalogue and frequent updates.
You’re scripting against Ollama-specific frameworks (LangChain’s Ollama integration is very mature).

Pick Docker Model Runner if…

You already run Docker Desktop.
You’re building containerized apps and want one networked model service.
You want the OpenAI-compatible endpoint without a translation layer.
You need better GPU acceleration on supported hardware.
You plan to ship the same compose file in dev and prod.

Most developers will end up using both: Ollama for quick experiments and the menu-bar UX, DMR when they containerize for deployment.

❌ Common mistakes

Picking a model too large for your hardware and watching the OS swap to disk. Start small.
Forgetting to enable host-side TCP support on Docker Model Runner — you’ll get connection refused on 12434.
Expecting tiny models (sub-1B) to handle reasoning. They’re for completion and classification, not chain-of-thought.
Not setting a timeout on HTTP calls. Local inference still blocks — runaway prompts hang your script.
Leaving Ollama or DMR running 24/7 on a laptop you actually use. Memory pressure and battery hit add up.
Skipping prompt-caching strategies on hardware-limited setups. Re-prompting from scratch wastes compute the cache could cover.

💡 Pro tips

Write your code against the OpenAI SDK and treat the base_url as configuration. Free swap between local and cloud.
Pair a small local model with a cloud fallback — cheap and fast for routine prompts, escalate to GPT-4o / Claude only when needed.
Use q4_K_M quantizations for the best size/quality ratio on consumer hardware.
Tail Ollama logs (~/.ollama/logs) when models load slowly — you usually find the bottleneck is paging.
For agent workflows, prefer tool-calling-capable models (Qwen 2.5, Llama 3.1+, Gemma 2). Old models without function-calling support break LangChain agents.

Conclusion

The barrier between “cloud API” and “local model” has collapsed for developers. Pick Ollama for simplicity or Docker Model Runner for containerized work; write your code against the OpenAI SDK; switch between them with one environment variable. The 2026 default for AI-powered apps isn’t “always cloud” or “always local” — it’s “whichever fits the request,” and that flexibility is finally free.

Run LLMs Locally with Ollama and Docker Model Runner