What Ollama actually is
Ollama is a small Go binary that wraps llama.cpp and exposes it as a long-running local service on localhost:11434. When you run a model it boots a child process holding the weights in memory, accepts requests over a REST API, and unloads automatically after five minutes of inactivity. The CLI you type at (ollama run, ollama pull) is a thin client that talks to that same service. Everything lives on your machine — no telemetry, no cloud round-trip, no API key.
The reason to use it instead of raw llama.cpp is that Ollama handles the rough edges: model discovery and download from a CDN, GGUF quantisation defaults, GPU detection (Metal on Apple Silicon, CUDA on NVIDIA, ROCm on AMD), model lifecycle, and an OpenAI-compatible /v1 endpoint that drops into any existing OpenAI SDK call site without code changes. If your goal is to ship local inference inside an app this week, Ollama is the shortest path. If you need fine-grained control over sampling or custom kernels, drop down to llama.cpp directly.
Install Ollama
macOS
brew install ollamaOr download the .dmg from ollama.com — it installs as a menu bar app.
Linux
curl -fsSL https://ollama.com/install.sh | shInstalls the Ollama service and CLI. Starts automatically on boot via systemd.
Windows
Download OllamaSetup.exe from ollama.com. Requires Windows 10/11. GPU acceleration requires an NVIDIA card with CUDA or an AMD card with ROCm.
After install, verify it works:
ollama --versionPull and run your first model
One command downloads the model and starts an interactive chat:
ollama run llama3.2Downloads Llama 3.2 3B (~2 GB) on first run, then opens a chat prompt. Type your message and press Enter. Type /bye to exit.
Other useful commands
ollama pull llama3.2 # download without running
ollama list # list downloaded models
ollama show llama3.2 # model info (size, parameters)
ollama rm llama3.2 # delete a model
ollama ps # show running modelsREST API
Ollama exposes a local REST API at http://localhost:11434. Use it to integrate Ollama into any application.
Generate (single completion)
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Why is the sky blue?",
"stream": false
}'Chat (multi-turn)
curl http://localhost:11434/api/chat -d '{
"model": "llama3.2",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is Python?"}
],
"stream": false
}'OpenAI-compatible endpoint
Ollama also exposes an OpenAI-compatible API — swap the base URL in any OpenAI client:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:11434/v1",
api_key="ollama", # required but not used
)
response = client.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)Python client
The official ollama Python package provides a cleaner interface than raw HTTP.
pip install ollamaimport ollama
# Single completion
response = ollama.generate(model="llama3.2", prompt="Why is the sky blue?")
print(response["response"])
# Chat
response = ollama.chat(
model="llama3.2",
messages=[{"role": "user", "content": "Explain recursion simply."}],
)
print(response["message"]["content"])
# Streaming
for chunk in ollama.generate(model="llama3.2", prompt="Count to 5.", stream=True):
print(chunk["response"], end="", flush=True)Model library
Ollama's model library at ollama.com/library hosts hundreds of models. Each model has tagged variants for different sizes and quantizations.
ollama pull llama3.2 # 3B, default (Q4_K_M)
ollama pull llama3.2:1b # 1B — faster, less capable
ollama pull llama3.1:8b # 8B — stronger reasoning
ollama pull llama3.1:70b # 70B — requires ~40 GB RAM
ollama pull mistral # Mistral 7B
ollama pull gemma3 # Google Gemma 3
ollama pull phi4 # Microsoft Phi-4 14B
ollama pull qwen2.5:14b # Alibaba Qwen 2.5 14B
ollama pull nomic-embed-text # embedding modelThe tag after the colon specifies the variant. :latest is the default if no tag is specified.
Modelfile — custom system prompts
A Modelfile lets you package a model with a custom system prompt and parameters into a named local variant.
Modelfile
FROM llama3.2
SYSTEM """
You are a Python tutor. Explain concepts simply with
code examples. Never write pseudocode.
"""
PARAMETER temperature 0.3Build and run
ollama create python-tutor -f Modelfile
ollama run python-tutorNotes
Models live in ~/.ollama/models and they are big
A pulled 8B model is ~5 GB; 70B is ~40 GB. Set OLLAMA_MODELS to an external drive path before your first pull if your boot drive is small. Moving models after the fact works but requires re-pointing the env var and restarting the service.
Default context window is 2048 tokens — much smaller than the model supports
Even though Llama 3.1 supports 128k context, Ollama caps at num_ctx=2048 by default to save RAM. Override per request with "options": {"num_ctx": 8192} in the API body, or set it in a Modelfile with PARAMETER num_ctx 8192. Long-context summarisation will silently truncate without this.
Models unload after 5 minutes — control with keep_alive
First request after the model unloads pays a 5–30 second reload cost. For a chat app, pass "keep_alive": "30m" in requests, or set OLLAMA_KEEP_ALIVE=24h globally. To unload immediately, pass "keep_alive": 0.
Bind to 0.0.0.0 if you want LAN access
By default Ollama only listens on 127.0.0.1. To call it from another machine (Docker, phone, server), set OLLAMA_HOST=0.0.0.0:11434 and restart. There is no built-in auth — use a reverse proxy or SSH tunnel before exposing it beyond localhost.
Modelfiles are not Dockerfiles, they are layered manifests
A Modelfile change rebuilds only the changed layer — the base weights are not re-downloaded. Useful for iterating on system prompts. PARAMETER sets sampling defaults; TEMPLATE overrides the chat template; ADAPTER attaches a LoRA. See using Ollama with LangChain for the chain integration.
GPU offload is partial, not all-or-nothing
If a model is bigger than your VRAM, Ollama offloads as many layers as fit and runs the rest on CPU. ollama ps shows the split. To force all layers to GPU (and fail loudly if they do not fit), set PARAMETER num_gpu 999. To force CPU-only, set num_gpu 0.
Quick summary
- Install:
brew install ollamaon Mac; one-line script on Linux ollama run llama3.2— downloads and opens an interactive chat in one command- REST API at
localhost:11434— OpenAI-compatible endpoint available at/v1 - Python:
pip install ollamafor a clean native client - Modelfile: package a model + system prompt + parameters into a named local variant
Next, explore the open-source model landscape to see which models are available, then learn how to choose the right model for your specific task.
Ollama FAQ
Is Ollama free to use?
Yes. Ollama is completely free and open-source. You download models from the Ollama library at no cost and run them entirely on your own hardware with no API fees or usage limits.
What models can I run with Ollama?
Ollama supports hundreds of models including Llama 3, Mistral, Gemma 3, Phi-4, Qwen 2.5, CodeLlama, and DeepSeek. You can browse the full list with 'ollama list' or at the Ollama model library.
Does Ollama support GPU acceleration?
Yes. Ollama automatically uses Apple Silicon GPU on Mac, NVIDIA CUDA on Linux and Windows, and AMD ROCm on Linux. It also supports partial GPU offloading when the model does not fully fit in VRAM.
Can I use Ollama with Python?
Yes. Install the official client with 'pip install ollama' and call ollama.chat() or ollama.generate() from your code. Ollama also exposes an OpenAI-compatible API at localhost:11434/v1, so any OpenAI SDK client works too.
How do I create a custom model with Ollama?
Create a Modelfile that specifies a base model, system prompt, and parameters. Then run 'ollama create my-model -f Modelfile' to build it. You can run it with 'ollama run my-model' like any other model.