Why run LLMs locally?
Privacy
Your prompts and data never leave your machine. No cloud logging, no training on your inputs, no terms-of-service risk for sensitive data.
Zero cost per call
Once the model is downloaded, inference is free. No API bills, no rate limits, no quota exhaustion during development.
Works offline
No internet connection required after download. Run models on a plane, in a restricted network, or in an air-gapped environment.
Experiment freely
Try different models, quantizations, and system prompts without worrying about cost. Rapid iteration is much cheaper locally.
Hardware requirements
The bottleneck for local LLMs is RAM (or VRAM for GPU inference). The model weights must fit in memory during inference — the more parameters, the more memory required.
| RAM / VRAM | Models you can run | Expected quality |
|---|---|---|
| 8 GB | 1B–3B models (Q4), 7B models (Q4 tight) | Good for simple tasks, limited reasoning |
| 16 GB | 7B–8B models (Q4/Q8), 13B (Q4) | Solid for most everyday tasks |
| 32 GB | 13B–14B (Q8), 30B (Q4) | Strong reasoning, near GPT-3.5 level |
| 64 GB+ | 70B models (Q4), 34B (Q8) | GPT-4 class performance on many tasks |
The main tools
Ollama — recommended
CLI + REST API. One command to download and run any model. Works on macOS, Linux, and Windows. The fastest way to get started.
LM Studio
GUI app for discovering, downloading, and running models. Exposes a local OpenAI-compatible API. Best for users who prefer not to use the command line.
llama.cpp
The inference engine that powers both Ollama and LM Studio under the hood. Direct access for advanced use cases — most users don't need this.
Series overview
Introduction ← You are here
Why run locally, hardware requirements, and an overview of the main tools.
Ollama
Install, pull models, ollama run, REST API, and the model library.
Open-source Model Landscape
Llama 3, Mistral, Gemma, Phi-3, Qwen — size vs quality tradeoffs and use-case guide.
Model Selection Strategy
Chat vs embedding vs coding models, small vs large tradeoffs, when local is NOT enough.
LM Studio
GUI setup, model download, local OpenAI-compatible server — ideal for non-technical users.
Quantization & Performance
GGUF vs GPTQ, Q4 vs Q8, GPU offloading, and CPU-only performance tips.
Limitations of Local Models
Hallucination rates, weak reasoning in small models, context limits, maintenance overhead.
Local Models with LangChain
ChatOllama, OpenAI-compatible endpoint, drop-in replacement for GPT in your chains.
Local RAG Pipeline
End-to-end RAG with local embeddings, hybrid search, reranking, and Docker deployment.
Common beginner mistakes
The same handful of mistakes account for most of the "local LLMs are too slow / too dumb" frustration online. Avoid these and you skip months of trial and error.
1. Choosing a model bigger than your RAM
A 13B Q4 model needs ~8 GB of free memory at minimum, plus headroom for KV cache. If you load a model that almost fits, your OS swaps to disk and tokens/sec collapses by 10–50×. Pick a smaller variant or a tighter quantization before blaming the tool.
2. Comparing a 7B local model to GPT-4o
A 7B model has roughly 5% the parameters of frontier cloud models. Judging it against GPT-4o on reasoning tasks is unfair. Compare 7B local against GPT-3.5-class targets; compare 70B local against GPT-4-class. Match the weight class.
3. Forgetting the context window costs memory too
Long prompts eat KV-cache RAM on top of the model weights. A 32k-token context on Llama 3.1 can consume 4+ GB by itself. Set num_ctx to what you actually need, not the model's maximum.
4. Skipping the system prompt
Small models are more sensitive to instructions than large ones. A clear, specific system prompt is the single highest-leverage change you can make — often more impactful than swapping models.
5. Running CPU-only when a small GPU is available
Even 4 GB of dedicated VRAM can offload key layers and 2–4× throughput. Check whether your inference engine is actually using the GPU — Ollama and LM Studio both expose this in their status output.
What you'll be able to build by the end
This 9-part series is structured so each page builds on the previous one. By the end you'll have a working, private document Q&A system running entirely on your laptop.
By page 4
You understand the open-source model landscape (Llama, Mistral, Gemma, Phi, Qwen), can pick the right model for chat vs embedding vs coding, and know when a local model is genuinely insufficient.
By page 6
You can read quantization labels (Q4_K_M, Q8_0, GGUF, GPTQ), pick the right one for your hardware, and tune GPU offload settings for maximum throughput.
By page 8
You can wire local models into LangChain via ChatOllama, swap between local and cloud with one environment variable, and run structured-output extraction on local weights.
By page 9
You have a complete local RAG pipeline — PDF ingestion, ChromaDB vector store, hybrid retrieval, and a Docker Compose deployment. Fully offline, fully private.
Run LLMs Locally FAQ
Can I run LLMs locally for free?
Yes. Tools like Ollama and LM Studio let you download and run open-source models such as Llama 3 and Mistral on your own machine at zero cost after the initial download.
How much RAM do I need to run a local LLM?
8 GB of RAM handles 7B-parameter models at Q4 quantization. 16 GB is the sweet spot for everyday use, running 7B–13B models comfortably. 32 GB+ lets you run 30B–70B models.
Is Apple Silicon good for running local LLMs?
Apple Silicon is currently the best consumer hardware for local LLM inference. Its unified memory architecture lets CPU and GPU share the same RAM pool, so an M2 Pro with 16 GB runs 7B models very well.
What is the difference between Ollama and LM Studio?
Ollama is a CLI tool ideal for scripting and server deployments. LM Studio is a GUI desktop app for visually discovering, downloading, and chatting with models. Both expose OpenAI-compatible APIs.
Do local LLMs work offline?
Yes. Once you download a model, inference runs entirely on your machine with no internet connection required. This makes local LLMs ideal for air-gapped environments and offline development.
Are local LLMs as good as GPT-4 or Claude?
Smaller local models (7B–13B) are weaker on complex reasoning and factual recall compared to frontier cloud models. However, 70B local models approach GPT-4 quality on many tasks, and for focused use cases like text classification or summarization, even 7B models perform well.
Continue the series
Ready to start running models? Head to the Ollama setup guide for a step-by-step walkthrough. If you prefer a GUI, skip ahead to the LM Studio tutorial. For a real-world project, jump to building a local RAG pipeline. Browse all guides in our tutorial catalog.
Quick summary
- Local LLMs offer privacy, zero API cost, offline access, and free experimentation
- 8 GB RAM handles 7B models; 16–32 GB is the sweet spot for quality and speed
- Apple Silicon unified memory makes it the best consumer hardware for local inference
- Ollama (CLI) and LM Studio (GUI) are the two recommended entry points