Why model selection is the highest-leverage decision
The most common mistake when shipping a local LLM feature is treating model choice as interchangeable — picking whatever was at the top of the leaderboard last week and moving on. In practice, the right model for your task can be ten times cheaper to run, two times faster, and noticeably more accurate than the wrong one of the same size. A 3B model handling a templated extraction job will beat a 70B general chat model on every dimension that matters: latency, cost, predictability, and very often quality. The selection decision compounds because it constrains everything downstream — prompt structure, evaluation harness, retry policy, even what you can promise users.
This page walks through the four selection axes that actually move the needle: model type (chat / embedding / code / reasoning), size (1B–70B and the realistic latency at each tier), a 4-step decision framework for working through the tradeoffs, and honest limits on when local is not the right answer. If you have not already, the companion open-source model landscape covers the specific families you will be choosing between.
Model types — not all models are the same
The first selection question is model type — what the model was trained and fine-tuned to do. Using the wrong type for the job produces poor results regardless of size. The four buckets below have meaningfully different architectures, training objectives, and inference profiles, and they are not substitutes for each other.
Chat / Instruction models
Fine-tuned to follow instructions and have conversations. These are what you use for most applications — Q&A, summarisation, coding assistance, analysis.
Examples: llama3.1:8b, mistral:7b, phi4:14b
Embedding models
Convert text into vectors for semantic search and RAG. Do NOT use a chat model for embeddings — dedicated embedding models produce far better vectors and are much faster.
Examples: nomic-embed-text, mxbai-embed-large, all-minilm
Code-specialised models
Fine-tuned on large code corpora. Outperform general chat models on code generation, debugging, and code explanation tasks — especially at smaller sizes.
Examples: qwen2.5-coder:7b, deepseek-coder-v2, codellama:13b
Reasoning models
Train the model to "think before answering" using extended reasoning chains. Slower and more expensive per call, but significantly better on multi-step logic, maths, and complex problem-solving.
Examples: qwq:32b, deepseek-r1:14b
Size vs quality tradeoffs
Bigger is not always better — especially locally. The right size depends on your task, hardware, and latency requirements.
| Size | Speed | Quality | Best for |
|---|---|---|---|
| 1B–3B | Very fast (>30 t/s) | Limited reasoning | Classification, extraction, edge devices |
| 7B–8B | Fast (15–30 t/s) | Good for most tasks | Everyday assistant, RAG, summarisation |
| 12B–14B | Medium (8–15 t/s) | Strong reasoning | Code, analysis, complex instructions |
| 30B–70B | Slow (2–8 t/s) | Near GPT-4 on many tasks | High-quality production use |
Decision framework
What type of model do I need?
Chat → instruction model. Semantic search/RAG → embedding model. Coding → code-specialised. Hard reasoning → reasoning model.
How much RAM do I have?
8 GB → 7B (Q4). 16 GB → 13B (Q4) or 7B (Q8). 32 GB → 30B (Q4) or 13B (Q8). 64 GB → 70B (Q4).
Does latency matter?
Interactive chat → smaller model. Batch processing → size up for quality. Streaming UI → test token/s for your target model.
Evaluate, don't guess
Run your actual prompts on two candidate models. Compare outputs on 10–20 real examples before committing. Benchmarks don't always reflect your specific task.
When local models are NOT enough
Local models are powerful, but they have real ceiling — knowing when to reach for a cloud API saves you from wasted effort.
Complex multi-step reasoning
If your task requires legal analysis, medical diagnosis, financial modelling, or deep strategic reasoning — smaller local models still fall significantly behind GPT-4o and Claude 3.5 Sonnet. Even 70B models have gaps here.
Very long contexts
Processing 100k+ token documents locally is slow and often produces poorer results than cloud models. Cloud APIs handle long contexts more gracefully at current model sizes.
Production scale with latency SLAs
Serving hundreds of concurrent users from a local machine is not feasible. Cloud APIs are infinitely horizontally scalable; local inference is not.
Multimodal tasks
GPT-4o and Claude 3.5 are significantly better at image understanding than most locally available vision models. For vision-heavy applications, cloud models are still ahead.
Notes
Quantisation level matters more than people assume
A Q8 7B model is often a closer match to its full-precision parent than a Q4 13B is. If you have spare RAM, prefer Q8 of a smaller model over Q4 of a larger one — quality degrades more sharply below Q4 than people assume. See the quantization chapter for measured numbers.
Instruct / chat variants are not always the right choice
For classification, extraction, or templated rewrites, the base (non-instruct) model with a few-shot prompt is often more reliable and significantly cheaper than the chat-tuned variant. Chat-tuning adds verbosity, hedging, and refusals that hurt structured output tasks.
Test on YOUR prompts, not benchmarks
Public benchmarks are partially contaminated and rarely reflect a real workload. Build a 20-prompt eval set from your actual product before picking a model — a candidate that scores 5 MMLU points higher but fails on half your prompts is the wrong call.
Concurrency collapses local throughput
A model that runs at 30 tokens/sec for one user runs at 5 tokens/sec when four users hit it simultaneously — local inference engines do not batch the way cloud providers do. If you need to serve multiple concurrent users from one machine, plan for serialised queuing or move to a cloud API.
Embedding dimensions affect storage cost downstream
A 768-dim embedding (nomic-embed-text) costs 4× less in vector store storage and ANN search than a 3072-dim embedding (mxbai-embed-large). Pick the smallest embedding that meets your retrieval quality bar — the difference at scale is substantial.
Quick summary
- Choose the right type first: chat / embedding / code / reasoning — wrong type beats wrong size
- 7B–8B is the practical sweet spot for 16 GB RAM; 13B–14B for higher quality
- Use dedicated embedding models for RAG — never a chat model for embeddings
- Start small, step up only if quality is insufficient on your actual prompts
- Go cloud for: complex reasoning, 100k+ contexts, production scale, advanced vision
Need to understand what models are available? See the open-source model landscape. Want to reduce model size without losing quality? Read about quantization and performance tuning.
Model Selection FAQ
Should I use a chat model or an embedding model for RAG?
Use both. Embedding models (like nomic-embed-text) convert text into vectors for semantic search. Chat models (like Llama 3.1 8B) generate answers from the retrieved context. Never use a chat model for embeddings — dedicated embedding models produce far better vectors.
What size LLM should I run on 16 GB RAM?
On 16 GB RAM, 7B-14B parameter models at Q4 quantization are the sweet spot. Llama 3.1 8B and Phi-4 14B both fit comfortably with room for your OS and applications.
When should I use a cloud API instead of a local LLM?
Use cloud APIs for complex multi-step reasoning, processing documents over 100k tokens, production workloads with latency SLAs, or advanced vision tasks. Local models excel at development, prototyping, and privacy-sensitive tasks.
Is a bigger model always better for local inference?
No. A 7B model at 30 tokens per second is more usable for interactive tasks than a 13B model at 3 tokens per second. Start with the smallest model that meets your quality bar, and only step up if outputs are insufficient on your actual prompts.
Which local model is best for reasoning and math tasks?
For reasoning-heavy tasks, use dedicated reasoning models like QwQ 32B (needs 32 GB+ RAM) or DeepSeek-R1 14B (fits in 16 GB). These models use extended chain-of-thought and significantly outperform general chat models on logic and math.