What is quantization?
Model weights are stored as floating-point numbers. A full-precision model uses 16 or 32 bits per weight. Quantization reduces this to 4 or 8 bits — shrinking the model size by 50–75% with a small quality cost.
Mechanically, quantisation walks the weight matrices block by block, finds the min and max of each block, and stores each weight as an integer index into that range plus a small scale factor. At inference time the integers are dequantised on-the-fly back to FP16 (or directly multiplied in fused kernels) before matrix multiplication. The clever part of modern K-quants (the Q*_K_M family in GGUF) is that they use a mixture of bit widths within a single model — important layers like attention projections get more bits than feed-forward layers — so the average bits-per-weight is lower than the name suggests, with less quality loss than naive uniform quantisation.
Quantisation is what makes local inference economically viable. Without it, a 70B model needs 140 GB of memory and a multi-GPU server; with Q4_K_M it needs 40 GB and runs on a single workstation. The trade-off you are managing on every page below is the bits-per-weight vs quality vs speed triangle: lower bits means less memory and faster reads but more rounding error in every matmul.
GGUF — the format Ollama and LM Studio use
GGUF (GPT-Generated Unified Format) is the standard quantized model format for CPU and mixed CPU/GPU inference. It's what you get when you pull a model from Ollama or download via LM Studio.
| Variant | Bits/weight | Quality | Use when |
|---|---|---|---|
| Q2_K | ~2.6 bits | Notable degradation | Extreme RAM constraint only |
| Q4_K_M | ~4.8 bits | Good — recommended default | Most use cases on 8–16 GB |
| Q5_K_M | ~5.7 bits | Very good | When 16 GB RAM available, want better quality |
| Q8_0 | ~8.5 bits | Near-lossless | Maximum quality, RAM is not limiting |
| F16 | 16 bits | Full precision | Fine-tuning or benchmarking |
Practical default: Q4_K_M gives you 85–95% of full-precision quality at 30% of the memory. It's the right choice for almost every use case on consumer hardware.
GPTQ & AWQ — GPU-only formats
GPTQ and AWQ are quantization formats designed specifically for NVIDIA GPUs. They're faster than GGUF on GPU but don't support CPU fallback.
GGUF
- CPU + GPU (mixed)
- Used by Ollama, LM Studio
- Best for consumer hardware
GPTQ
- GPU only (NVIDIA)
- Used with HuggingFace
- Good speed on CUDA GPUs
AWQ
- GPU only (NVIDIA)
- Better quality than GPTQ at same size
- Preferred for CUDA deployments
If you're using Ollama or LM Studio, you're using GGUF. You only need to think about GPTQ/AWQ if you're running models directly through HuggingFace Transformers on a CUDA GPU.
GPU offloading
When a model is too large to fit entirely in VRAM, you can split it across GPU and CPU RAM — GPU handles as many layers as will fit; CPU handles the rest.
# Ollama automatically offloads as many layers as fit in VRAM
# To force a specific number of GPU layers:
OLLAMA_NUM_GPU=20 ollama run llama3.1:8b
# Check how many layers were offloaded:
ollama show llama3.1:8b --verboseEach layer offloaded to GPU significantly speeds up inference. Even partial GPU offload (e.g. 20 of 32 layers) can double token generation speed compared to full CPU inference.
CPU-only performance tips
No GPU? You can still run models effectively — here's how to get the most out of CPU inference.
Use Q4_K_M, not Q8
On CPU, memory bandwidth is the bottleneck. Q4 processes fewer bytes per token → faster inference. The quality difference between Q4 and Q8 is often smaller than the 2× speed difference.
Set thread count to physical cores only
Hyperthreaded (logical) cores can hurt LLM inference. Set threads to your physical core count:
OLLAMA_NUM_THREADS=8 ollama run llama3.2
Pick a smaller model over a slower large one
3B at 30 tokens/s is more usable than 13B at 3 tokens/s for interactive tasks. Batch processing tolerates slow speeds better.
Close memory-hungry apps
If the model doesn't fit entirely in RAM, the OS swaps to disk — reducing speed to unusable levels. Close browsers, IDEs, and other apps to free RAM before loading large models.
Notes
The Q4 → Q3 quality cliff is real; the Q5 → Q8 climb is not
Perplexity stays flat from Q8 down to about Q4_K_M (typically +1–3% over FP16), then degrades sharply below Q4. Q3_K_S and Q2_K models often produce visibly worse output — looser instruction following, broken JSON, off-topic tangents. If a model "feels dumb," check the tag before assuming it is the model itself.
IQ (importance-weighted) quants beat older quants at the same size
Newer GGUF variants tagged IQ4_XS, IQ3_M, etc. use an importance matrix derived from a calibration corpus to preserve the weights that matter most. They are slower per token than vanilla Q-quants on CPU but noticeably higher quality at the same byte count — especially valuable below 4 bits.
Apple Silicon has unified memory — VRAM rules do not apply
On M-series Macs, CPU and GPU share the same memory pool, so there is no offload split to manage. By default macOS caps the GPU at ~75% of total RAM; for inference on a 64 GB M2 Max you can lift this with sudo sysctl iogpu.wired_limit_mb=49152. Q4_K_M of a 70B model is genuinely usable on an M3 Max 64 GB once that cap is raised.
KV cache quantisation is the next lever after weight quantisation
For long contexts, the KV cache (per-token attention state) consumes more memory than the weights themselves. llama.cpp / Ollama support --kv-cache-type q8_0 or q4_0 to halve or quarter that cost. Quality impact is small (< 2% perplexity) for q8_0, larger for q4_0.
Speed depends on memory bandwidth, not just bit width
Token generation is memory-bandwidth bound — every generated token reads the entire model from RAM. A DDR4 laptop pushes ~25 GB/s, DDR5 ~50 GB/s, an M3 Max ~400 GB/s, a 4090 ~1000 GB/s. The same Q4 model runs ~16× faster on a 4090 than on DDR4 even with no compute bottleneck. If tokens-per-second feels off, check bandwidth before blaming the quant.
Do not quantise the model you fine-tune
Fine-tuning on quantised weights (QLoRA is the exception, designed for it) compounds rounding error and degrades the adapter. Train in FP16 or BF16, then quantise the merged result for deployment. The companion model landscape calls out which families publish FP16 vs only Q4 checkpoints.
Quick summary
- Quantization reduces bits-per-weight — Q4 uses ~30% of full-precision memory with ~5–10% quality loss
- Q4_K_M is the default choice — best quality/size tradeoff on consumer hardware
- GGUF (Ollama/LM Studio) works on CPU+GPU; GPTQ/AWQ are NVIDIA GPU-only
- GPU offloading splits layers — even partial offload gives significant speed gains
- CPU-only tips: Q4 over Q8, physical cores only, smaller models for interactive use
Not sure which model to quantize? Check the open-source model landscape for a comparison of Llama, Mistral, Gemma, Phi, and Qwen. Need help choosing the right size? See the model selection strategy guide.
Quantization FAQ
What is model quantization in LLMs?
Quantization reduces the precision of model weights from 16 or 32 bits down to 4 or 8 bits. This shrinks the model size by 50-75% with a small quality cost, letting you run larger models on consumer hardware.
What is the best quantization level for local LLMs?
Q4_K_M is the recommended default. It gives you 85-95% of full-precision quality at about 30% of the memory. For most use cases on 8-16 GB RAM machines, it is the best balance of quality and size.
What is the difference between GGUF and GPTQ?
GGUF is the standard format for CPU and mixed CPU/GPU inference, used by Ollama and LM Studio. GPTQ and AWQ are GPU-only formats that require an NVIDIA card with CUDA. GGUF is more flexible; GPTQ is faster on dedicated GPU setups.
Does GPU offloading help if my GPU has limited VRAM?
Yes. GPU offloading splits model layers between GPU and CPU. Even offloading 20 of 32 layers to the GPU can double token generation speed compared to full CPU inference. You do not need all layers in VRAM to benefit.
How can I speed up LLM inference on a CPU-only machine?
Use Q4 quantization instead of Q8 (fewer bytes per token means faster inference), set the thread count to your physical core count only (avoid hyperthreads), pick a smaller model for interactive tasks, and close memory-hungry apps to avoid disk swapping.