Popular Embedding Models
You have three options: expensive but best (OpenAI), fast and free (open-source), or custom (fine-tuned).
1️⃣ OpenAI Embeddings (Closed-source, proprietary)
Models:
text-embedding-3-small— 1536 dims, $0.02 per 1M tokens ⭐ RECOMMENDEDtext-embedding-3-large— 3072 dims, $0.13 per 1M tokens (overkill for most)
Pros: Best quality, multilingual, OpenAI keeps improving them, no setup.
Cons: Proprietary, cost per query, requires internet, data privacy concerns.
2️⃣ Open-Source Models (Free, run locally)
Popular models:
sentence-transformers/all-mpnet-base-v2— 768 dims, excellent quality (~$0)nomic-ai/nomic-embed-text-v1— 768 dims, very competitive (~$0)jinaai/jina-embeddings-v2-base-en— 768 dims, good for long docsbge-large-en— 1024 dims, strong Chinese/multilingual support
Pros: Free, private (no API calls), fast, easy to fine-tune.
Cons: Requires local infra, slower on CPU, less quality than OpenAI, self-hosted maintenance.
3️⃣ Fine-tuned Models (Custom, domain-specific)
Fine-tune open-source models on your domain data for perfect relevance.
When to use: If you have 1000+ labeled (query, relevant_doc) pairs and retrieval quality is critical.
🎯 Recommendation for most: Start with text-embedding-3-small (OpenAI). Cost is minimal (~$0.001 per document). If you need privacy or offline, use all-mpnet-base-v2.
Embedding Dimensions: Size vs Quality
Larger embeddings capture more nuance but cost more and require more storage. Smaller are fast but lose quality.
768 dimensions (most open-source models)
Sweet spot for general use. Good quality, fast, low storage.
1024 dimensions (some large models)
Better quality for nuanced retrieval. 33% more storage. 10-15% better accuracy.
1536 dimensions (OpenAI-3-small)
High quality. 2x storage cost vs 768. Best for complex domains.
3072 dimensions (OpenAI-3-large)
Highest quality but 4x cost and storage. Overkill for most uses.
Implementing Embeddings
Using OpenAI
from openai import OpenAI
client = OpenAI(api_key="sk-...")
# Single embedding
response = client.embeddings.create(
model="text-embedding-3-small",
input="The cat sat on the mat"
)
embedding = response.data[0].embedding
print(f"Embedding size: {len(embedding)}") # 1536
# Batch embeddings (cheaper, faster)
chunks = ["chunk 1", "chunk 2", "chunk 3"]
response = client.embeddings.create(
model="text-embedding-3-small",
input=chunks
)
embeddings = [d.embedding for d in response.data]Using Open-Source (Local)
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-mpnet-base-v2')
# Single
embedding = model.encode("The cat sat on the mat")
print(f"Embedding size: {len(embedding)}") # 768
# Batch (much faster)
chunks = ["chunk 1", "chunk 2", "chunk 3"]
embeddings = model.encode(chunks, batch_size=32)
# GPU support
embeddings = model.encode(
chunks,
device='cuda', # Use GPU if available
batch_size=32,
show_progress_bar=True
)Batch Processing & Scaling
Embedding 1 million documents one-at-a-time takes forever. Batch them for massive speedup.
from openai import OpenAI
from tqdm import tqdm
import time
client = OpenAI()
chunks = [...] # 1M chunks
# ❌ SLOW: One at a time
# for chunk in chunks:
# emb = client.embeddings.create(model="...", input=chunk)
# ✓ BETTER: Batch them
batch_size = 100 # OpenAI allows up to 2048 per request
embeddings = []
for i in tqdm(range(0, len(chunks), batch_size)):
batch = chunks[i:i+batch_size]
response = client.embeddings.create(
model="text-embedding-3-small",
input=batch
)
embeddings.extend([d.embedding for d in response.data])
time.sleep(1) # Rate limiting
print(f"Embedded {len(embeddings)} chunks")Batch Size Guidelines:
Batch up to 2048 inputs per request. Larger batches are cheaper per token.
Use 32-128 batch size depending on your GPU VRAM. Larger = faster.
OpenAI: ~3,500 requests/min for paid accounts. Space out batches with sleep().
Caching Embeddings
Don't re-embed the same chunk twice. Cache embeddings in your vector DB or use a hash-based cache.
import hashlib
import json
def get_embedding(text, model="text-embedding-3-small"):
"""Get embedding, checking cache first."""
text_hash = hashlib.md5(text.encode()).hexdigest()
# Check if already cached
cache_file = f"embeddings_cache/{text_hash}.json"
try:
with open(cache_file) as f:
return json.load(f)
except FileNotFoundError:
pass
# Generate new embedding
response = client.embeddings.create(
model=model,
input=text
)
embedding = response.data[0].embedding
# Cache it
with open(cache_file, 'w') as f:
json.dump(embedding, f)
return embedding💾 Better approach: Store embeddings in your vector DB directly. Once indexed, you never re-embed unless the chunk changes.
Notes
Never mix embedding models across the same index
If you embed documents with text-embedding-3-small and query with text-embedding-ada-002, cosine similarity scores become meaningless — the two models occupy completely different vector spaces. This is a silent failure: queries return results but they're semantically wrong. Lock the model name in a config constant and enforce it at both ingest and query time.
Batch your embedding API calls
OpenAI's embeddings endpoint accepts up to 2,048 inputs per request. Sending one document per call is 100× slower and exhausts rate limits quickly. Use LangChain's embed_documents() which batches automatically, or call the API directly with lists. For large corpora (> 100k chunks), use the Batch API to cut cost by 50%.
Dimension truncation is safe but has trade-offs
OpenAI's text-embedding-3 models support Matryoshka-style truncation — you can reduce from 1536 to 256 dimensions without re-training. Smaller dimensions cut storage and query latency significantly, with only minor accuracy loss on most tasks. Benchmark on your own queries before committing; accuracy loss is uneven across domains.
Cache embeddings keyed by content hash
For query-time embeddings, a simple LRU cache of (query_text → vector) eliminates redundant API calls when users ask similar questions repeatedly. For document embeddings, store vectors alongside content hashes so you can detect unchanged chunks and skip re-embedding during incremental refreshes. The savings compound quickly in long-running systems.
RAG Embeddings FAQ
What is the best embedding model for RAG?
OpenAI's text-embedding-3-small is the best cost-performance choice for cloud RAG. For local/private deployments, nomic-embed-text and bge-m3 from Hugging Face are top performers. Always evaluate on your own data with BEIR.
What embedding dimensions should I use for RAG?
OpenAI text-embedding-3-small supports 1536 dimensions (default) or truncated to 512/256. Smaller dimensions use less memory and are faster, with modest accuracy trade-offs. 768 dimensions is a common sweet spot.
Should I fine-tune embeddings for my RAG system?
Fine-tuning helps significantly when your domain has specialized vocabulary (legal, medical, code). Use the MTEB leaderboard to find a base model and fine-tune on 1,000–10,000 query-document pairs from your domain.
Can I use different embedding models for indexing and querying in RAG?
No — the embedding model for queries and documents must be identical, otherwise the vector spaces won't align and retrieval will fail. If you switch embedding models, you must re-embed and re-index all documents.
What is the cost of embedding documents for RAG?
OpenAI text-embedding-3-small costs $0.02 per million tokens. Embedding a 1,000-page PDF (~500K tokens) costs ~$0.01. Local models like sentence-transformers are free to run but require a GPU for reasonable speed.