Overview
RAG costs divide into three categories: indexing (one-time or periodic), query-time inference (continuous), and infrastructure (monthly). Most teams underestimate indexing costs initially because they only embed a small pilot dataset, then experience sticker shock when scaling to production volume. The numbers in this lesson are representative but drift as providers change pricing — always verify against current published rates before building a cost model.
The single most impactful cost lever is the LLM choice. The difference between GPT-3.5 and GPT-4 for the generation step is roughly 20× in token cost; the quality difference for retrieval-grounded factual questions is much smaller — often 10–20% on faithfulness metrics. A common optimization is to route simple questions to a cheaper model and complex multi-step queries to a more capable one, reducing average cost per query while preserving quality for queries that need it.
The second biggest lever is caching. Embedding repeated queries costs the same as new ones. A Redis cache keyed on the normalized query string — with a 15-minute TTL for time-sensitive domains or a 24-hour TTL for stable knowledge bases — can eliminate 30–60% of embedding API calls in enterprise Q&A applications where employees ask similar questions repeatedly. Combined with LLM response caching for identical prompts, caching frequently cuts total RAG infrastructure cost by half.
Cost Breakdown Per Query
| Component | Cost (OpenAI) | Cost (Local) |
|---|---|---|
| Query Embedding | $0.00002 | $0 |
| Vector DB Search | $0.0001 (Pinecone) | $0 |
| LLM (GPT-4) | $0.015 | $0 |
| Total | $0.0151 | $0 |
For 1M queries/month: OpenAI = ~$15,000, Local = ~$0 (+ infra)
Vector DB Storage Costs
Storage Calculation:
1M documents × 400 tokens/doc × 3 chars/token = 1.2B chars
1.2B chars / 1M = 1.2M tokens of embeddings
Storage per model:
- 768 dimensions × 4 bytes = 3,072 bytes per vector
- 1,536 dimensions × 4 bytes = 6,144 bytes per vector
Cost with Pinecone:
- 1M vectors × 1536 dims = $100-300/monthModel Comparison
✓ Cost-Optimized
GPT-3.5 + Chroma (free) + local embeddings = ~$5K/month for 1M queries
Balanced
GPT-4 + Pinecone + OpenAI embeddings = ~$15K/month
High-Quality
Claude 3 Opus + Qdrant (self-hosted) + fine-tuned embeddings = ~$25K+/month
ROI Calculation
Example: Customer Support RAG
Setup Cost: $10K (infrastructure, training)
Monthly Cost: $5K (cloud services, embeddings)
Benefits:
- Reduce support tickets by 30%: $50K/month saved
- Reduce support team by 2 FTEs: $200K/year = $16.7K/month
- Improved CSAT (customer satisfaction): ~$5K/month value
Monthly ROI: ($50K + $16.7K + $5K) - $5K = $66.7K profit
Payback period: ~1.5 months
Annual ROI: 800%Notes
Storage costs scale with embedding dimension count
A 1536-dimension vector (OpenAI text-embedding-3-small) uses exactly 2× the storage of a 768-dimension vector (all-mpnet-base-v2). At 1M vectors, the difference is roughly 3 GB vs 6 GB. This matters for managed services like Pinecone that charge by vector count and dimension; switching to a smaller-dimension model can directly reduce monthly storage costs without necessarily sacrificing retrieval quality.
Cross-encoder reranking runs locally at near-zero marginal cost
The cost tables show LLM-based reranking. Cross-encoder models like cross-encoder/ms-marco-MiniLM or Cohere Rerank are separate inference tasks that run either locally (free beyond compute) or via API (~$1 per million queries for Cohere). Self-hosted cross-encoders add latency but eliminate per-query cost entirely — a good trade-off for high-volume applications where the monthly reranking bill would otherwise be significant.
Managed vector DB read and write costs are billed separately
Pinecone and similar managed services charge for both upserts (writes) and queries (reads). For pipelines with frequent document updates — live news ingestion, real-time product catalog sync — write costs can rival or exceed read costs. Model both in your cost projection before selecting a provider. Self-hosted Qdrant eliminates both write and read per-query costs at the expense of infrastructure management.
Caching the LLM response is often more impactful than caching embeddings
Embedding a query costs fractions of a cent. Generating a GPT-4 response costs $0.01–0.03. If 20% of your queries are repeats (common in enterprise Q&A), caching full LLM responses keyed on the exact query text saves far more than caching just the embedding. Use a two-level cache: embedding-level for approximate matches (semantic cache) and response-level for exact matches. Redis works well for both layers.
RAG Cost Breakdown FAQ
How much does it cost to run a RAG system?
Costs vary widely. Embedding 1 million tokens with OpenAI text-embedding-3-small costs ~$0.02. Vector DB storage on Pinecone starts at $70/month for 100K vectors. LLM generation is usually the largest cost at $1–30 per million output tokens.
How can I reduce RAG embedding costs?
Switch to a smaller embedding model (text-embedding-3-small vs text-embedding-3-large), cache embeddings for documents that don't change, and batch embed offline rather than at query time.
Are local vector databases cheaper than cloud ones?
Local vector databases (FAISS, Chroma, Qdrant self-hosted) have zero per-query cost but require compute infrastructure. Cloud-hosted options (Pinecone, Qdrant Cloud) trade infrastructure management for monthly fees.
How do I estimate RAG costs before building?
Calculate: (number of documents × avg tokens per chunk × embedding cost/token) for indexing, plus (monthly queries × avg tokens retrieved × LLM cost/token) for ongoing inference. Build a spreadsheet with these inputs.
Does retrieval quality affect RAG costs?
Yes — better retrieval means fewer tokens passed to the LLM per query, reducing generation costs. Reranking adds a small cross-encoder inference cost but pays off by shrinking the LLM context window needed.