Production RAG Architecture

Production Architecture Diagram

Python

┌─ Client Requests ─┐
│   API Gateway     │ ← Rate limiting, auth
└────────┬──────────┘
         │
┌────────▼──────────┐
│  Load Balancer    │ ← Distribute across servers
└────────┬──────────┘
         │
   ┌─────┴─────┐
   │           │
┌──▼──┐    ┌──▼──┐
│ RAG │    │ RAG │ ← Multiple replicas
│ Srv │    │ Srv │
└──┬──┘    └──┬──┘
   │           │
   └─────┬─────┘
         │
    ┌────┴─────┬──────────┬─────────┐
    │           │          │         │
┌───▼──┐  ┌────▼─┐  ┌────▼──┐  ┌──▼────┐
│Redis │  │Vector│  │LLM    │  │Logging│
│Cache │  │  DB  │  │Cache  │  │Service│
└──────┘  └──────┘  └───────┘  └───────┘

API Design

Python

from fastapi import FastAPI, HTTPException
from typing import Optional

app = FastAPI()

@app.post("/api/v1/query")
async def query(
    query: str,
    k: int = 5,
    user_id: Optional[str] = None,
    timeout: int = 30
) -> dict:
    """Query the RAG system."""
    try:
        # Rate limit
        if not rate_limiter.check(user_id):
            raise HTTPException(429, "Rate limited")

        # Check cache
        cache_key = f"{user_id}:{query}"
        if cached := redis.get(cache_key):
            return json.loads(cached)

        # Retrieve and generate
        result = await rag.query(query, k=k, timeout=timeout)

        # Cache result
        redis.setex(cache_key, 3600, json.dumps(result))

        return result
    except Exception as e:
        logger.error(f"Query failed: {e}")
        raise HTTPException(500, "Internal error")

Async Pipelines

Python

async def process_query(query: str) -> dict:
    """Async pipeline."""
    # Parallel execution
    tasks = [
        retrieve_documents(query),
        expand_query(query),
        log_query(query)
    ]
    retrieval, expansions, _ = await asyncio.gather(*tasks)

    # Stream LLM response
    response = await llm.generate_streaming(
        query,
        retrieval,
        stream=True
    )

    return response

Queuing Systems

Use Celery (RabbitMQ/Redis) for:

• Batch indexing (slow, non-blocking)

• Log aggregation

• Data refresh jobs

• Non-critical async work

Python

from celery import Celery

celery = Celery('rag')

@celery.task
def index_documents(doc_paths):
    """Background indexing job."""
    for path in doc_paths:
        rag.index(path)
    return f"Indexed {len(doc_paths)} docs"

# Queue job
result = index_documents.delay(["file1.pdf", "file2.pdf"])

Monitoring & Observability

Metrics to Track

Response latency, error rate, retrieval quality, cache hit rate, vector DB latency, LLM cost, token usage.

Python

from prometheus_client import Counter, Histogram

queries = Counter('rag_queries_total', 'Total queries')
latency = Histogram('rag_latency_seconds', 'Query latency')
cache_hits = Counter('rag_cache_hits', 'Cache hits')

@app.post("/query")
async def query(query: str):
    queries.inc()
    with latency.time():
        result = await rag.query(query)
    if result.from_cache:
        cache_hits.inc()
    return result

Notes

Semantic caching hits can mask stale knowledge

A semantic cache returns a stored answer when the new query is sufficiently similar to a cached one. If your document corpus was updated since the cache entry was created, users may receive outdated answers without knowing it. Always set a TTL on cache entries that is shorter than your document refresh cycle — for daily-refreshed corpora, a 12-hour TTL is reasonable.

Multi-tenant isolation requires namespace separation, not just filtering

Filtering by a tenant_id metadata field is convenient but risky — a buggy filter clause can expose cross-tenant data. Use hard namespace separation: separate vector collections per tenant in Qdrant, separate indices in Pinecone, or separate schemas in Weaviate. The storage overhead is worth the security guarantee in any multi-tenant product.

Async ingestion queues prevent API timeouts

Document ingestion (load → chunk → embed → index) takes seconds to minutes depending on document size. Doing it synchronously in an HTTP request will time out for large files. Use a task queue (Celery, ARQ, or cloud-native like Cloud Tasks) with a webhook or polling endpoint so the user gets an immediate acknowledgment and the ingestion happens in the background.

Audit logs are a compliance requirement, not a nice-to-have

In regulated industries (healthcare, finance, legal), every RAG query and every retrieved document must be logged with user identity, timestamp, and query text. Build structured audit logging into the retrieval layer from day one — retrofitting it later requires touching every query path. Store logs in an append-only sink (S3 + Athena, or a SIEM) separate from your application database.

Production RAG Architecture FAQ

How do I scale a RAG system to handle millions of queries?

Use a distributed vector database (Qdrant, Weaviate, or Milvus) with horizontal sharding. Put a semantic cache (e.g., GPTCache or a Redis similarity index) in front to serve repeated queries without re-embedding or re-generating.

What is semantic caching in RAG and how do I set it up?

Semantic caching stores previous query embeddings and their answers. When a new query arrives, it checks for a cached result with cosine similarity above a threshold (e.g., 0.95). On a cache hit it returns the stored answer instantly.

How do I handle multi-tenancy in a RAG system?

Use per-tenant namespaces in the vector database (Pinecone namespaces, Qdrant collections, or metadata filter on tenant_id). Never mix tenant documents in a single flat index without strict metadata filtering.

What async patterns work best for RAG ingestion at scale?

Use a message queue (Redis Streams, Kafka, or SQS) to decouple document upload from embedding and indexing. Worker processes consume from the queue, embed in batches, and upsert to the vector store asynchronously.

How do I handle RAG failures gracefully in production?

Implement circuit breakers for the vector DB and LLM API. When retrieval fails, fall back to a BM25 keyword search. When the LLM is unavailable, queue the request and notify the user of a delay rather than returning an error.