RAG Retrieval Mechanisms

1. Similarity Search (Dense Retrieval)

The standard approach: embed query, find most similar vectors.

Python

query = "How do I file taxes?"
query_embedding = embed(query)

# Search vector DB
results = vector_db.search(query_embedding, k=10)
# Returns: [(doc_id, score), ...] sorted by score

for doc_id, score in results:
    print(f"{doc_id}: {score:.3f}")

Pros: Fast, semantic, works across languages. Cons: Can retrieve unrelated docs with high semantic similarity.

2. Reranking with Cross-Encoders ⭐ IMPORTANT

After retrieving top-100 candidates, use a smarter model to rerank top-10. This dramatically improves quality.

Python

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/qnli-distilroberta-base")

# Get candidates from vector search
candidates = vector_db.search(query_embedding, k=100)

# Rerank
query_doc_pairs = [(query, doc.text) for doc in candidates]
scores = reranker.predict(query_doc_pairs)

# Sort by reranker score
reranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
top_10 = reranked[:10]

Result: Retrieving candidates (fast embedding search) then reranking (slow but accurate) is 10x better than pure embedding search.

3. Hybrid Search (Dense + Sparse)

Combine semantic search with BM25 keyword search. Keywords catch exact matches, dense embeddings find related concepts.

Python

# Dense search (semantic)
dense_results = vector_db.search(query_embedding, k=50)

# Sparse search (BM25 keyword)
sparse_results = bm25_index.search(query, k=50)

# Combine and rerank
combined = list(set(dense_results + sparse_results))
final_scores = reranker.predict([(query, doc) for doc in combined])
top_10 = sorted(final_scores)[:10]

Perfect for: Technical docs, FAQs, any domain where exact keywords matter.

4. Query Expansion & Reformulation

Rewrite vague queries to improve retrieval. Use LLM or template-based expansion.

Python

# LLM-based expansion
original_query = "How?"
prompt = f"""Expand this vague query into 3 specific variations:
Query: {original_query}

Variations:"""

expanded = llm.generate(prompt)
# Returns: "How do I file taxes?", "How do I claim deductions?", ...

# Search with all expanded queries
results = []
for expanded_q in expanded:
    results.extend(vector_db.search(embed(expanded_q), k=5))

# Deduplicate and rerank
unique_docs = list(set(results))
final = reranker.predict([(original_query, doc) for doc in unique_docs])

5. Smart Ranking & Context Ordering

How you order retrieved documents affects LLM output. Recent research shows position matters.

Recency Bias

LLMs pay more attention to early documents in context. Reorder by importance, not just similarity score.

Python

def rank_documents(docs, query, scores):
    """Rank by composite score: relevance + importance."""
    ranked = []
    for doc, score in zip(docs, scores):
        # Relevance: similarity score
        # Importance: doc quality, recency, popularity
        composite = (
            0.7 * score +  # Similarity
            0.2 * (doc.updated_at / now) +  # Recency
            0.1 * doc.popularity  # Citation count
        )
        ranked.append((doc, composite))
    return sorted(ranked, key=lambda x: -x[1])

6. Metadata Filtering

Filter documents before or after retrieval (permissions, category, date range).

Python

# Filter before search (faster)
results = vector_db.search(
    query_embedding,
    k=10,
    filter={
        "user_permissions": {"$in": ["PUBLIC", current_user_id]},
        "category": {"$eq": "tax_guide"},
        "date": {"$gte": "2024-01-01"}
    }
)

# Filter after search (simpler but slower)
all_results = vector_db.search(query_embedding, k=100)
filtered = [doc for doc in all_results
            if doc.user_id in current_user_permissions
            and doc.category == "tax_guide"
            and doc.date >= "2024-01-01"][:10]

Notes

Hybrid search alpha tuning requires real query data

The alpha parameter controlling the dense/sparse balance in hybrid search is often set to 0.5 as a default. This is rarely optimal. Keyword-heavy domains (legal, regulatory text, product SKUs) need higher BM25 weight (alpha < 0.5); semantic domains (FAQ, knowledge base, general Q&A) do better with more dense weight (alpha > 0.7). Tune alpha empirically on a held-out query set, not based on intuition.

Multi-query expansion costs extra but pays off on vague queries

Multi-query retrieval generates 3–5 reformulations of the user's question and retrieves for all of them. This is effective when users ask short, ambiguous queries ("how does billing work?") but adds N× embedding API calls and N× vector search latency. Gate it behind a query classifier — apply only to short queries or queries that returned zero results on the first pass.

ANN index parameters affect recall non-linearly

HNSW's ef_search parameter controls recall vs. latency. Doubling ef_search from 64 to 128 might gain 1–2% recall at 2× latency cost — diminishing returns set in quickly. Profile your actual workload and set the smallest ef_search that meets your recall target; don't just leave it at the library default.

MMR is slow on large result sets

Maximal Marginal Relevance re-ranks by computing pairwise similarity between all candidate chunks — an O(n²) operation. With a candidate pool of 100 chunks it's imperceptible, but at 500+ candidates it noticeably slows retrieval. Reduce your initial ANN retrieval pool size before applying MMR, or apply it only on the top-20 pre-ranked candidates rather than all results.

RAG Retrieval Mechanisms FAQ

What is dense retrieval in RAG?

Dense retrieval embeds both the query and documents into continuous vectors, then finds nearest neighbors using approximate nearest neighbor (ANN) search. It captures semantic meaning but can miss exact keyword matches.

What is BM25 sparse retrieval and when should I use it?

BM25 is a term-frequency-based ranking algorithm that excels at exact keyword matching. Use it when users query with specific product names, error codes, or terminology that dense embeddings may not capture as exact matches.

What is hybrid search in RAG?

Hybrid search runs both dense (vector) and sparse (BM25) retrievers and merges their results using Reciprocal Rank Fusion (RRF) or a learned score combiner. It outperforms either approach alone on most benchmarks.

What is Maximum Marginal Relevance (MMR) in RAG retrieval?

MMR selects retrieved documents to be both relevant to the query and diverse from each other. This prevents the LLM prompt from being filled with near-duplicate chunks that all say the same thing.

What is multi-query retrieval expansion in RAG?

Multi-query expansion uses the LLM to rephrase the original query into 3-5 variants, retrieves results for each, and merges them. It improves recall for ambiguous or under-specified queries by covering multiple phrasings.