RAG Core Concepts Explained

Overview

Retrieval-Augmented Generation works by converting "the LLM doesn't know the answer" into "find the document that contains the answer and give it to the LLM." This requires three core capabilities: a way to represent text as searchable vectors (embeddings), a way to efficiently find the most similar vectors across millions of candidates (vector search), and a way to present the found text to an LLM so it can produce a grounded, citeable answer. Understanding each component deeply determines how well you can diagnose and improve a RAG system when it fails.

The concepts here are foundational to every other lesson in this series. Reranking becomes obvious once you understand why cosine similarity over bi-encoder embeddings is inherently less precise than a joint cross-encoder score. Wrong document retrieval becomes diagnosable once you understand HNSW index recall parameters. Inconsistent LLM answers become traceable once you understand where non-determinism enters the query-to-prompt pipeline.

This lesson covers the four essential building blocks in sequence: how text becomes a vector (Section 1), how to measure similarity between vectors (Section 2), how vector databases find nearest neighbors at scale without comparing every vector (Section 3), and how the indexing and querying phases fit together into the full RAG architecture (Section 4). The final two sections explain why RAG works and where it still fails, so you can set accurate expectations before building.

1. Embeddings: Text as Vectors

An embedding is a numerical representation of text. Instead of storing "The cat sat on the mat" as a string, we convert it to a list of numbers that capture its meaning. For a quick reference on model selection, distance metrics, and batching costs see the Embeddings cheatsheet.

Python

Text: "The cat sat on the mat"
↓
Embedding: [0.23, -0.41, 0.15, 0.89, ..., 0.05]
           (1536 numbers for OpenAI's text-embedding-3-small)

Why embeddings? Because embeddings capture semantic meaning. Similar texts have similar embeddings, even if they use different words.

Example: "The cat sat on the mat" and "A feline rested on the carpet" have different words but similar embeddings because they mean the same thing. This is why embeddings enable semantic search.

Key Properties:

Fixed dimension: Always the same length (e.g., 1536). You can compute distances.

Semantic: Similar meanings → similar vectors. "Dog" is closer to "cat" than to "pizza".

Language-agnostic: Multilingual embeddings work across languages.

Deterministic: Same text always produces the same embedding (within the same model).

2. Vector Similarity: Finding Neighbors

Once you have vectors, you can measure how similar two pieces of text are by computing the distance between their embeddings.

Cosine Similarity (Most Common)

Measures the angle between two vectors. Range: -1 to 1 (but typically 0 to 1 for text).

Python

similarity = (A · B) / (||A|| × ||B||)

Example:
Text A: "Python programming language"
Text B: "Python is great for coding"
Cosine Similarity: 0.92 (very similar!)

Text C: "What's the weather?"
Cosine Similarity to A: 0.15 (not similar)

Euclidean Distance (L2)

Straight-line distance between vectors. Lower is more similar. Slower but works well for some use cases.

Manhattan Distance (L1)

Sum of absolute differences. Used for sparse vectors. Faster but less accurate for dense embeddings.

For RAG: Use cosine similarity. It's the standard, fast, and works best with normalized embeddings. Most vector databases default to it.

3. Vector Databases: Searching at Scale

Brute-force similarity search is slow. With a million documents, comparing your query to every embedding takes seconds. Vector databases use smart indexing to search in milliseconds.

How Vector DBs Work:

1. Indexing: When you add documents, embeddings are indexed using algorithms like:

HNSW (Hierarchical Navigable Small World) — Fast, approximate, most popular
IVF (Inverted File) — Clusters vectors, faster for large datasets
LSH (Locality Sensitive Hashing) — For approximate nearest neighbors

2. Querying: When you search, the DB uses the index to find similar vectors without comparing all of them.

Python

query_embedding = embed("How do I use Python?")
# Instead of comparing to 1M docs,
# find ~100 candidates using index
# Then return top-10 by similarity
top_10_docs = vector_db.search(query_embedding, k=10)

Speed tradeoff: Approximate search (HNSW) is much faster but might miss the truly best match. Exact search is slow. RAG typically uses approximate.

4. The RAG Architecture

Now let's see how all these pieces fit together:

Phase 1: Indexing (Offline, happens once or periodically)

Python

Step 1: Load Documents
documents = ["tax_guide.pdf", "FAQ.html", "policies.txt", ...]

Step 2: Chunk Them
chunks = [
  "A tax deduction is...",
  "Common deductions include...",
  "Self-employed people can...",
  ...
]  # Each chunk ~500 tokens

Step 3: Embed Chunks
embeddings = [
  [0.23, -0.41, 0.15, ...],  # for chunk 1
  [0.19, -0.38, 0.22, ...],  # for chunk 2
  ...
]

Step 4: Store in Vector DB
vector_db.add(chunks, embeddings)
# Now ready for search!

Phase 2: Querying (Online, happens per request)

Python

Step 1: Embed Query
user_query = "How do I maximize tax deductions?"
query_embedding = embed(user_query)

Step 2: Search Vector DB
similar_chunks = vector_db.search(query_embedding, k=5)
# Returns top-5 most similar chunks with scores

Step 3: Build Prompt
prompt = f"""
Answer the question using these documents:

{similar_chunks}

Question: {user_query}
Answer:
"""

Step 4: Call LLM
answer = llm.generate(prompt)
# LLM now has ground truth to answer from

Key insight: Indexing happens once (or periodically). Querying happens for every user request. Both are critical, but retrieval speed matters more for users.

Why RAG Works

RAG solves LLM hallucinations by providing grounding — anchoring answers in real documents. The LLM works better with context.

Without RAG (Hallucination Risk)

Text

User: "What's the tax deduction limit for 2024?"
LLM: "The limit is $15,000... (confidently wrong)"

Problem: Model was trained on 2023 data, guesses about 2024.

With RAG (Grounded Answer)

Text

User: "What's the tax deduction limit for 2024?"
Retrieved: "2024 Tax Deduction Limits: $14,600 for single..."
LLM: "The limit for 2024 is $14,600. (Based on retrieved docs)"

Result: Accurate, up-to-date, cited sources.

RAG Tradeoffs

✓ Advantages

• Current information: Works with today's data, no retraining
• Sourceable: Cite which documents the answer came from
• Scalable: Add 1M documents without model changes
• Cost-effective: No expensive fine-tuning
• Reduces hallucinations: Answers grounded in facts

❌ Limitations

• Imperfect retrieval: Wrong documents retrieved first
• Latency: Search + LLM generation takes ~500-2000ms
• Context limits: LLM can only see top-K documents
• Cost per query: Embedding + LLM calls add up
• Still hallucinates: LLM invents facts despite good docs

Notes

Embedding models produce incompatible vector spaces

You cannot mix embeddings from different models in the same index. OpenAI text-embedding-3-small and text-embedding-3-large live in different 1536-dimensional spaces — documents embedded with one model cannot be retrieved by queries embedded with the other. If you switch embedding models, you must re-embed your entire corpus. Plan embedding model upgrades as full re-index events.

Cosine similarity equals dot product on L2-normalized vectors

Most embedding APIs return L2-normalized vectors (magnitude = 1). On normalized vectors, cosine similarity and dot product are mathematically equivalent, so "cosine" and "dot product" distance metrics in vector DB settings produce identical results. Verify whether your chosen embedding model normalizes its output — if it does not, use cosine similarity explicitly or normalize yourself before inserting.

HNSW recall is tunable at query time without rebuilding the index

The ef parameter (search beam width) in HNSW controls the recall-speed trade-off at query time. Higher ef = more candidates explored = higher recall but slower search. You can tune this dynamically without rebuilding the index, which means you can tighten the trade-off based on query priority: fast recall for autocomplete, high recall for research queries. Qdrant exposes this as hnsw_ef in the search parameters.

Increasing top-K improves recall but raises LLM cost linearly

Retrieving top-10 instead of top-5 roughly doubles the tokens sent to the LLM per query. The marginal quality gain follows a diminishing returns curve: the 6th through 10th chunks are less likely to contain the answer than the first 5. A common production pattern is to retrieve top-20, rerank, then pass only the top 3–5 to the LLM — gaining the recall benefit of a wide first-stage retrieval without the cost of a large LLM context.

RAG Core Concepts FAQ

What is an embedding in RAG?

An embedding is a fixed-length numerical vector that represents the semantic meaning of a piece of text. Similar texts have vectors that are close together in high-dimensional space, which is how RAG finds relevant documents.

What is cosine similarity in vector search?

Cosine similarity measures the angle between two vectors, ignoring their magnitude. A score of 1.0 means identical direction (semantically identical); 0 means orthogonal (unrelated). Most RAG systems use it to rank retrieved chunks.

What is the difference between dense and sparse retrieval?

Dense retrieval uses embedding vectors (semantic similarity); sparse retrieval uses term frequency (BM25/keyword matching). Dense retrieval understands paraphrases; sparse excels at exact keyword matches.

What is reranking in a RAG pipeline?

Reranking re-scores the initial retrieval results using a more accurate (but slower) cross-encoder model. It is applied after fast ANN search to push the most relevant chunks to the top before passing them to the LLM.

How many documents should I retrieve per RAG query?

A common default is top-K=5. Retrieve 10–20 with a fast retriever, then rerank and keep the top 3–5 for the LLM context. More chunks = more cost; fewer = risk of missing the answer.