RAG Chunking Strategies

Q: What chunk size should I use for RAG?

Start with 512 tokens and 64-token overlap as a baseline. Smaller chunks (256 tokens) improve precision for factual Q&A; larger chunks (1024 tokens) work better when the answer requires broad context.

Q: What is semantic chunking in RAG?

Semantic chunking splits documents at natural semantic boundaries — paragraph or topic changes — rather than at fixed token counts. This keeps related ideas together and improves retrieval coherence.

Q: What is sliding window chunking?

Sliding window chunking moves a fixed-size window across the document with overlap (e.g., 512 tokens, 64-token step). Overlap ensures no sentence is split across chunk boundaries without context.

Q: How does chunk overlap affect RAG quality?

Overlap preserves cross-boundary context so that sentences at the end of one chunk and the start of the next are both retrievable. Too little overlap loses context; too much wastes tokens and increases storage.

Q: Should I chunk by sentence, paragraph, or page?

Chunking by paragraph or semantic section typically outperforms sentence-level chunks because LLMs need surrounding context to generate accurate answers. Page-level chunks are too large for precise retrieval.

Overview

Chunking is the step that converts raw documents into retrieval units. The choice of chunk size, chunking method, and overlap strategy directly controls three things that determine RAG quality: retrieval precision (does the right chunk rank first?), context density (does each chunk contain enough context to be useful on its own?), and cost (larger chunks cost more to embed, store, and pass to the LLM). Getting chunking wrong is the single most common reason RAG systems fail in production.

The four main strategies — fixed-size character splitting, recursive character splitting, token-based splitting, and semantic splitting — are not interchangeable. Fixed-size is fast but naïve; recursive is the safe default for most text documents because it respects natural boundaries; token-based is necessary when you need to guarantee no chunk exceeds a model's context limit; semantic splitting is slower but produces coherent topic boundaries. The right choice depends on your document type, your LLM, and how much ingest latency you can tolerate.

The most impactful thing you can do after reading this lesson is to generate 20–30 chunks from a representative document in your corpus and read them manually. That single exercise will surface broken sentences, mixed topics, and dangling references that automated metrics miss entirely. Good chunking should produce units that make sense when read in isolation.

Why Chunking Matters

Chunking is the most impactful decision in RAG. Get it wrong and you'll retrieve irrelevant documents or miss important context.

❌ Too Large (5000+ tokens)

Chunks contain too many topics. Query "How do I file taxes?" might retrieve a 10-page document. LLM spends tokens on irrelevant content.

❌ Too Small (<50 tokens)

Chunks lose context. "Self-employed people can..." (alone) is confusing without the full deduction explanation.

✓ Just Right (200-500 tokens)

Focused on single topic. Enough context to understand alone. Fits in context window with multiple retrievals.

🎯 Sweet spot: 256-512 tokens (for most use cases). Adjust based on your LLM's context window and retrieval requirements.

Chunking Methods

1. Fixed-Size Character Splitting

How: Split every N characters (e.g., 1000 chars = ~200 tokens).

Text

text = "Long document text..."
chunk_size = 1000  # characters
chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]

❌ Problem: Chunks may break mid-sentence. Hard to predict actual token count.

2. Recursive Character Splitting (Recommended)

How: Split on delimiters in order (paragraph, sentence, word) until chunk reaches target size.

Python

from langchain.text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_text(document_text)

✓ Advantage: Preserves logical boundaries (paragraphs, sentences). Most reliable.

3. Token-Based Splitting ⭐ BEST PRACTICE

How: Count actual tokens using the model's tokenizer. Split when tokens reach limit.

Python

import tiktoken
from langchain.text_splitters import TokenTextSplitter

# For GPT models
splitter = TokenTextSplitter(
    chunk_size=400,      # tokens, not characters
    chunk_overlap=20,    # tokens
    encoding_name="cl100k_base"  # GPT-4 encoding
)
chunks = splitter.split_text(text)

# Verify token count
enc = tiktoken.encoding_for_model("gpt-4")
for chunk in chunks:
    token_count = len(enc.encode(chunk))
    print(f"Chunk: {token_count} tokens")

✓ Why: Exact token counts. No surprises with context limits. Model-specific.

4. Semantic Chunking

How: Use embeddings to detect when topic changes. Split there, not by size.

Python

from semantic_chunkers import StatisticalChunker

chunker = StatisticalChunker(
    encoder="openai",  # or "huggingface"
)
chunks = chunker.split_text(document_text)
# Chunks vary in size based on topic coherence

⚠️ Tradeoff: Better quality but slower (requires embedding during chunking). Use for critical documents.

Model-Specific Chunk Sizes

Different LLMs have different strengths. Adjust chunk size accordingly:

Model	Context Window	Recommended Chunk	Max Retrievals
GPT-4o	128K tokens	400-500 tokens	10-15 chunks
GPT-3.5-turbo	16K tokens	250-300 tokens	5-8 chunks
Claude 3 Opus	200K tokens	600-800 tokens	20-25 chunks
Llama 2 (7B)	4K tokens	200-256 tokens	3-4 chunks
Mixtral 8x7B	32K tokens	400-500 tokens	8-10 chunks

Formula: Max chunk size = (context_window × 0.6) / max_retrievals

This reserves ~40% of context for prompt, system message, and LLM's response.

Overlap: Context Preservation

Chunks are split independently, but important context might fall on boundaries. Overlap solves this.

Example:

Python

Text: "Tax deductions are allowed. Common examples include mortgages,
charitable donations, and medical expenses. Self-employed..."

WITHOUT overlap (chunk_overlap=0):
Chunk 1: "Tax deductions are allowed. Common examples include mortgages,
charitable donations, and medical expenses."
Chunk 2: "Self-employed..."  ← Lost context about deductions!

WITH overlap (chunk_overlap=30):
Chunk 1: "Tax deductions are allowed. Common examples include mortgages,
charitable donations, and medical expenses."
Chunk 2: "...charitable donations, and medical expenses. Self-employed..."
         ↑ Overlaps with chunk 1

Overlap Tuning:

Small (10-20 tokens)

Fast, low storage cost. Use for well-structured docs (articles, manuals).

Medium (30-50 tokens)

Recommended. Good balance. Preserves context without too much duplication.

Large (50-100+ tokens)

For complex, interdependent docs. Higher storage/embedding cost.

Custom Chunking Logic

Sometimes you need domain-specific chunking. Write custom logic for your use case.

Example: Code-Aware Chunking

Python

def chunk_code(code_text, max_tokens=400):
    """Chunk code by functions/classes, not arbitrary lines."""
    import re
    import tiktoken

    # Split by function/class definitions
    pattern = r'(def |class )(\w+)'
    functions = re.split(f'(?={pattern})', code_text)

    enc = tiktoken.encoding_for_model("gpt-4")
    chunks = []
    current = ""

    for func in functions:
        tokens = len(enc.encode(func))
        if len(enc.encode(current + func)) > max_tokens:
            if current:
                chunks.append(current)
            current = func
        else:
            current += func

    if current:
        chunks.append(current)
    return chunks

Notes

Overlap increases storage and embedding costs proportionally

A 10% overlap on 400-token chunks means every 10th chunk is redundant content you pay to embed and store. At 50% overlap the cost roughly doubles. For large corpora (millions of documents), model this explicitly: at $0.02 per 1M tokens, the extra cost of 30-token overlap over 1M chunks is $0.60 — negligible. At 100-token overlap over 100M chunks it becomes significant. Keep overlap proportional to how often answers span chunk boundaries in your specific document type.

Semantic chunking embeds during ingest, not just at query time

Unlike fixed or recursive chunkers, semantic chunking calls the embedding model during document ingestion to detect topic boundaries. This 3–10× slowdown at ingest time is acceptable for a nightly batch job but too slow for real-time document ingestion pipelines. Use recursive splitting as the default ingest strategy and reserve semantic chunking for static, high-value document collections where retrieval quality is the dominant concern.

Chunk size affects LLM cost, not just storage

Each retrieved chunk is passed as context tokens to the LLM. At top-K=5, doubling chunk size from 256 to 512 tokens doubles the LLM input token cost per query. At $0.003 per 1K input tokens on GPT-4, over 1M queries this adds up to thousands of dollars per month. The sweet spot is the smallest chunk that reliably contains enough context to answer the target question type — this requires measuring on your actual query distribution.

PDFs, HTML, and code each need different chunking approaches

RecursiveCharacterTextSplitter assumes plain text with paragraph and sentence boundaries. PDF text often has broken hyphenation, header/footer noise, and table content extracted as flat strings — pre-process with a PDF parser (pypdf, pdfplumber) before chunking. HTML should have tags stripped first. Code should chunk at function and class boundaries rather than at character count to preserve semantic units. Generic chunkers applied to structured formats produce poor results.