What is RecursiveCharacterTextSplitter?
RecursiveCharacterTextSplitter is the recommended general-purpose splitter for RAG pipelines. It works by attempting to split on a priority-ordered list of separators: first double newlines (paragraph boundaries), then single newlines, then spaces, and finally individual characters if no other boundary fits within chunk_size. This strategy keeps semantically related sentences together far more reliably than a naive fixed-size character split.
The two most important parameters are chunk_size (maximum characters per chunk) and chunk_overlap (characters repeated between adjacent chunks). chunk_overlap ensures context is not cut off at a chunk boundary—a sentence that starts near the end of chunk N will also appear at the beginning of chunk N+1. For most RAG workloads, chunk_size=1000 and chunk_overlap=200 is a reasonable starting point, but optimal values depend on your embedding model's context window and the density of your source documents.
For language-specific content, the from_language() class method sets the separator list to language-appropriate tokens—useful for splitting Python source files by function/class boundaries or Markdown by heading level. Token-based sizing via tiktoken is available through from_tiktoken_encoder(), letting you specify chunk_size in tokens rather than characters, which is more directly related to what the embedding model sees.
When to Use
You need to split documents for embedding. Use this for most RAG applications as it handles semantic boundaries intelligently.
Use Cases
- • Prepare documents for embedding
- • Create chunks for RAG
- • Split long documents
- • Maintain semantic coherence
- • Handle multiple formats
- • Token limit enforcement
Key Features
- ✓ Semantic boundary preservation
- ✓ Configurable overlap
- ✓ Multiple language support
- ✓ Format-aware splitting
- ✓ Metadata preservation
- ✓ Token-based sizing
When NOT to Use
For fixed-size requirements where semantics don't matter.
Notes
chunk_overlap adds token cost
chunk_overlap copies characters between chunks—it does not deduplicate. Overlapping chunks add cost on every embedding call. Monitor chunk distribution with [len(c.page_content) for c in chunks] before committing to production parameters.
from_language for code and markdown
RecursiveCharacterTextSplitter.from_language(Language.PYTHON) sets separators to Python class/function boundaries. from_language(Language.MARKDOWN) splits by heading levels. These produce far cleaner chunks than splitting code by raw characters.
split_text vs split_documents
split_text(text) takes a plain string and returns list[str]. split_documents(docs) takes list[Document] and returns list[Document], preserving metadata from the source. Always use split_documents() in RAG pipelines to keep source, page, and URL metadata intact.
Token count vs character count
chunk_size defaults to character count, not tokens. With text-embedding-ada-002 (8191 token limit), chunk_size=4000 chars is conservative. Use from_tiktoken_encoder() to set chunk_size in tokens directly for predictable model context usage.
Import
from langchain_text_splitters import RecursiveCharacterTextSplitterKey Parameters
| Parameter | Type | Default | Purpose |
|---|---|---|---|
| chunk_size | int | 1000 | Target chunk size in characters |
Code Examples
Split Document
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, chunk_overlap=200
)
chunks = splitter.split_documents(docs)Split Python Code by Function Boundaries
from langchain_text_splitters import Language
splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON,
chunk_size=2000, chunk_overlap=200
)
chunks = splitter.split_text(python_source_code)Token-Based Chunking with tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
model_name="gpt-4o",
chunk_size=512, chunk_overlap=64
)
chunks = splitter.split_documents(docs)Common Mistakes
❌ Chunk size too large (loses specificity)
✅ Use 1000 characters as baseline, adjust for your domain
Related LangChain References
Browse the full LangChain API reference index to explore more classes, methods, and decorators, or start with the LangChain introduction tutorial for end-to-end context on building with RecursiveCharacterTextSplitter and the wider framework.
RecursiveCharacterTextSplitter FAQ
What is RecursiveCharacterTextSplitter in LangChain?
Split text into chunks while respecting semantic boundaries. RecursiveCharacterTextSplitter is the recommended general-purpose splitter for RAG pipelines. It works by attempting to split on a priority-ordered list of separators: first double newlines (paragraph boundaries), then single newlines, then spaces, and finally individual characters if no other boundary fits within chunk_size. This strategy keeps semantically related sentences together far more reliably than a naive fixed-size character split. The two most important parameter…
Which package provides RecursiveCharacterTextSplitter?
DevShelfHub documents RecursiveCharacterTextSplitter from the langchain-text-splitters package. Pin your installed LangChain version and match imports to the snippet on this page.
When should I use RecursiveCharacterTextSplitter?
You need to split documents for embedding. Use this for most RAG applications as it handles semantic boundaries intelligently.
When should I avoid using RecursiveCharacterTextSplitter?
For fixed-size requirements where semantics don't matter.
How do I import RecursiveCharacterTextSplitter in Python?
from langchain_text_splitters import RecursiveCharacterTextSplitter
Where can I explore more LangChain API reference pages?
Open the LangChain API reference index on DevShelfHub to browse classes, methods, and decorators, each with runnable examples, parameters, common mistakes, and cross-links.