DS DevShelfHub Projects · AI tools
Tutorials / LangChain / Reference / Methods / .lazy_load()
Method Document loaders

.lazy_load(): Reference Guide

By DevShelfHub

Lazily load documents as iterator.

What is .lazy_load()?

.lazy_load() is the memory-efficient counterpart to .load() on all LangChain document loaders. Where .load() reads the entire source into memory at once and returns a List[Document], .lazy_load() returns an Iterator[Document] — a generator that yields one Document at a time as the source is read. The underlying I/O (file read, HTTP request, database cursor) happens incrementally as you consume the iterator, so only one document worth of data needs to be in memory at any given moment.

This makes .lazy_load() the right choice when you are processing directories with thousands of files, crawling large websites, or reading multi-gigabyte PDFs page by page. The typical pattern is a for-loop that processes and stores each document immediately rather than accumulating them all: for doc in loader.lazy_load(): vector_store.add_documents([doc]). This keeps memory flat regardless of source size.

Not all loaders implement lazy_load() natively — some fall back to calling .load() internally and yielding from the result list, which provides the same API without the memory benefit. Check the loader's source if memory is a constraint. For loaders that do support true streaming (like DirectoryLoader, WebBaseLoader, and S3DirectoryLoader), lazy_load() is significantly more memory-efficient. There is no async variant on the base class, but some loaders expose alazy_load() as an extension.

Use Cases

  • • Large documents
  • • Memory efficiency
  • • Streaming processing
  • • Batch loading
  • • Progressive indexing
  • • Resource management

Key Features

  • ✓ Lazy iteration
  • ✓ Memory efficient
  • ✓ Streaming
  • ✓ Iterator pattern
  • ✓ Process as-you-go
  • ✓ Low memory

When NOT to Use

For small documents—load() is simpler.

Notes

Iterator is consumed once — store results if you need to re-use

lazy_load() returns a generator, which can only be iterated once. If you need to pass the documents to multiple downstream steps, collect them into a list first: docs = list(loader.lazy_load()). Otherwise the second iteration yields nothing.

Some loaders fake lazy loading

Loaders that do not override lazy_load() inherit a default implementation that calls self.load() internally and yields from the result. This means memory usage is the same as .load() for those loaders. Check the loader class if you need genuine streaming — look for a native generator or file cursor in the implementation.

Combine with batch embedding to control memory and rate limits

Rather than indexing one document at a time, collect N documents in a buffer and call vector_store.add_documents(buffer) to reduce API round trips: buffer = []; for doc in loader.lazy_load(): buffer.append(doc); if len(buffer) >= 50: vector_store.add_documents(buffer); buffer = [].

Method Signature

python 
for doc in loader.lazy_load():
    process(doc)

Return Value

Type:

Iterator[Document]

Description:

Iterator of documents

Example Output:

for doc in loader.lazy_load(): ...

Code Examples

Basic lazy loading

python 
from langchain_community.document_loaders import TextLoader
loader = TextLoader('large_file.txt')
for doc in loader.lazy_load():
    process_and_store(doc)  # Process as we go

Incremental indexing of a directory

python 
from langchain_community.document_loaders import DirectoryLoader
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
loader = DirectoryLoader('./docs/', glob='**/*.md')
embeddings = OpenAIEmbeddings()
db = Chroma(embedding_function=embeddings)
for doc in loader.lazy_load():
    db.add_documents([doc])  # Index incrementally

Lazy load with per-document filtering

python 
from langchain_community.document_loaders import WebBaseLoader
urls = ["https://example.com/page1", "https://example.com/page2"]
loader = WebBaseLoader(urls)
docs = []
for doc in loader.lazy_load():
    if len(doc.page_content) > 200:  # skip stubs
        docs.append(doc)

Common Mistakes

❌ Iterate lazy_load() but don't consume

✅ Use in loop to actually load documents

Related LangChain References

Browse the full LangChain API reference index to explore more classes, methods, and decorators, or start with the LangChain introduction tutorial for end-to-end context on building with .lazy_load() and the wider framework.

.lazy_load() FAQ

What does .lazy_load() do in LangChain?

Lazily load documents as iterator. .lazy_load() is the memory-efficient counterpart to .load() on all LangChain document loaders. Where .load() reads the entire source into memory at once and returns a List[Document], .lazy_load() returns an Iterator[Document] — a generator that yields one Document at a time as the source is read. The underlying I/O (file read, HTTP request, database cursor) happens incrementally as you consume the iterator, so only one document worth of data needs to be in memory at any given…

Which LangChain classes support .lazy_load()?

.lazy_load() is available on Document loaders. Pin your installed LangChain version and verify the method exists in that release before deploying.

When should I use .lazy_load()?

Use .lazy_load() when your LangChain chains, agents, or pipelines need the behavior described in this guide.

What does .lazy_load() return?

.lazy_load() returns a Iterator[Document]. Iterator of documents

Does .lazy_load() have an async equivalent?

.lazy_load() does not have a documented async variant. Avoid .lazy_load() For small documents—load() is simpler.

Where can I explore more LangChain API reference pages?

Open the LangChain API reference index on DevShelfHub to browse classes, methods, and decorators, each with runnable examples, parameters, common mistakes, and cross-links.