WebBaseLoader: Reference Guide

What is WebBaseLoader?

WebBaseLoader is one of the simplest document loaders in the langchain-community package. It uses Python requests to fetch one or more URLs and then BeautifulSoup4 to parse the HTML and extract text content. The resulting Document objects contain the raw text in page_content and a metadata dict with the source URL. Because it runs synchronous HTTP requests, WebBaseLoader is most appropriate for small-scale ingestion: loading documentation pages, blog posts, or landing pages into a RAG pipeline.

Under the hood, WebBaseLoader respects standard HTTP headers and you can pass custom header_template and requests_kwargs to control User-Agent strings, authentication tokens, or proxy settings. It does not render JavaScript — pages that load content dynamically via React, Vue, or Next.js will return empty or partial page_content. For JavaScript-rendered pages, the correct alternative is FireCrawlLoader (cloud-based) or PlaywrightURLLoader (headless browser, local).

The web_paths parameter accepts a single URL string or a list. When multiple URLs are provided, load() fetches them sequentially. For concurrent fetching, use aload() which fires all requests concurrently using aiohttp and can dramatically reduce wall-clock time for large URL lists. The bs_kwargs parameter accepts BeautifulSoup options like SoupStrainer to extract only specific HTML elements, reducing noise in the extracted text.

When to Use

You need to index web pages or documentation. Use WebBaseLoader for simple static websites.

Use Cases

• • Index documentation
• • Scrape static websites
• • Research collection
• • Content aggregation
• • Web-based RAG
• • Documentation indexing

Key Features

• ✓ Simple HTML parsing
• ✓ Text extraction
• ✓ No JavaScript support
• ✓ BeautifulSoup-based
• ✓ Batch loading
• ✓ Metadata

When NOT to Use

Notes

Import

from langchain_community.document_loaders import WebBaseLoader

Key Parameters

Usage Examples

loader = WebBaseLoader('https://docs.example.com/intro')
docs = loader.load()
for doc in docs:
    print(doc.page_content[:100])

from langchain_community.document_loaders import WebBaseLoader
from bs4 import SoupStrainer

urls = [
    'https://docs.langchain.com/docs/introduction',
    'https://docs.langchain.com/docs/get-started',
    'https://docs.langchain.com/docs/use-cases',
]
loader = WebBaseLoader(
    web_paths=urls,
    bs_kwargs={'parse_only': SoupStrainer('article')},
)
docs = loader.load()
print(f'Loaded {len(docs)} pages')

import asyncio
from langchain_community.document_loaders import WebBaseLoader

urls = [f'https://docs.example.com/page-{i}' for i in range(20)]
loader = WebBaseLoader(web_paths=urls)

async def load_all():
    return await loader.aload()

docs = asyncio.run(load_all())
print(f'Loaded {len(docs)} pages concurrently')

Common Mistakes

Related LangChain References

Browse the full LangChain API reference index to explore more classes, methods, and decorators, or start with the LangChain introduction tutorial for end-to-end context on building with WebBaseLoader and the wider framework.

WebBaseLoader FAQ