Web Crawling with Firecrawl: Site Scraping

Scrape vs Crawl: Understanding the difference

scrape_url()

Scrapes a single page. You provide one URL, get back one page's content.

crawl_url()

Crawls a site. Discovers links, follows them, and scrapes all pages. Returns an array of page results.

Basic crawl example

python

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-your-api-key")

crawl_result = app.crawl_url(
    "https://docs.example.com",
    params={
        "limit": 50,              # max 50 pages
        "maxDepth": 3,            # follow links up to 3 levels deep
        "includePaths": ["/docs/.*"],  # only docs pages
        "excludePaths": ["/blog/.*", "/changelog/.*"],
        "scrapeOptions": {
            "formats": ["markdown"],
            "onlyMainContent": True,
        },
    }
)

for page in crawl_result["data"]:
    print(page["metadata"]["sourceURL"])
    print(page["markdown"][:200])
    print("---")

Crawl parameters

`maxDepth`

How deep to follow links. Depth 1 = only direct links from the start URL. Depth 3 = links from links from links. Start with 2-3.

`limit`

Maximum number of pages to crawl. Set this conservatively — 50-100 for testing, thousands for production.

`includePaths`

Regex pattern to include only matching URLs. Example: ["/docs/.*"] to crawl only docs pages.

`excludePaths`

Regex to exclude URLs. Example: ["/blog/.*", "/tag/.*"] to skip blog and tag pages.

Asynchronous crawling

For large crawls, use async crawling so you don't block while waiting for results. The crawl happens in the background.

# Start async crawl
async_result = app.async_crawl_url("https://docs.example.com", {
    "limit": 100,
    "maxDepth": 2,
})
crawl_id = async_result["id"]

# Check status later
import time
time.sleep(30)  # wait for crawl to progress
status = app.check_crawl_status(crawl_id)
print(f"Status: {status['status']}")  # 'running', 'completed', etc
print(f"Pages crawled: {len(status['data'])}")

# Get results when done
if status['status'] == 'completed':
    pages = status['data']

Ethical scraping: robots.txt and rate limits

✓ Do: Respect robots.txt

Firecrawl respects robots.txt by default. It won't crawl paths that disallow you.

✓ Do: Add delays between requests

Use time.sleep(1-2) between crawls to avoid overloading servers.

✗ Don't: Crawl sites that forbid it

Always check the site's Terms of Service before scraping. Some sites explicitly forbid crawling.

✗ Don't: Crawl personal data

Respect privacy. Don't crawl or store personal information without consent.

Key takeaway

Use crawl_url() to crawl multiple pages
Start with small limit and maxDepth values
Use includePaths to filter URLs
Always respect robots.txt and add rate-limit delays

Crawling strategies for AI applications

The crawl endpoint is where Firecrawl delivers the most value for AI teams. A single API call can traverse an entire documentation site, respect robots.txt, follow links to a specified depth, and return clean Markdown for every page discovered. This turns what would be weeks of custom spider development into a few lines of Python code.

When crawling for RAG pipelines, set a reasonable depth limit (2-3 levels) and use URL filters to stay within the relevant sections of a site. Crawling an entire 10,000-page site when you only need the API reference section wastes credits and pollutes your vector database with irrelevant content. For training data collection, broader crawls with content-type filtering work well — crawl a site, filter for pages that contain code examples or technical explanations, and use these as seed material for fine-tuning datasets. The key is matching your crawl scope to your downstream task.

Web Crawling FAQ

How do I crawl an entire website with Firecrawl?

Use app.crawl_url with the starting URL. Firecrawl discovers and follows links automatically, returning content from every page it finds within your configured limits.

Can I limit how many pages Firecrawl crawls?

Yes. Set the limit parameter to cap the number of pages. You can also set max_depth to control how many link levels deep the crawler goes from the starting URL.

How do I filter which pages get crawled?

Use include and exclude patterns with glob syntax. For example, include only blog posts with '/blog/*' or exclude image galleries with '!/gallery/*'.

Does Firecrawl handle pagination?

Yes. Firecrawl follows pagination links as part of its normal crawl. For infinite-scroll pages, it uses the headless browser to load additional content before extracting.

How long does a full site crawl take?

It depends on site size and your plan limits. A 100-page site typically completes in 5-15 minutes. Firecrawl runs crawls asynchronously so you can poll for results.

Continue learning with our RAG integration tutorial and best practices guide.

Web Crawling