DS DevShelfHub Projects · AI tools
Tutorials / CrewAI / Reference / Classes / FirecrawlScrapeWebsiteTool
Class tools

FirecrawlScrapeWebsiteTool: Reference Guide

By DevShelfHub

LLM-friendly web scraping via Firecrawl — handles JS, returns clean Markdown.

See the CrewAI API reference index, CrewAI introduction, and core concepts for surrounding context.

What is FirecrawlScrapeWebsiteTool?

FirecrawlScrapeWebsiteTool wraps the Firecrawl hosted scraping API so agents can pull JavaScript-rendered pages into Markdown snippets the model can reason over — a step up from ScrapeWebsiteTool, which only handles static HTML. Typical stacks pair SerperDevTool (URL discovery) with Firecrawl for full article text, especially on SPAs, authenticated marketing sites, or documentation portals with heavy client-side routing.

Because every call bills against your Firecrawl quota, cap concurrency with max_rpm on the Crew and teach agents to reuse URLs instead of re-scraping unchanged pages. Handle failures explicitly: Firecrawl returns structured errors for blocked sites, bot challenges, or timeouts — bubble those strings up so the planner can retry with a different strategy instead of looping blindly.

Operational hygiene mirrors other API tools: load FIRECRAWL_API_KEY from the environment, redact scraped HTML from logs, and respect robots/terms for the domains you target.

When to Use

Dynamic/JS sites, structured extraction.

Use Cases

  • • SPA scraping
  • • Bulk article extraction

Key Features

  • ✓ JS rendering
  • ✓ Markdown output

When NOT to Use

Simple static pages — ScrapeWebsiteTool is cheaper.

Notes

Quota and cost

Rendered pages cost more than snippets. Cache Markdown in your own store when the same URL is hot.

Latency vs static scraper

JS rendering adds seconds. Set Crew timeouts and consider ScrapeWebsiteTool for known-static docs.

403 and anti-bot pages

Some sites always block third-party scrapers. Teach agents to detect short error bodies and pivot to official APIs or PDFs.

Key rotation

Rotate FIRECRAWL_API_KEY on the same schedule as other third-party keys; update deployment secrets before revoking the old key.

Import

python 
from crewai_tools import FirecrawlScrapeWebsiteTool

Key Parameters

Parameter Type Default Purpose
api_key str (env) $FIRECRAWL_API_KEY Firecrawl API key.

Code Examples

Researcher with Serper + Firecrawl

python 
import os
from crewai import Agent
from crewai_tools import FirecrawlScrapeWebsiteTool, SerperDevTool

os.environ.setdefault('FIRECRAWL_API_KEY', '***')

researcher = Agent(
    role='Web researcher',
    goal='Gather cited facts',
    backstory='Search first, scrape second.',
    tools=[SerperDevTool(), FirecrawlScrapeWebsiteTool()],
)

Load API key from the environment only

python 
import os

assert os.environ.get('FIRECRAWL_API_KEY'), 'set FIRECRAWL_API_KEY before kickoff'

# FirecrawlScrapeWebsiteTool reads the key from the environment by default.
from crewai_tools import FirecrawlScrapeWebsiteTool

tool = FirecrawlScrapeWebsiteTool()

Tight tool surface on a summarizer

python 
from crewai import Agent
from crewai_tools import FirecrawlScrapeWebsiteTool

summarizer = Agent(
    role='Summarizer',
    goal='Return Markdown bullets',
    backstory='Only call scrape when URL is provided.',
    tools=[FirecrawlScrapeWebsiteTool()],
)

Common Mistakes

❌ Shipping FIRECRAWL_API_KEY inside repo YAML

✅ Inject via environment or secret manager at deploy time.

FirecrawlScrapeWebsiteTool FAQ

What is FirecrawlScrapeWebsiteTool in CrewAI?

LLM-friendly web scraping via Firecrawl — handles JS, returns clean Markdown. FirecrawlScrapeWebsiteTool wraps the Firecrawl hosted scraping API so agents can pull JavaScript-rendered pages into Markdown snippets the model can reason over — a step up from ScrapeWebsiteTool, which only handles static HTML. Typical stacks pair SerperDevTool (URL discovery) with Firecrawl for full article text, especially on SPAs, authenticated marketing sites, or documentation portals with heavy client-side routing. Because every call bills against your Firecrawl quota,…

Which package defines the CrewAI class FirecrawlScrapeWebsiteTool?

DevShelfHub maps FirecrawlScrapeWebsiteTool to Python module crewai_tools (package path crewai_tools in this reference). Pin your installed crewai version and match imports to the snippet on this page.

When should I use FirecrawlScrapeWebsiteTool?

Dynamic/JS sites, structured extraction.

When should I avoid using FirecrawlScrapeWebsiteTool?

Simple static pages — ScrapeWebsiteTool is cheaper.

How do I import FirecrawlScrapeWebsiteTool in Python?

from crewai_tools import FirecrawlScrapeWebsiteTool

Where can I explore more CrewAI API reference pages?

Open the CrewAI API reference index on DevShelfHub to search 58 classes, 30 methods, and 16 decorators, each with runnable examples, parameters, common mistakes, and cross-links.