DS DevShelfHub Projects · AI tools
Tutorials / CrewAI / Reference / Classes / ScrapeWebsiteTool
Class tools

ScrapeWebsiteTool: Reference Guide

By DevShelfHub

Fetches and cleans the text content of a URL.

See the CrewAI API reference index, CrewAI introduction, and core concepts for surrounding context.

What is ScrapeWebsiteTool?

ScrapeWebsiteTool is CrewAI's lightweight HTML fetcher: it downloads a page, strips boilerplate, and returns plain-ish text suitable for LLM prompts. No third-party API key is required, which makes it the default teaching tool for research pipelines — but that also means you inherit all HTTP responsibilities (timeouts, TLS validation, robots.txt norms, and politeness delays).

It excels on static marketing sites, blogs, and documentation rendered server-side. Client-rendered SPAs, infinite-scroll dashboards, and anti-bot pages often return shells or empty content; reach for FirecrawlScrapeWebsiteTool or Selenium-based scrapers when the DOM only materializes after JavaScript. Respect copyright and terms of service: scraping is powerful and easy to abuse.

In multi-tool crews, keep ScrapeWebsiteTool behind SerperDevTool or explicit URLs so agents are not guessing arbitrary hosts. Add domain allow lists at the orchestration layer when agents run unsupervised.

When to Use

Static documentation, news articles, and public help centers where simple HTTP GET plus HTML parse is enough.

Use Cases

  • • Article extraction
  • • Light scraping
  • • Follow-up fetch after SerperDevTool URL discovery

Key Features

  • ✓ No API key
  • ✓ Returns cleaned text
  • ✓ Low setup cost
  • ✓ Pairs with search tools

When NOT to Use

Authenticated portals, heavy JavaScript SPAs, CAPTCHA-protected sites, or bulk crawling at aggressive rates.

Notes

Timeouts and user agents

Slow hosts stall the whole task. Set conservative HTTP timeouts where your CrewAI version exposes them, and identify your traffic responsibly so site owners can allow list you.

HTML noise vs paywalls

Paywalls and cookie banners may yield empty text without errors. Detect tiny outputs and fall back to another tool or human escalation rather than hallucinating missing sections.

SSRF risk

Agents can request internal IPs if prompts are poisoned. Block RFC1918 ranges and metadata endpoints in staging and production, not only in demos.

Legal and rate limits

Automated scraping may violate site terms. Cache aggressively, respect robots directives where applicable, and throttle parallel scrapes to avoid accidental DDoS patterns.

Import

python 
from crewai_tools import ScrapeWebsiteTool

Code Examples

Attach to a researcher agent

python 
from crewai import Agent
from crewai_tools import ScrapeWebsiteTool

researcher = Agent(
    role='Researcher',
    goal='Read cited pages',
    backstory='You quote only what appears in fetched text.',
    tools=[ScrapeWebsiteTool()],
)

Two-task search then scrape pattern

python 
from crewai import Agent, Task, Crew
from crewai_tools import SerperDevTool, ScrapeWebsiteTool

agent = Agent(role='Researcher', goal='Gather sources', backstory='...', tools=[SerperDevTool(), ScrapeWebsiteTool()])
find = Task(description='Find 3 URLs about {topic}', expected_output='Markdown links', agent=agent)
read = Task(description='Scrape each URL and summarize claims', expected_output='Bullets with quotes', agent=agent, context=[find])
Crew(agents=[agent], tasks=[find, read]).kickoff(inputs={'topic': 'vector databases'})

Scrape a known documentation URL

python 
from crewai import Agent, Task, Crew
from crewai_tools import ScrapeWebsiteTool

reader = Agent(role='Doc reader', goal='Extract facts', backstory='You cite only scraped text.', tools=[ScrapeWebsiteTool()])
fetch = Task(description='Fetch https://docs.python.org/3/library/asyncio-task.html and list 4 APIs', expected_output='Markdown bullets', agent=reader)
Crew(agents=[reader], tasks=[fetch]).kickoff()

Common Mistakes

❌ Expecting JS-rendered dashboards to scrape cleanly

✅ Switch to Firecrawl or a browser automation tool for client-rendered pages.

❌ Letting agents scrape file:// or internal admin URLs

✅ Enforce URL validation in hooks or a wrapper tool.

ScrapeWebsiteTool FAQ

What is ScrapeWebsiteTool in CrewAI?

Fetches and cleans the text content of a URL. ScrapeWebsiteTool is CrewAI's lightweight HTML fetcher: it downloads a page, strips boilerplate, and returns plain-ish text suitable for LLM prompts. No third-party API key is required, which makes it the default teaching tool for research pipelines — but that also means you inherit all HTTP responsibilities (timeouts, TLS validation, robots.txt norms, and politeness delays). It excels on static marketing sites, blogs, and documentation rendered server-side. Client-rendered …

Which package defines the CrewAI class ScrapeWebsiteTool?

DevShelfHub maps ScrapeWebsiteTool to Python module crewai_tools (package path crewai_tools in this reference). Pin your installed crewai version and match imports to the snippet on this page.

When should I use ScrapeWebsiteTool?

Static documentation, news articles, and public help centers where simple HTTP GET plus HTML parse is enough.

When should I avoid using ScrapeWebsiteTool?

Authenticated portals, heavy JavaScript SPAs, CAPTCHA-protected sites, or bulk crawling at aggressive rates.

How do I import ScrapeWebsiteTool in Python?

from crewai_tools import ScrapeWebsiteTool

Where can I explore more CrewAI API reference pages?

Open the CrewAI API reference index on DevShelfHub to search 58 classes, 30 methods, and 16 decorators, each with runnable examples, parameters, common mistakes, and cross-links.