Basic Web Scraping with Firecrawl

scrape_url() deep dive

The scrape_url() method has many options. Let's explore the most useful ones.

python

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-your-api-key")

result = app.scrape_url(
    "https://docs.example.com/introduction",
    params={
        "formats": ["markdown", "links"],
        "onlyMainContent": True,
        "waitFor": 1000,          # wait 1s for JS to render
        "timeout": 15000,         # 15s timeout
    }
)

print(result["markdown"])         # clean text content
print(result["links"])            # all links on the page
print(result["metadata"]["title"]) # page title

Output formats

Firecrawl can output in multiple formats. Choose based on your use case.

Markdown

Clean, readable, preserves structure (headings, lists, links). Best for LLMs and RAG.

JSON

Structured, machine-readable, perfect for APIs and databases.

HTML

Raw HTML for custom parsing. Good for debugging or extracting specific elements.

Extract (Schema)

Use a Pydantic schema to extract specific fields (e.g., price, title, rating).

Structured data extraction

You can extract specific fields using a Pydantic schema. Useful for e-commerce, product data, or any structured extraction.

from pydantic import BaseModel

class Article(BaseModel):
    title: str
    author: str
    publish_date: str
    content: str

result = app.scrape_url(url, {
    "formats": ["extract"],
    "extract": {
        "schema": Article.model_json_schema(),
        "prompt": "Extract the article title, author, publication date, and content."
    }
})

article = result["extract"]

Handling different content types

✓ Blog posts and articles

Perfect for Firecrawl. Clear title, body, metadata. Works great with onlyMainContent: true.

✓ Documentation pages

Structured headings and code blocks extract cleanly. Often JS-heavy, so Firecrawl's rendering is crucial.

⚠ E-commerce product pages

Dynamic content and infinite scroll. Use schema extraction with waitFor for lazy-loaded images.

✗ Login-required pages

Firecrawl can pass cookies or headers, but it can't automate login flows yet. Use extraHeaders if you have auth tokens.

Error handling

Always wrap Firecrawl calls in try/except blocks.

import time
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-...")

urls = ["https://example.com/page1", "https://example.com/page2"]

for url in urls:
    try:
        result = app.scrape_url(url)
        print(f"✓ Scraped {url}")
    except Exception as e:
        print(f"✗ Failed {url}: {e}")
    time.sleep(2)  # respect rate limits

Summary

Use Markdown for LLMs, JSON for APIs, extract for specific fields
Set onlyMainContent: true to strip boilerplate
Use waitFor for dynamic/JS-heavy sites
Always add error handling and rate-limit delays

Optimizing scrape quality

The quality of your scraped data directly impacts downstream AI performance. For RAG pipelines, noisy scraped text (navigation elements, cookie banners, footer content mixed into the main text) degrades retrieval accuracy because irrelevant tokens dilute the semantic signal. Always use the onlyMainContent option and inspect a sample of outputs before feeding them into your pipeline.

Structured extraction with Pydantic schemas is particularly powerful for building training datasets. Instead of scraping raw text and parsing it later, define exactly what fields you need (title, author, date, content, tags) and let Firecrawl's LLM-powered extraction fill them. This produces clean, consistent records that can go directly into a JSONL file for fine-tuning data preparation. For multi-page extraction jobs, the crawl endpoint (covered in the next lesson) handles link discovery and pagination automatically.

Web Scraping FAQ

What output formats does Firecrawl support?

Firecrawl can return content as clean Markdown, raw HTML, or structured JSON via its extract mode. Markdown is the default and works best for LLM consumption.

How do I scrape a single page with Firecrawl?

Call app.scrape_url with the target URL. The response includes the page content in your chosen format, plus metadata like the page title and description.

Can Firecrawl extract specific data from a page?

Yes. Use the extract mode with a JSON schema to pull structured fields like prices, names, or dates. Firecrawl uses an LLM to map page content to your schema.

How do I handle scraping errors?

Wrap scrape calls in try-except blocks. Common errors include timeout for slow pages, 403 for blocked requests, and 404 for missing pages. Firecrawl retries automatically on transient failures.

What is the difference between scrape and crawl?

Scrape processes a single URL and returns its content. Crawl discovers and follows links from a starting URL, processing multiple pages across a site.

Continue learning with our web crawling tutorial and advanced features guide.

Basic Web Scraping