Firecrawl Advanced: JS, Auth, Selectors

Custom extraction with selectors

Use CSS selectors to target specific elements on a page. Great for schema-based extraction when you know the page structure.

python

from firecrawl import FirecrawlApp
from pydantic import BaseModel

app = FirecrawlApp(api_key="fc-your-api-key")

# Structured extraction with schema
class Product(BaseModel):
    title: str
    price: str
    description: str
    rating: float | None

result = app.scrape_url(
    "https://shop.example.com/product/123",
    params={
        "formats": ["extract"],
        "extract": {
            "schema": Product.model_json_schema(),
            "prompt": "Extract the product title, price, description, and rating.",
        }
    }
)

product = result["extract"]
print(f"{product['title']} — {product['price']}")

JavaScript actions

Firecrawl can perform actions before extracting content: click buttons, scroll, type in forms, wait for elements.

Click a button

{"type": "click", "selector": ".load-more"}

Wait for an element

{"type": "waitForElement", "selector": ".data-loaded"}

Scroll down

{"type": "scroll", "direction": "down", "amount": 3}

Handling dynamic content

Modern sites load content dynamically. Use these parameters to handle them.

`waitFor`

Wait N milliseconds for the page to fully load before extraction. Default: 0. Try 1000-3000 for heavy JS pages.

`waitForSelector`

Wait until a specific CSS selector appears. Better than waitFor since it waits for actual content, not just time.

Authentication and protected content

If a site requires authentication, pass headers or cookies.

# Pass custom headers (e.g., auth token)
result = app.scrape_url(
    "https://api.example.com/protected",
    params={
        "extraHeaders": {
            "Authorization": "Bearer your-token-here"
        }
    }
)

# Pass cookies
result = app.scrape_url(
    "https://example.com",
    params={
        "extraHeaders": {
            "Cookie": "session_id=abc123; user=john"
        }
    }
)

Note: Firecrawl can't automate login flows yet (fill forms, click login). If you need that, use Playwright instead.

Proxy support

Use a proxy if you're being rate-limited or need to bypass geo-blocking.

result = app.scrape_url(
    url,
    params={
        "proxyUrl": "http://proxy.example.com:8080"
    }
)

Caching and incremental crawls

Firecrawl automatically caches responses. Don't re-scrape unchanged pages.

Benefit: Re-running the same crawl uses cached results, saving credits and time.

Use case: Incremental content updates. Crawl your docs site weekly. Only changed pages are re-scraped.

When to use advanced features

Selectors: When you know the HTML structure and need specific fields
JavaScript actions: When buttons must be clicked or forms filled before content appears
waitFor: When JavaScript libraries (React, Vue) render content after page load
Authentication: When scraping protected APIs or member-only pages
Proxies: When you're being rate-limited by origin IP

Advanced patterns in production

Advanced Firecrawl features are most valuable when building repeatable data pipelines. Custom JavaScript execution lets you interact with pages before extraction — clicking "load more" buttons, dismissing modals, or scrolling to trigger lazy-loaded content. Combined with wait strategies, this handles even the most complex single-page applications that would break traditional scrapers.

Authentication support unlocks a category of data that is invisible to standard scrapers: gated documentation, internal wikis, and member-only content. Pass session cookies or bearer tokens via the extraHeaders parameter to scrape authenticated pages. This is particularly useful when building internal knowledge bases or collecting proprietary training data for fine-tuning custom LLMs. For teams running at scale, the caching layer avoids redundant scrapes of pages that haven't changed, reducing both cost and latency.

Advanced Firecrawl FAQ

Can Firecrawl scrape pages behind a login?

Yes. You can pass cookies or authentication headers with your scrape request. For more complex flows, use the actions API to fill in login forms before extracting content.

How do I use CSS selectors with Firecrawl?

Pass include or exclude selectors in the scrape options to target specific page sections. For example, use 'article.main-content' to extract only the article body.

Can Firecrawl execute JavaScript on a page?

Yes. Use the actions API to click buttons, scroll, fill forms, or wait for elements before scraping. Actions run in sequence in a real browser environment.

How do I handle dynamic content that loads after page load?

Use wait_for options to pause until a specific CSS selector appears or a timeout elapses. Combine with JavaScript actions to trigger lazy-loaded content.

Can Firecrawl take screenshots?

Yes. Enable the screenshot option in your scrape request and Firecrawl returns a base64-encoded image of the rendered page alongside the extracted content.

Continue learning with our basic scraping tutorial and RAG integration guide.

Advanced Features