Ethical scraping
✓ Check the Terms of Service
Some sites forbid scraping. Always read ToS before crawling.
✓ Respect robots.txt
Firecrawl respects it by default. Don't override it unless you own the site.
✓ Add rate-limit delays
Use time.sleep(2-5) between requests to avoid hammering servers.
✓ Don't scrape personal data
Be careful with PII. Respect privacy laws (GDPR, CCPA).
Cost optimization
Firecrawl's free tier gives 500 credits/month. 1 page = 1 credit. Here's how to optimize:
Set aggressive limits
Use limit parameter aggressively in testing. 50 pages is often enough to validate extraction logic.
Use caching
Firecrawl caches responses. Re-running the same crawl is free (within 24h).
Crawl incrementally
Don't re-crawl everything daily. Track what changed and only re-crawl changed pages.
Use selective formats
Request only what you need. If you only want Markdown, don't request HTML or links.
Common errors and solutions
401 Unauthorized
Cause: Invalid API key. Fix: Verify FIRECRAWL_API_KEY environment variable.
429 Too Many Requests
Cause: Rate limited. Fix: Add delays between requests. Check credit balance.
Empty markdown response
Cause: Page might be JavaScript-heavy or login-required. Fix: Try waitFor: 2000 or add auth headers.
Site blocked/403
Cause: Anti-bot detection. Fix: Use a proxy with proxyUrl parameter.
Debugging tips
Request raw HTML first
When Markdown looks wrong, request formats: ["html"] to see the raw HTML and understand what's happening.
Test on simple pages first
Before crawling a complex site, test on a single simple page to debug extraction logic.
Use small limits during development
limit: 5 in dev, scale up in prod.
Log everything
Log URLs being crawled, response codes, timing. Helps diagnose patterns in failures.
Final checklist
- ✓ Checked site ToS and robots.txt
- ✓ Added rate-limit delays
- ✓ Set conservative limits in development
- ✓ Handled errors with try/except
- ✓ Tested on simple pages first
- ✓ Logged URLs and responses
- ✓ Optimized for cost (caching, incremental updates)
Operational excellence for scraping pipelines
Production scraping pipelines require the same operational rigor as any backend service. Implement monitoring for scrape success rates, average extraction quality, and cost per page. Set up alerts when success rates drop below 95% — this usually indicates site layout changes that break your extraction logic. Version your scraping configurations and treat them as code, not ad-hoc scripts.
Cost optimization is straightforward but often overlooked. Cache aggressively — if a page hasn't changed, don't re-scrape it. Use the crawl endpoint instead of individual scrape calls when processing entire sites (batch crawling is cheaper per page). Filter URLs before scraping to avoid wasting credits on irrelevant pages like privacy policies and cookie consent pages. For teams using scraped data in AI pipelines, these operational practices compound over time. A well-maintained scraping pipeline feeding into a RAG system or fine-tuning pipeline becomes a competitive advantage.
Firecrawl Best Practices FAQ
How do I reduce Firecrawl costs?
Use scrape for single pages instead of crawl when possible. Set tight URL filters and depth limits on crawls. Cache results locally to avoid re-scraping unchanged content.
What are the most common Firecrawl errors?
Timeout errors from slow-loading pages, 403 errors from bot detection, and empty content from JavaScript that requires specific user interaction. Most are solved with wait strategies and header configuration.
How do I scrape ethically with Firecrawl?
Respect robots.txt, add reasonable delays between requests, identify your bot with a user-agent string, and avoid overwhelming small sites with high-concurrency crawls.
Why is Firecrawl returning empty content?
The page likely requires JavaScript interaction beyond initial render. Use actions to click consent banners, scroll to trigger lazy loading, or wait for specific elements to appear.
How do I debug Firecrawl scraping issues?
Start with a single-page scrape and inspect the raw HTML output. Check if the content appears in the rendered page by enabling screenshot mode. Test with different wait strategies.
Related tutorials
Continue learning with our advanced features guide and all tutorials.