Integrating with RAG Pipelines

Why Firecrawl is perfect for RAG

RAG (Retrieval Augmented Generation) pipelines need clean, structured content to work well. Firecrawl solves the hardest part: extracting readable content from messy web pages.

The RAG pipeline with Firecrawl

Here's the complete flow:

Full working example

Here's a complete RAG pipeline using Firecrawl, OpenAI embeddings, and FAISS.

import os
from firecrawl import FirecrawlApp
from openai import OpenAI
import faiss, numpy as np

firecrawl = FirecrawlApp(api_key=os.environ["FIRECRAWL_API_KEY"])
openai_client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

# Step 1: Crawl the docs
crawl = firecrawl.crawl_url(
    "https://docs.example.com",
    params={"limit": 100, "scrapeOptions": {"formats": ["markdown"], "onlyMainContent": True}}
)

# Step 2: Chunk each page by heading
def chunk_by_heading(markdown: str) -> list[str]:
    sections = markdown.split("\n## ")
    return [s.strip() for s in sections if len(s.strip()) > 50]

chunks = []
for page in crawl["data"]:
    chunks.extend(chunk_by_heading(page.get("markdown", "")))

# Step 3: Embed
def embed(texts: list[str]) -> np.ndarray:
    res = openai_client.embeddings.create(input=texts, model="text-embedding-3-small")
    return np.array([r.embedding for r in res.data], dtype="float32")

embeddings = embed(chunks)

# Step 4: Store in FAISS
index = faiss.IndexFlatL2(embeddings.shape[1])
index.add(embeddings)

# Step 5: Query
def search(query: str, k: int = 3) -> list[str]:
    q_emb = embed([query])
    _, ids = index.search(q_emb, k)
    return [chunks[i] for i in ids[0]]

results = search("How do I authenticate?")
for r in results:
    print(r[:300])
    print("---")

Chunking strategies

Firecrawl gives you Markdown. You need to split it into chunks for embedding.

Preprocessing crawled content

Sometimes you need to clean up Firecrawl's output before storing in your vector DB.

Incremental updates

Don't re-crawl everything every day. Track changes and update only what's new.

# Strategy: hash-based change detection
import hashlib
from datetime import datetime

last_crawl = load_from_db("last_crawl_metadata")

for page in pages:
    content_hash = hashlib.sha256(page["markdown"].encode()).hexdigest()
    if content_hash != last_crawl.get(page["url"]):
        # Content changed — re-chunk, re-embed, update vectors
        update_vector_db(page)
        save_metadata(page["url"], content_hash, datetime.now())

Building production RAG with Firecrawl

The scrape-chunk-embed-store pipeline is the foundation of every production RAG system. Firecrawl handles the hardest part — extracting clean, structured text from messy web pages — so you can focus on chunking strategy, embedding model selection, and retrieval tuning. The quality of your scraping directly determines retrieval accuracy: garbage in, garbage out.

For best results, use Firecrawl's Markdown output and chunk by heading structure rather than fixed token counts. Headings provide natural semantic boundaries that preserve context within each chunk. Combine this with metadata extraction (page title, section heading, URL) stored alongside each chunk so your retrieval pipeline can filter by source. If your RAG system still produces low-quality answers after optimizing retrieval, consider fine-tuning the generation model for your specific domain — the combination of RAG for fresh knowledge and fine-tuning for domain behavior produces the best results in practice.

Firecrawl RAG FAQ

Related tutorials

Continue learning with our RAG pipeline tutorial series and real-world use cases.