Introduction
A 50-line scraping script works on day one. By day three you’re blocked. By week two your data is inconsistent across regions, sessions are dying, and you don’t trust the numbers anymore. That’s the moment web scraping stops being a coding problem and becomes a systems problem.
This is a production-grade architecture for a Python-native scraping system — specifically, an Amazon multi-region price comparator that holds up at scale. We’ll walk the full stack: Streamlit UI, FastAPI backend, Inngest for orchestration, BeautifulSoup for parsing, Bright Data residential proxies, MongoDB for storage, Qdrant for vector search, LangChain + OpenAI for the AI query layer, all containerized with Docker. The patterns generalize to any large-scale, blocked-by-default scraping target.
📚 Table of contents
- Why scraping at scale is hard
- The architecture in one diagram
- The full tech stack
- Residential proxies and why they matter
- Parsing pages with BeautifulSoup — the AI trick
- Storing in MongoDB + Qdrant for hybrid retrieval
- Orchestrating with Inngest
- The LangChain AI query layer
- Dockerizing the whole stack
- Best practices and ethics
- Common mistakes
- Frequently asked questions
⚔️ Why scraping at scale is hard
A naive script fails on five fronts in the first month:
- IP blocks — same IP hits Amazon 50 times, gets blacklisted
- CAPTCHAs and bot-detection — trigger after suspicious patterns
- Geo-localized content — the same product looks different in France vs UAE
- Session expiration — cookies and tokens die mid-scrape
- Inconsistent HTML — A/B tests, region-specific layouts, dynamic JS
Solving any one of these is easy. Solving all five reliably is what separates a script from a system.
🗺️ The architecture
- User enters an Amazon ASIN and target countries in the Streamlit UI.
- Streamlit calls the FastAPI backend, which dispatches an Inngest job.
- Inngest orchestrates the scrape function across countries in parallel.
- Each scrape uses the Bright Data residential proxy from the chosen country.
- Raw HTML comes back; BeautifulSoup extracts the product fields.
- Results land in MongoDB (canonical) and Qdrant (embeddings for AI search).
- UI displays the multi-region comparison.
- Optional AI chat queries Qdrant + MongoDB via a LangChain agent.
🧰 The tech stack
Application layer
- Streamlit — the UI
- FastAPI — the backend
- Inngest — orchestration and monitoring
Scraping layer
- Bright Data residential proxies
- BeautifulSoup for HTML parsing
- LXML as the parser backend
Storage layer
- MongoDB — canonical product records
- Qdrant — vector embeddings for semantic search
AI layer
- OpenAI for embeddings and the chat model
- LangChain for the tool-using agent
- Tools: vector search, MongoDB lookup, on-demand re-scrape
🌐 Residential proxies
The single decision that makes the whole project possible. Bright Data’s residential network proxies your requests through 60M+ real consumer devices across 190+ countries. From Amazon’s perspective, the traffic looks like a normal user from Germany, Canada, or the UAE — not a bot.
What you control
- Target country / city
- Session ID (stick with the same exit node across requests)
- Static vs rotating IPs
- Mobile vs desktop fingerprints (premium tier)
Wiring it up is a username/password auth on a proxy URL. In Python with requests:
set proxies={"http": url, "https": url}
and you’re done. 1 GB of traffic costs roughly $20 and goes a long way — product HTML is small.
🔍 Parsing pages with BeautifulSoup
BeautifulSoup wraps the LXML parser with a Python-friendly API. For each product page:
- Fetch HTML via the proxy.
- Parse with BeautifulSoup.
- Find specific tags:
span#productTitle,.a-price-whole,a#bylineInfo, etc. - Regex-match within element text for prices and currencies.
- Build the canonical JSON record.
The AI shortcut
Save one raw HTML file from a typical product page. Hand it to Claude or GPT and say: “Generate a BeautifulSoup scraper that extracts title, price, brand, rating, and review count. Use stable tags only.” You get production-ready parsing logic in seconds, and the model is great at spotting which selectors are stable across Amazon’s A/B tests.
Even better long-term: extract product data from the embedded JavaScript object in the DOM rather than visible HTML. It changes less often than the rendered markup.
🗄️ MongoDB + Qdrant: hybrid storage
Two stores, two jobs. MongoDB is the source of truth for product records. Qdrant indexes AI-friendly embeddings of the same products for semantic search.
MongoDB
- One document per product per region
- Historical price log subdocuments
- Indexed on ASIN + country
- Cheap to query with SQL-style filters
Qdrant
- Vector embeddings of product title + description
- Semantic search (“cordless drill 18V”) returns nearby items
- Used by the AI agent for fuzzy lookup before falling back to MongoDB
- Local instance via Docker for development
🎼 Orchestration with Inngest
Inngest is an event-driven orchestrator that handles the messy parts of running scrape jobs at scale — retries, parallelism, logging, monitoring, scheduling. You wrap functions as Inngest steps; Inngest gives you a dashboard with run history, retry control, and step-level visibility.
What Inngest gives you for free
- Automatic retries on transient failures
- Per-step traces in a dashboard you can actually use
- Re-run a failed run from the UI
- Scheduled scrapes via cron
- Concurrency limits to avoid hammering a proxy budget
🤖 The LangChain query layer
Instead of forcing users to write SQL or Mongo queries, expose a chat interface backed by a LangChain agent with three tools:
- vector_search: semantic search in Qdrant
- product_lookup: precise MongoDB lookups by ASIN/country
- trigger_scrape: queue a fresh scrape if data is stale or missing
The agent decides which tool fits the user’s question. Vague queries (“what’s the price of an iPad?”) get semantic search; specific ones (“ASIN B07XYZ in Germany”) go straight to MongoDB; missing data triggers a fresh scrape.
🐳 Dockerizing the whole stack
Five services to coordinate (Streamlit, FastAPI, Inngest, MongoDB, Qdrant). Running them by hand each
development session is a non-starter. A single docker-compose.yml
defines them all and brings them up with docker compose up.
Compose-file contents
- mongodb — with named volume for persistence
- qdrant — same pattern
- api — FastAPI image, depends on mongodb + qdrant
- inngest — orchestration server
- ui — Streamlit, exposed on a known port
Bonus: have Claude or Cursor generate the compose file from a description of your services. It’s a sweet spot task for LLMs — well-trodden territory with stable conventions.
✅ Best practices and ethics
- Respect
robots.txtand target site terms of service - Throttle requests — don’t fire thousands of requests per second even if you can
- Cache aggressively — re-scrape only when data is stale
- Persist the raw HTML for debugging when parsing breaks
- Use stable selectors first, regex fallback second
- Pin proxy spending limits to avoid surprise bills
- Add monitoring alerts for sudden parse-failure spikes
- Use the data ethically — for personal use or with proper licensing
❌ Common mistakes
- Trying to scrape Amazon (or similar) without proxies and burning your home IP
- Storing only the parsed JSON and losing the ability to debug parsing failures
- Skipping the orchestrator and trying to handle retries with a try/except loop
- Indexing everything in Qdrant and nothing in MongoDB — you lose precise lookup
- Embedding the raw HTML instead of clean text — vector search degrades
- Running scrapes synchronously and waiting 30 seconds for the UI to respond
- Letting an AI agent
trigger_scrapein a loop without budget caps
Conclusion
A production scraping system is a small distributed system. Proxies for evasion, orchestrator for reliability, dual storage for precise + semantic queries, AI on top for usability. None of the individual pieces are exotic; the value is in how they fit together.
Apply this pattern to any large-scale, blocked-by-default scraping target — e-commerce competitive intel, real-estate listings, jobs, search results. The architecture is roughly the same; swap the parser, swap the selectors, keep the rest.
Related reading: traditional RAG vs vectorless RAG — MCP explained: build your own server — LangChain review
Explore More on DevShelf
-
Defensive Python: Edge Cases and Validation
The validation and edge-case habits that prevent your scraping pipeline from blowing up on malformed HTML and unexpected responses.
-
Track Brand Visibility in AI Search
Apply the same scraping architecture to monitor brand mentions across ChatGPT, Perplexity, Gemini, and Copilot at scale.