Python Web Scraping: Amazon Competitor Analysis Tool

Introduction

E-commerce brands spend serious money trying to understand what their competitors are doing — pricing, ratings, reviews, market positioning. The tools that surface that data are an entire SaaS category, and most of them charge $200–$1,000+ per month per user.

This is a complete build for a Python web scraping app that powers exactly that kind of product: an Amazon competitor insight tool. Look up your product by ASIN, scrape the top competitors, run AI analysis over the results, surface pricing gaps and review patterns. The architecture (Streamlit + LangChain + Oxylabs + TinyDB) is realistic for production, and the foundation can scale into a real SaaS business with focus.

📚 Table of contents

Why this is a million-dollar idea, honestly

The pitch sounds inflated. The math doesn’t actually require it to be.

• Amazon’s third-party seller market has hundreds of thousands of active brands.
• Existing competitor-insight tools (Helium 10, Jungle Scout, SellerApp) charge $50–$500/month per seat.
• Many of those tools are old, generic, or hard to use. There’s genuine room for a better-targeted product.
• Adding AI summarization on top of raw scraped data is a meaningful product-level upgrade most incumbents haven’t shipped well yet.

A million dollars in ARR is 1,667 customers at $50/month, or 416 at $200/month. With a sharp niche (Amazon sellers in a specific category) and one clearly-better feature, that’s reachable. Not guaranteed, not easy, but reachable.

The stack: Streamlit + LangChain + Oxylabs + TinyDB

• Streamlit — the UI. Five-line Python → running web app. Perfect for internal tools and MVPs.
• LangChain + OpenAI — the AI layer. Summarize products, extract insights, compare competitors.
• Oxylabs — the scraping infrastructure. Handles proxies, anti-bot evasion, parsed JSON output.
• TinyDB — lightweight JSON file database. Fine for an MVP; swap for Postgres when you scale.

For a real production deploy, you’d add FastAPI as a backend, Postgres as the database, and Vercel/Render for hosting — but the MVP works fine with the stack above.

Why scraping Amazon yourself doesn’t work in 2026

A naive requests.get("https://amazon.com/dp/B0XXX") loop hits a wall within minutes:

• Amazon detects IP-based scraping and starts returning CAPTCHA pages.
• After a few hundred requests, the IP gets banned.
• The HTML structure changes weekly; parsers break constantly.
• Different domains (.com, .co.uk, .de) have different layouts.

Services like Oxylabs solve this by maintaining a pool of residential proxies, rotating them automatically, handling CAPTCHA challenges, and (importantly) returning already-parsed JSON instead of raw HTML. You send a product ASIN; you get back structured data. Saves weeks of parser maintenance.

The trade-off: it’s a paid service. For a real product, the cost is justified by the value; for a learning project, free tier credits work.

Project setup with uv

mkdir amazon-competitor && cd amazon-competitor
uv init .
uv add streamlit requests python-dotenv tinydb
uv add langchain langchain-openai

Create .env with your credentials:

OXYLABS_USERNAME=...
OXYLABS_PASSWORD=...
OPENAI_API_KEY=sk-...

The Streamlit input UI

import streamlit as st

def main():
    st.title("Amazon Competitor Insight Tool")
    asin = st.text_input("Product ASIN", placeholder="B08N5WRWNW")
    geo = st.selectbox("Marketplace", ["us", "uk", "de", "ca"])
    domain = st.selectbox("Domain", ["amazon.com", "amazon.co.uk", "amazon.de", "amazon.ca"])
    if st.button("Analyze") and asin:
        run_analysis(asin, geo, domain)

if __name__ == "__main__":
    main()

Run with uv run streamlit run main.py. The app opens at localhost:8501. No HTML, no JavaScript, no frontend framework — that’s Streamlit’s appeal.

Wiring up Oxylabs to fetch product data

Oxylabs’ Amazon scraping API takes an ASIN, returns parsed product JSON:

import os, requests
from dotenv import load_dotenv
load_dotenv()

OXY_URL = "https://realtime.oxylabs.io/v1/queries"
AUTH = (os.environ["OXYLABS_USERNAME"], os.environ["OXYLABS_PASSWORD"])

def scrape_product(asin: str, geo: str, domain: str) -> dict:
    payload = {
        "source": "amazon_product",
        "query": asin,
        "geo_location": geo,
        "domain": domain.replace("amazon.", ""),
        "parse": True,
    }
    r = requests.post(OXY_URL, auth=AUTH, json=payload, timeout=60)
    r.raise_for_status()
    return r.json()["results"][0]["content"]

One function. Send the ASIN, get back a structured product object with title, price, rating, review count, images, bullets, descriptions, seller info, all parsed.

Normalizing the messy product payload

Oxylabs returns a lot of fields. For the analysis you usually want a focused subset:

def normalize(p: dict) -> dict:
    return {
        "asin": p.get("asin"),
        "title": p.get("title"),
        "brand": p.get("brand"),
        "price": p.get("price"),
        "rating": p.get("rating"),
        "reviews_count": p.get("reviews_count"),
        "bullets": p.get("bullet_points", []),
        "description": p.get("description"),
        "category": p.get("category"),
        "image": (p.get("images") or [None])[0],
    }

Scraping competitor lists from search results

Same API, different source:

def search_competitors(query: str, geo: str, domain: str, n: int = 8):
    payload = {
        "source": "amazon_search",
        "query": query,
        "geo_location": geo,
        "domain": domain.replace("amazon.", ""),
        "parse": True,
    }
    r = requests.post(OXY_URL, auth=AUTH, json=payload, timeout=60)
    organic = r.json()["results"][0]["content"]["results"]["organic"]
    return [item["asin"] for item in organic[:n]]

Workflow: scrape the input ASIN, extract its title/category, use those as a search query, get top organic results back as ASINs, scrape each of those as products. End up with the input product plus N competitors, all structured.

AI analysis with LangChain over the scraped data

Once you have structured products, the AI layer earns its keep:

from langchain_openai import ChatOpenAI
import json

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def analyze(my_product, competitors):
    prompt = f"""
    You are an Amazon competitive-analysis assistant. Compare the user's product against the
    competitors. Identify pricing position, rating gaps, missing features in bullets, and three
    concrete improvements the user could make.

    USER PRODUCT:
    {json.dumps(my_product, indent=2)}

    COMPETITORS:
    {json.dumps(competitors, indent=2)}

    Return a structured markdown report with sections: Pricing, Ratings, Feature Gaps,
    Recommendations.
    """
    return llm.invoke(prompt).content

Display the markdown in Streamlit with st.markdown(report). That’s the customer-facing deliverable — structured analysis derived from real scraped data.

Persistence with TinyDB for repeat lookups

Scraping costs money. Cache results so users don’t re-pay for the same ASIN every time:

from tinydb import TinyDB, Query
from datetime import datetime, timedelta

db = TinyDB("cache.json")
products = db.table("products")
Q = Query()

def get_cached(asin: str, max_age_hours: int = 24):
    rec = products.get(Q.asin == asin)
    if not rec: return None
    age = datetime.utcnow() - datetime.fromisoformat(rec["fetched_at"])
    return rec["data"] if age < timedelta(hours=max_age_hours) else None

def set_cached(asin: str, data: dict):
    products.upsert(
        {"asin": asin, "data": data, "fetched_at": datetime.utcnow().isoformat()},
        Q.asin == asin,
    )

For a real SaaS, swap TinyDB for Postgres and add per-user usage limits. The architecture pattern stays the same.

Where to take this next as a product

From MVP to real business:

• Authentication — Clerk or FastAPI Users so each customer has an account.
• Stripe subscriptions — tier the product (free 5 lookups/month, Pro $49/month for unlimited, Agency $199/month for white-label reports).
• Scheduled re-scrapes — track competitor pricing over time, alert on drops.
• Niche down — market the product to one category first (supplements, kitchen, fitness). Specialist tools beat generalists.
• PDF reports — brandable, exportable, shareable. Agencies love these.
• Slack/Email digests — weekly automated competitor watch. Recurring engagement.
• Multi-marketplace — expand to Walmart, Etsy, Shopify, eBay. Same architecture.

❌ Common mistakes

💡 Pro tips

Conclusion

Python web scraping plus AI analysis plus a clear vertical (Amazon sellers) is a working template for a real SaaS business. The hard parts have been outsourced to specialized services (Oxylabs for scraping, OpenAI for analysis), leaving you with the work that actually creates differentiation: UX, niche focus, AI prompt quality, and customer relationships.

The build above produces a working MVP in a weekend. Turning it into a million-dollar business takes a year of focused execution on top of that — but the technical foundation is here, and the architecture scales.

Explore More on DevShelf

• →
  Production-Grade Web Scraping Architecture in Python

  Scale the Amazon scraper into a production system — proxies, orchestration, retries, and the architectural patterns that hold up under load.

• →
  Track Brand Visibility in AI Search

  Apply the same scraping architecture to monitor how ChatGPT, Perplexity, and Gemini mention your brand across countries and queries.