Website RAG Chatbot: Chat With Any URL Using LangChain and Chroma

Website RAG Chatbot is an intermediate-level project that enables users to interact directly with the content of any public webpage. By simply providing a URL, the system extracts and processes the page content, making it searchable and conversational.

The pipeline begins by scraping the webpage using BeautifulSoup, then splitting the text into overlapping chunks for better context retention. These chunks are converted into embeddings using local Hugging Face sentence-transformers, eliminating the need for external embedding APIs. The embeddings are stored in a Chroma vector database, enabling efficient similarity search.

When a user asks a question, the system retrieves the most relevant context using a Retrieval-Augmented Generation (RAG) workflow powered by LangChain, and generates accurate answers using Groq’s language model. This project demonstrates how modern LLM systems combine scraping, vector databases, and retrieval pipelines to turn static web content into an interactive knowledge source.

Purpose: learn how to turn arbitrary web content into a queryable knowledge base—the same pipeline that powers many AI research and customer-support tools.

Typical use: point it at a long documentation page, a product landing page, or a news article and ask follow-up questions without re-reading the whole thing.

Key ideas you will touch:

• Web scraping — requests + a BeautifulSoup tag-removal pass to strip navigation, scripts, and boilerplate before indexing.
• Chunking — RecursiveCharacterTextSplitter with overlap so facts that span paragraph boundaries land in a single retrievable chunk.
• Local embeddings — sentence-transformers/all-MiniLM-L6-v2 via HuggingFaceEmbeddings; no separate embedding API key needed.
• Vector store + retrieval — Chroma stores the chunk vectors and returns the top-4 matches for each question.
• RAG generation — RetrievalQA (chain type “stuff”) passes retrieved chunks to ChatGroq and surfaces source chunks in an expander.

Example prompts: “What pricing plans are available?” or “Summarise the main argument in the second section.”

Overall flow

User enters a URL → clicks "Scrape & Build Index"
      ↓
requests fetches the page HTML
      ↓
BeautifulSoup strips scripts / nav / footer / boilerplate
      ↓
RecursiveCharacterTextSplitter → chunks (500 chars, 80 overlap)
      ↓
HuggingFaceEmbeddings (local) → Chroma vector store
      ↓
"Indexed N chars" confirmation
      ↓
User types a question → clicks "Ask"
      ↓
Retriever finds top-4 similar chunks
      ↓
ChatGroq (Groq) generates grounded answer
      ↓
Answer + optional Source Chunks on screen

Demo

Step-by-Step Implementation

Follow these steps in order:

1. Set up the project
   • Install Python 3.10+
   • Create a folder and a virtual environment
2. Install dependencies
   • streamlit, httpx, truststore, requests, beautifulsoup4, groq, langchain, langchain-core, langchain-community, langchain-text-splitters, langchain-groq, langchain-huggingface, chromadb, sentence-transformers
   • Pin versions in requirements.txt; the first run downloads the embedding model weights (~22 MB)
3. Secrets
   • Put GROQ_API_KEY in .streamlit/secrets.toml. No embedding key needed.
4. Scrape and clean
   • requests.get with a browser-like User-Agent header reduces bot-blocking
   • Decompose script, style, nav, footer, aside, noscript, and form tags before extracting text
   • Join non-empty lines and cap at MAX_SCRAPED_CHARS to keep prompts manageable
5. Build the index
   • RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=80)
   • HuggingFaceEmbeddings with all-MiniLM-L6-v2 — runs on-device
   • Chroma.from_documents stores the in-memory index in st.session_state
6. Ask with RAG
   • as_retriever(search_kwargs={"k": 4}) for top chunks
   • RetrievalQA.from_chain_type with chain_type="stuff" and return_source_documents=True
   • Store each Q&A turn in st.session_state["history"]
7. Guardrails
   • Warn on empty URL, empty scraped text, and empty question
   • Both _scrape and _ask paths are wrapped in try/except with st.error on failure

Code implementation

The snippet below is the complete app.py. LangChain package layout changes between releases—if imports fail, align pins with the LangChain docs and your requirements.txt.

# Website RAG Chatbot — Streamlit + LangChain + Chroma + Groq
# Save as app.py, add GROQ_API_KEY to .streamlit/secrets.toml, then: streamlit run app.py

import ssl

import httpx
import streamlit as st
import truststore

MAX_SCRAPED_CHARS = 50_000

st.set_page_config(page_title="Website RAG Chatbot", layout="centered")
st.title("Website RAG Chatbot")
st.caption(
    "Enter any public URL, scrape its content, and ask questions about it "
    "using Retrieval-Augmented Generation."
)

_ssl_ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)


def _scrape(url: str) -> str:
    import requests
    from bs4 import BeautifulSoup

    headers = {"User-Agent": "Mozilla/5.0 (compatible; DevShelfHubRAGBot/1.0)"}
    resp = requests.get(url, headers=headers, timeout=15)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    for tag in soup(["script", "style", "nav", "footer", "header", "aside", "noscript", "form"]):
        tag.decompose()
    text = soup.get_text(separator="\n")
    lines = [line.strip() for line in text.splitlines() if line.strip()]
    return "\n".join(lines)[:MAX_SCRAPED_CHARS]


def _build_index(text: str):
    from langchain_community.vectorstores import Chroma
    from langchain_core.documents import Document
    from langchain_huggingface import HuggingFaceEmbeddings
    from langchain_text_splitters import RecursiveCharacterTextSplitter

    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=80)
    docs = splitter.split_documents([Document(page_content=text)])
    embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
    return Chroma.from_documents(docs, embeddings)


def _ask(vector_store, question: str) -> dict:
    from langchain.chains import RetrievalQA
    from langchain_groq import ChatGroq

    llm = ChatGroq(
        api_key=st.secrets["GROQ_API_KEY"],
        model_name="llama-3.3-70b-versatile",
        http_client=httpx.Client(timeout=120.0, verify=_ssl_ctx),
    )
    retriever = vector_store.as_retriever(search_kwargs={"k": 4})
    qa = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff",
        retriever=retriever,
        return_source_documents=True,
    )
    return qa.invoke({"query": question})


url_input = st.text_input("Website URL", placeholder="https://example.com/page")

if st.button("Scrape & Build Index", type="primary"):
    if not url_input.strip():
        st.warning("Please enter a URL first.")
    else:
        with st.spinner("Scraping and indexing — first run downloads embedding model weights…"):
            try:
                raw_text = _scrape(url_input.strip())
                if not raw_text.strip():
                    st.error("Could not extract any text from that URL. Try a different page.")
                else:
                    st.session_state["vs"] = _build_index(raw_text)
                    st.session_state["scraped_url"] = url_input.strip()
                    st.session_state["scraped_chars"] = len(raw_text)
                    st.session_state.pop("history", None)
                    st.success(
                        f"Indexed {len(raw_text):,} characters from {url_input.strip()}"
                    )
            except Exception as err:
                st.error(f"Scraping failed: {err}")

if "vs" in st.session_state:
    st.markdown("---")
    st.markdown(
        f"**Ready.** Ask anything about `{st.session_state['scraped_url']}` "
        f"({st.session_state['scraped_chars']:,} chars indexed)."
    )

    question = st.text_input("Your question", placeholder="What is this page about?")

    if st.button("Ask", type="primary"):
        if not question.strip():
            st.warning("Please type a question.")
        else:
            with st.spinner("Retrieving and generating answer…"):
                try:
                    result = _ask(st.session_state["vs"], question.strip())
                    answer = (result.get("result") or "").strip()
                    sources = result.get("source_documents", [])
                    st.session_state.setdefault("history", []).append(
                        {"q": question.strip(), "a": answer}
                    )
                    st.markdown("**Answer:**")
                    st.markdown(answer)
                    if sources:
                        with st.expander("Source chunks"):
                            for i, doc in enumerate(sources, 1):
                                st.caption(f"Chunk {i}")
                                st.text(doc.page_content[:400])
                except Exception as err:
                    st.error(f"Something went wrong: {err}")

    history = st.session_state.get("history", [])
    if len(history) > 1:
        with st.expander("Earlier Q&A"):
            for turn in reversed(history[:-1]):
                st.markdown(f"**Q:** {turn['q']}")
                st.markdown(f"**A:** {turn['a']}")
                st.divider()

📖 How the Code Works (Step-by-Step)

A walkthrough of app.py from top to bottom.