Document Ingestion

LangChain Example

from langchain_community.document_loaders import (
    PyPDFLoader,
    TextLoader,
    WebBaseLoader,
    UnstructuredExcelLoader
)

# Load a PDF
loader = PyPDFLoader("report.pdf")
docs = loader.load()  # Returns list of Document objects

# Load a webpage
loader = WebBaseLoader("https://example.com/docs")
docs = loader.load()

# Load plain text
loader = TextLoader("notes.txt")
docs = loader.load()

# Load Excel
loader = UnstructuredExcelLoader("data.xlsx")
docs = loader.load()

Document Structure

Document {
  page_content: "Tax deductions include...",
  metadata: {
    "source": "tax_guide.pdf",
    "page": 3,
    "author": "IRS",
    "date": "2024-01-01"
  }
}

📄 PDFs

Challenge: Images, tables, forms, multiple columns.

from langchain_community.document_loaders import PyPDFLoader
loader = PyPDFLoader("document.pdf")
docs = loader.load()
# Problem: Loses formatting, images as "Document page X"

Better: Use PyMuPDF or pdfplumber for layout-aware extraction. For scanned PDFs, use OCR (pytesseract).

📊 Excel / CSV

Challenge: Tabular data, headers, types.

import pandas as pd
df = pd.read_csv("data.csv")
# Convert to documents
docs = [
  Document(
    page_content=f"Name: {row['name']}, Age: {row['age']}",
    metadata={"source": "data.csv", "row": i}
  )
  for i, row in df.iterrows()
]

Tip: For structured data, consider storing in a database and querying by filters, not RAG.

🌐 Web Pages

Challenge: HTML/CSS noise, JavaScript content, dynamic pages.

from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader("https://example.com/docs")
docs = loader.load()
# Often includes navigation, ads, etc.

Better: Use BeautifulSoup to clean HTML. Extract only main content. For JavaScript-heavy sites, use Playwright.

🗄️ Databases

Challenge: Queries, joins, permissions, realtime updates.

import psycopg2
conn = psycopg2.connect("dbname=mydb")
cursor = conn.cursor()
cursor.execute("SELECT * FROM articles")
docs = [
  Document(
    page_content=row['content'],
    metadata={"id": row['id'], "title": row['title']}
  )
  for row in cursor.fetchall()
]

Note: Keep chunk_id aligned with DB row IDs for easy deletion on update.

Loader selection matters more than you think

PyPDFLoader is convenient but loses layout information — tables become scrambled text, multi-column layouts merge incorrectly. Use pdfplumber or PyMuPDF when your PDFs have structured content. For scanned PDFs, budget for an OCR step; pytesseract works locally but AWS Textract or Google Document AI produce far better results on complex layouts.

Hash documents before embedding to avoid duplicates

Store a SHA-256 hash of each document's content in a simple key-value store (Redis or even a SQLite table) before calling the embedding API. On re-ingestion, skip any document whose hash already exists. Without this, incremental refreshes silently double-embed the same content, degrading retrieval precision and burning API budget.

Web loaders need aggressive cleaning

WebBaseLoader returns the full DOM text including navigation menus, cookie banners, and footer links. Always strip boilerplate with BeautifulSoup's get_text(separator=' ', strip=True) targeting only the main content element. For JavaScript-rendered pages, swap to PlaywrightURLLoader — async rendering adds 2–3 seconds per page but is unavoidable for SPAs.

Attach a chunk_id before the vector store, not after

Set a deterministic chunk_id (e.g., f"{source_hash}_{chunk_index}") during the loading step, before chunking. If you try to reconstruct IDs later, you lose the ability to do targeted deletes when a single document is updated — you end up re-ingesting everything instead.