Traditional RAG vs. Vectorless RAG: When to Skip Vector Databases

Introduction

For the last few years, “RAG” has meant essentially one architecture: chunk a document, embed the chunks, store them in a vector database, do similarity search at query time, stuff the top-K results into a prompt. It works—and it still works for a huge class of problems—but it has well-known failure modes when documents are long and structured: chunking destroys context, embeddings match the wrong things confidently, and cross-section reasoning falls apart.

Enter vectorless RAG: a newer pattern (popularized by libraries like PageIndex) that throws away chunking and embeddings entirely. Instead of vector similarity, it builds an LLM-readable tree of the document’s structure and lets the model navigate it like a human would navigate a book—table of contents first, then section, then paragraph.

This guide is a side-by-side comparison and a hands-on tutorial: how each approach works, where each shines, where each breaks, a full PageIndex code walkthrough you can copy into a notebook, and the hybrid pattern that production RAG teams are increasingly converging on in 2026.

📚 Table of contents

🧠 A 30-second refresher: what is RAG?

Retrieval-Augmented Generation is the pattern where a language model doesn’t answer from its weights alone—it answers using retrieved context fetched from an external knowledge source at query time. The model becomes a reasoning engine, and your documents become the source of truth.

🧮 How traditional vector RAG works

The classic pipeline is two-phase: build-time indexing and query-time retrieval.

The core algorithm is similarity search: find the nearest vectors. That’s fast, cheap, and works at internet scale—but the metric is “closeness in embedding space,” not “relevance to the question.” And the chunking step is the silent killer: a single logical section can be split across three chunks, and the one chunk that actually answers the query may not make the top-K cutoff.

🌳 How vectorless RAG works

Vectorless RAG takes a different bet: large documents already have structure (table of contents, chapters, sections, sub-sections). If you preserve that structure, you don’t need a vector database—you can let the LLM walk the tree directly, exactly the way a human expert would skim a book before reading the relevant chapter.

👉 No chunking. No embeddings. No vector database. The cost is multiple LLM calls per query instead of one, in exchange for far better section-level reasoning and explainable citations.

📭 What if the PDF has no table of contents?

The first reaction most people have when they hear “tree-based RAG” is: my documents don’t have a clean TOC, so this won’t work for me. That’s a fair worry, but PageIndex (and most tree builders) handle the no-TOC case explicitly.

📌 The key win over vector chunking: splits follow logical boundaries, not token counts. That single difference is why the resulting tree is reason-able by an LLM.

💾 Where do you store the JSON tree?

This is the most-asked question after the first vectorless RAG tutorial: the tree is just JSON, so—where does it live? The honest answer: anywhere that handles structured documents well.

📌 Tree size scales with document length and section count, not with chunk count. A 500-page annual report typically produces a JSON tree in the low hundreds of kilobytes—tiny compared to the vectors a traditional RAG would need.

⚖️ Traditional RAG: strengths and weaknesses

⚖️ Vectorless RAG: strengths and weaknesses

📊 Side-by-side comparison

🛠️ Hands-on: build a vectorless RAG pipeline with PageIndex

Enough theory. Here’s the shortest path from a fresh notebook to a working vectorless RAG over your own PDF, using the open-source PageIndex SDK and OpenAI as the reasoning model. The free tier is generous (around 1,000 documents at the time of writing), which is plenty to prototype.

pip3 install pageindex openai python-dotenv

# .env
PAGEINDEX_API_KEY=your_pageindex_key
OPENAI_API_KEY=your_openai_key

import os, json, time
from dotenv import load_dotenv
from pageindex import PageIndexClient
from openai import OpenAI

load_dotenv()

pipeline = PageIndexClient(api_key=os.getenv("PAGEINDEX_API_KEY"))
client   = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

pdf_path = "./sample_document.pdf"

result = pipeline.submit_documents(pdf_path)
doc_id = result["doc_id"]

# Poll until the tree is ready.
while True:
    status = pipeline.get_document(doc_id)["status"]
    if status == "completed":
        break
    print("Building tree index...")
    time.sleep(5)

tree = pipeline.get_tree(doc_id, node_summary=True)
raw_tree = tree["result"]["tree"]

def walk(nodes, depth=0):
    for n in nodes:
        print("  " * depth + f"{n['title']}  (p.{n.get('page_index')})")
        walk(n.get("nodes", []), depth + 1)

walk(raw_tree)
print(f"Total nodes: {sum(1 for _ in walk(raw_tree))}")

def compress_nodes(nodes):
    out = []
    for n in nodes:
        out.append({
            "node_id": n["node_id"],
            "title":   n["title"],
            "summary": n.get("summary", ""),
            "children": compress_nodes(n.get("nodes", [])),
        })
    return out

TREE_SEARCH_PROMPT = """You are given a query and a document tree (title + summary per node).
Identify which nodes most likely contain the answer. Think step by step.

Query: {query}

Document tree:
{tree}

Return JSON: {{"thinking": "...", "node_ids": ["..."]}}"""

def llm_tree_search(query, tree_nodes):
    prompt = TREE_SEARCH_PROMPT.format(
        query=query,
        tree=json.dumps(compress_nodes(tree_nodes), indent=2),
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

def find_nodes_by_id(nodes, target_ids):
    found = []
    for n in nodes:
        if n["node_id"] in target_ids:
            found.append(n)
        found.extend(find_nodes_by_id(n.get("nodes", []), target_ids))
    return found

ANSWER_PROMPT = """You are an expert document analyst. Answer the question using
ONLY the provided context. For every claim, cite the section title and page number
in parentheses.

Question: {query}

Context:
{context}"""

def generate_answer(query, nodes):
    if not nodes:
        return "No relevant section found in the document."
    context = "\n\n".join(
        f"[{n['title']} (p.{n.get('page_index')})]\n{n.get('text', '')}"
        for n in nodes
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": ANSWER_PROMPT.format(
            query=query, context=context
        )}],
    )
    return resp.choices[0].message.content

def vectorless_rag(query):
    search = llm_tree_search(query, raw_tree)
    nodes  = find_nodes_by_id(raw_tree, search["node_ids"])
    return generate_answer(query, nodes)

print(vectorless_rag("What is the syllabus covered in modern LLM fine-tuning?"))

📌 Notice what’s missing: no vector database, no embedding model, no chunk-size tuning, no reranker. Two LLM calls per query and a JSON file on disk. That’s the entire retrieval stack.

🧪 Try it without code: the chat.pageindex.ai playground

If you want to feel the difference before writing any code, the PageIndex team hosts a free chat playground at chat.pageindex.ai. Upload a PDF, watch the tree get built in seconds, and then ask cross-section questions like “what are the main challenges in pattern recognition?” The UI surfaces the tree-walk explicitly: you can see the model identify relevant section titles, descend into the right subtree, pull page summaries, and only then generate the answer.

Two things tend to stand out the first time you try it. The retrieval is visibly faster than spinning up a vector store on the same PDF, and the citations are real—each answer comes with the section title and page number it came from, not a cosine score you have to take on faith.

📥 When to use traditional RAG

📥 When to use vectorless RAG

🔀 The rise of hybrid RAG

The smartest production teams aren’t picking one architecture and forcing every document through it. They’re running both in parallel, with a router that decides per query (or per document type) which engine to use.

📌 Traditional RAG ≈ scale. Vectorless RAG ≈ reasoning for structure. They are complementary, not competitors.

Common mistakes & pro tips

🎯 Best practices for choosing an approach

Conclusion

Vectorless RAG isn’t a replacement for traditional RAG—it’s a sharper tool for a different job. If your corpus is long, structured, and reasoning-heavy, tree navigation will routinely beat similarity search. If your corpus is huge, heterogeneous, and latency-sensitive, vectors still win.

The PageIndex code path above is short enough that you can stand up a working prototype in an afternoon. That’s the real takeaway: you don’t need a vector DB, an embedding pipeline, or a reranker to ship a credible RAG system over structured documents in 2026. You need a tree builder, a JSON file, and two LLM calls.

The honest answer for most production teams is “both, intelligently routed.” The architectures are complementary, not competitive, and the right pick depends on the document, not the hype.

Related reading: MCP explained: build your own server — how AI actually works (tokens & context engineering) — LangChain review

Explore More on DevShelf

• →
  Learn Agentic AI in 7 Steps

  Step 4 of this path is entirely RAG — where the vector vs. vectorless decision sits in the broader agentic AI stack.

• →
  PG Text Search: BM25 to Replace Elasticsearch

  The keyword-ranking layer that pairs with vector search in hybrid RAG — how BM25 inside Postgres closes the gap.