RunPod Serverless RAG Pipeline: Deploy LLMs and Embeddings in 2 Minutes

Introduction

You have a clean RAG idea, you’ve written half of the code, and then you hit the same wall every AI developer hits at some point—you need a GPU. Not a hobby card, a real one. An H100, an A40, maybe even an RTX 5090 for a few hours of fine-tuning. Buying hardware is expensive and almost always sits idle. Provisioning on a hyperscaler like AWS or GCP buries you under VPCs, IAM policies, and quotas for a week before you even pull a container.

RunPod is the platform that has quietly become the default answer to this problem for AI researchers, startups, and educators in 2026. With its serverless infrastructure, you can go from idea to deployed model in under two minutes, pay only for the seconds your model is hot, and stop worrying about scaling entirely.

In this guide, we’ll walk through how to use RunPod Serverless to build a full retrieval-augmented generation (RAG) pipeline end to end—deploying an embedding model, a large language model, and wiring them up with LangChain to answer questions over your own documents.

📚 Table of contents

🔧 The GPU problem that RunPod solves

Modern AI work—fine-tuning, agentic workflows, RAG, multimodal pipelines—all share one appetite: GPUs. Today, you have three uncomfortable options:

RunPod’s pitch is simple: skip the friction. Pick a model, click deploy, get a URL. Pay only for the seconds your GPU is actually running. Scale automatically with traffic. No tickets, no Terraform, no surprise bills the size of a car.

⚡ What is RunPod Serverless?

RunPod has two main products. Pods are dedicated GPU instances you SSH into—great for training and long-running notebooks. Serverless is the other half: managed, autoscaling endpoints that wake on demand, run your model, and shut down when traffic dies. You pay per second of compute.

🏗️ Architecture of the RAG pipeline we’ll build

Our goal is a working RAG pipeline over a 33-page climate-change PDF. The flow looks like this:

The two RunPod endpoints become the only inference infrastructure you need—no GPU on your laptop, no Triton or TGI setup, no Kubernetes.

🪪 Step 1 — Set up your RunPod account

1. Go to runpod.io and sign up.
2. Add a small amount of credit—$10 is plenty for following this guide end to end.
3. Open Settings → API Keys and create a new key. Save it; you’ll use it from Python.

👉 Treat your RunPod API key like any other secret. Don’t commit it to GitHub, and rotate it if you share it on a stream or video.

🧬 Step 2 — Deploy an embedding model with Infinity

In the RunPod dashboard, open Serverless → Repos. You’ll see a gallery of templates: vLLM, Faster-Whisper, Ollama, Infinity embeddings, video and image models, and more. Pick Infinity Embedding—a high-throughput, OpenAI-compatible text embedding and reranking server.

Click Next, then Create Endpoint. Within seconds, RunPod provisions a serverless worker behind a unique endpoint ID. Copy that ID and the base URL—you’ll need both.

🧪 Step 3 — Test the embedding endpoint from Python

Create a quick sanity-check script. The endpoint accepts an OpenAI-compatible request body, so the integration is trivially simple.

import os, requests

ENDPOINT_ID = "your-endpoint-id-here"
API_KEY = os.environ["RUNPOD_API_KEY"]

url = f"https://api.runpod.ai/v2/{ENDPOINT_ID}/run"

payload = {
    "input": {
        "model": "BAAI/bge-small-en-v1.5",
        "input": "What is machine learning?"
    }
}

headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)
print(response.json())

The first call will be slower—RunPod is warming the worker. Subsequent calls take a few hundred milliseconds. Once you see a vector come back, the embedding service is live.

🦙 Step 4 — Deploy an LLM with vLLM (Llama 3.2)

Back in Serverless → Repos, pick the vLLM template—“deploy OpenAI-compatible blazing fast LLM endpoints powered by vLLM.” This is the workhorse for RAG generation.

Hit Create Endpoint. RunPod fetches the model, warms a worker, and gives you an OpenAI-style chat endpoint. From here, anything that speaks the OpenAI chat completion format can talk to your Llama deployment.

🧠 Step 5 — Test the LLM endpoint

Use the same test pattern as before, with a different payload:

payload = {
    "input": {
        "model": "meta-llama/Llama-3.2-3B-Instruct",
        "messages": [
            {"role": "user", "content": "Tell me the best things to do in life."}
        ],
        "max_tokens": 256
    }
}

First request: a slight warm-up wait. Every request after that should be fast and predictable. You now have two production-grade endpoints—embeddings and chat completion—ready to plug into LangChain.

🧩 Step 6 — Build the RAG pipeline with LangChain

Install the LangChain packages plus the RunPod integration:

pip3 install langchain langchain-community langchain-runpod faiss-cpu pypdf

The pipeline itself is straightforward:

from langchain_runpod import ChatRunPod, RunPodEmbeddings
from langchain_community.vectorstores import FAISS
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader
from langchain.chains import RetrievalQA

# 1. Load the PDF
docs = PyPDFLoader("data/climate-change.pdf").load()

# 2. Split into chunks
chunks = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=150
).split_documents(docs)

# 3. Embeddings via RunPod
embeddings = RunPodEmbeddings(
    endpoint_id=EMBED_ENDPOINT_ID,
    api_key=RUNPOD_API_KEY,
    model="BAAI/bge-small-en-v1.5",
)

# 4. Vector store
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("vector_store")

# 5. LLM via RunPod
llm = ChatRunPod(
    endpoint_id=LLM_ENDPOINT_ID,
    api_key=RUNPOD_API_KEY,
    model="meta-llama/Llama-3.2-3B-Instruct",
)

# 6. RAG chain
qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
)

print(qa.invoke("What are hydrogen fuel cells?"))
print(qa.invoke("What is agroforestry?"))

That’s the entire pipeline. The first run is slower because both serverless endpoints are warming up. After that, queries respond in a couple of seconds end to end, including retrieval and generation. You now have a working, production-shaped RAG system without owning a single GPU.

🎯 Who RunPod is built for

💰 Pricing and cost optimization tips

RunPod Serverless bills per second of active GPU time. Costs depend on the card you choose—an H100 costs more per second than an A40 or RTX 5090. New accounts often get a small credit bonus to explore the platform.

• Right-size your GPU. A 3B model fits comfortably on an A40 or RTX 4090—don’t pay H100 prices for it.
• Use idle timeouts. Serverless workers shut down automatically; tune the timeout so they don’t stay warm during long quiet stretches.
• Quantize when you can. AWQ or GPTQ versions of a model often run on smaller, cheaper GPUs with negligible quality loss.
• Cache embeddings. Don’t re-embed the same documents. Persist your FAISS index and reload it.
• Batch requests. Send batched embedding calls instead of one per chunk during ingestion.

⚠️ Common mistakes to avoid

• Hardcoding API keys. Use environment variables or a secrets manager. Never push keys to GitHub—leaked tokens get scanned within minutes.
• Forgetting the cold-start. The first request after idle will be slower. For user-facing apps, keep a minimum worker warm or accept the warm-up latency.
• Picking an oversized model out of habit. A 70B model on H100s is rarely needed for ordinary RAG. Start with a 3B–8B model and measure quality first.
• Skipping retrieval evaluation. Bad retrieval is the number-one source of bad RAG output. Test with a question set before blaming the LLM.
• Not tuning chunking. Chunk size, overlap, and splitting strategy dramatically affect quality. Don’t accept defaults blindly.
• Ignoring logs. RunPod exposes request logs, latency, and worker status. Use them when something feels slow or wrong.

💡 Pro tips for production use

📈 Best practices

• Keep your embedding and LLM endpoints in separate workers—they have different memory and concurrency profiles.
• Set sensible idle timeouts so workers don’t linger and burn credit.
• Always benchmark with your real documents, not a synthetic example, before going to production.
• Log every request and response during development; turn it off and sample once you go live.
• Treat prompts and chunking parameters as configuration, not constants—version them.

🎬 Conclusion

The hardest part of shipping AI in 2026 used to be the infrastructure. RunPod Serverless flips that on its head. Two deployed endpoints, a small LangChain script, a PDF, and you have a working RAG pipeline that scales automatically and costs you nothing when idle. Whether you’re a PhD student racing a deadline, a startup founder validating a demo, or an instructor building a course around real production patterns, this stack collapses the distance between an AI idea and a live endpoint to about two minutes.

Pick a small model, deploy it, send your first request, and start iterating. The infrastructure finally got out of the way—now the only thing between your idea and your users is the work itself.

Related reading: production RAG with Redis and BetterDB — NVIDIA DGX Spark hands-on