Prompt Engineering for RAG
Bad RAG Prompt
"Use these documents to answer: {docs}
Question: {query}
Answer:"
Problems: Ambiguous instructions. LLM doesn't know to cite sources.
Result: Hallucinations, made-up citations.✓ Better RAG Prompt
You are a helpful assistant. Your job is to answer questions using ONLY
the provided documents. If the answer is not in the documents, say:
"I don't have this information in the available documents."
Always cite your sources using the [Source: doc_title] format.
{DOCUMENTS}
Question: {QUERY}
Answer: Let me search the documents...
[Your answer with citations]Prompt Tips:
- Be explicit about using only provided docs
- Define what to do when info isn't available
- Show citation format with examples
- Use lower temperature (0.3-0.5) for consistency
- Add examples in few-shot format
Guardrails & Structured Output
JSON Mode (Force Structure)
response = client.chat.completions.create(
model="gpt-4",
temperature=0,
response_format={"type": "json_object"},
messages=[{
"role": "user",
"content": """Answer in this JSON format:
{
"answer": "your answer here",
"sources": ["doc1", "doc2"],
"confidence": "high/medium/low",
"needs_more_info": true/false
}
Documents: ...
Question: ..."""
}]
)
result = json.loads(response.choices[0].message.content)
# Guaranteed structure, easy to parseFunction Calling (For Complex Tasks)
tools = [
{
"type": "function",
"function": {
"name": "answer_with_sources",
"parameters": {
"type": "object",
"properties": {
"answer": {"type": "string"},
"sources": {"type": "array", "items": {"type": "string"}},
"confidence": {"type": "string", "enum": ["high", "medium", "low"]}
},
"required": ["answer", "sources", "confidence"]
}
}
}
]
response = client.chat.completions.create(
model="gpt-4",
tools=tools,
messages=[...]
)
# LLM must call function with correct structureCitation Enforcement
def enforce_citations(answer, docs):
"""Verify answer only uses facts from docs."""
import re
# Extract citations from answer
citations = re.findall(r'\[Source: (.*?)\]', answer)
if not citations:
return False, "No citations found"
# Verify cited docs exist
doc_ids = {doc.id for doc in docs}
for citation in citations:
if citation not in doc_ids:
return False, f"Citation '{citation}' not found in docs"
# Use LLM to verify facts are actually in docs
for doc_id in citations:
doc_text = next(d.text for d in docs if d.id == doc_id)
verify_prompt = f"""
Does this text support this claim?
Text: {doc_text}
Claim: {answer}
"""
# Score faithfulness...
return True, "Citations valid"Chain Patterns for Long Documents
Stuff (Simple)
Put all docs in context, ask LLM to answer. Best for <10K tokens.
Map-Reduce (For parallel)
Summarize each doc separately, then combine summaries. Works for large datasets.
Refine (Iterative)
Start with first doc, iteratively refine with next docs. Best quality but slow.
Recommendation: Use "Stuff" for RAG. If docs exceed context, retrieve better quality docs with reranking instead of using chains.
Notes
Set temperature to 0 for factual RAG answers
Higher temperatures introduce variability that's useful for creative tasks but harmful for factual Q&A over documents. At temperature 0, the LLM will stick closely to the retrieved context and produce deterministic outputs that are easier to evaluate and debug. If you need occasional variation (e.g., answer phrasing diversity), use temperature 0.2 at most.
Context window positioning affects retrieval quality
LLMs suffer from "lost in the middle" — they recall content from the beginning and end of the context window better than the middle. If you retrieve 5 chunks, put the most relevant one first (not last). Rerankers help select which chunk deserves position 1. This is a known production-grade optimization that measurably improves faithfulness scores.
Always instruct the model to cite or decline
Without an explicit instruction, LLMs will blend retrieved context with training-data knowledge and hallucinate confidently. Add a system prompt clause: "Answer only using the provided context. If the context does not contain enough information to answer, say 'I don't have enough information'." This single instruction dramatically reduces hallucination rates in production RAG systems.
Streaming adds latency perception complexity
Streaming makes the first token appear faster (better perceived latency) but the total time-to-completion is identical to non-streaming. The real risk is that a streaming answer can start correctly and drift into hallucination mid-response. Monitor the full streamed output, not just the first few tokens, when evaluating answer quality.
RAG LLM Integration FAQ
How do I format the RAG prompt for an LLM?
Place retrieved context in the system prompt or a clearly labeled "Context:" block before the user question. Instruct the LLM to answer only using the provided context and to say "I don't know" if the answer isn't there.
How do I handle token limits when integrating LLMs with RAG?
Count tokens in the retrieved context before building the prompt. If chunks exceed the model's context limit, truncate the lowest-scored chunks first (keep the most relevant ones). Use a tokenizer like tiktoken to count accurately.
Which LLMs work best with RAG?
GPT-4o and Claude 3.5 Sonnet produce the highest-quality answers on retrieved context. For cost-sensitive production use, GPT-4o-mini and Claude Haiku are competitive. Llama 3.1 70B is the best open-source option.
How do I stream LLM output in a RAG pipeline?
Use the LLM provider's streaming API (e.g., OpenAI's stream=True parameter). Yield each token as it arrives and send it to the frontend via server-sent events. LangChain's streaming callbacks simplify this pattern.
Can I use multiple LLMs in a RAG pipeline?
Yes — use a cheaper model for retrieval-intensive steps (query expansion, reranking judgments) and a more capable model for final answer generation. This hybrid routing reduces cost without sacrificing output quality.