The three message roles
Modern chat models expect messages in a structured format — not a single blob of text. Each message has a role that tells the model where it came from and how to interpret it.
system
Sets the model's identity, behavior, and constraints for the entire conversation. Evaluated before anything else. This is your most powerful lever — use it to define who the model is and what it must or must not do.
{"role": "system", "content": "You are a senior Python engineer. Answer concisely. Never write pseudocode."}
user
The human's message — your question, instruction, or task. This is what the model responds to. In single-turn prompts this is usually the only message you send.
{"role": "user", "content": "Explain list comprehensions with an example."}
assistant
The model's previous response. In multi-turn conversations, you include prior assistant messages so the model has context. You can also seed an assistant message to prime the model's output style.
{"role": "assistant", "content": "List comprehensions are..."}
Writing a strong system prompt
The system prompt is the foundation of any reliable LLM application. A weak system prompt means the model fills gaps with assumptions — often wrong ones.
Weak system prompt
You are a helpful assistant.
Vague. The model will make up its persona, tone, format, and scope. Every response will be different.
Strong system prompt
You are a customer support agent for Acme Software.
Rules:
- Only answer questions about Acme products.
- If you don't know, say "I'll escalate this to the team."
- Be concise — 2–3 sentences max per answer.
- Never discuss competitors.Specific identity, explicit rules, defined scope, fallback behavior, and format constraint. Predictable across every call.
Temperature
Temperature controls how "random" the model's output is. It scales the probability distribution over tokens before sampling.
| Value | Behaviour | Use for |
|---|---|---|
| 0 | Near-deterministic — always picks the most likely token | Classification, data extraction, factual Q&A |
| 0.3–0.7 | Balanced — some variation but stays on topic | Summarisation, code generation, analysis |
| 0.8–1.0 | Creative — diverse, sometimes surprising outputs | Brainstorming, writing, ideation |
| >1.0 | Very random — often incoherent | Rarely useful in practice |
Start at 0 for anything factual or structured. Add temperature only when you want variety — and test the results.
Top-p (nucleus sampling)
Top-p is an alternative to temperature. Instead of scaling all probabilities, it limits sampling to the smallest set of tokens whose cumulative probability reaches p.
Context window management
Every token in your prompt consumes context. When the window fills, the model either truncates or fails. Here's how to manage it.
Know your limits
GPT-4o: 128k tokens. Claude 3.5 Sonnet: 200k tokens. Llama 3 8B: 8k tokens. 1k tokens ≈ 750 words. A 10-page PDF is roughly 5k–8k tokens.
Position your instructions wisely
Put your most critical instructions in the system prompt (start) and just before the user's question (end). Content in the middle of a large context is attended to less reliably — the "lost in the middle" problem.
Leave room for the output
The context window covers input and output combined. If your prompt is 100k tokens and the model's limit is 128k, you only have 28k tokens left for the response. Set max_tokens explicitly to avoid surprises.
Tokens & cost
API pricing is per token (input + output). Understanding token counts helps you estimate and control cost.
import tiktoken # OpenAI's tokeniser
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Your prompt text here")
print(len(tokens)) # number of tokensRule of thumb: 1 token ≈ 4 characters ≈ 0.75 words in English. Code, JSON, and non-English text can be 2–4× more tokens per character.
Notes
The "lost in the middle" problem is real at scale
Research confirms that LLM attention drops significantly for content positioned in the middle of long contexts (50k+ tokens). For retrieval-augmented apps, put the most relevant retrieved chunks at the start or end of the context window, never buried in the middle.
Set both temperature and top_p explicitly
On some APIs, setting temperature=0 does not fully override top_p. Set top_p=1 explicitly whenever you set temperature=0 to ensure near-deterministic output. OpenAI's documentation recommends changing only one of the two at a time.
max_tokens defaults vary by provider
Anthropic Claude generates until the stop sequence or context limit by default. OpenAI defaults to 4096 output tokens on most models. Always set max_tokens explicitly in production to prevent unexpected truncation or runaway output cost on verbose responses.
Assistant prefill can force output structure on Anthropic's API
Claude allows you to pre-populate the beginning of the assistant response (prefill). Prefilling with {" forces the model to continue in JSON format, more reliably than an instruction alone. This is not available on OpenAI's chat completions API.
Prompt Anatomy FAQ
What are the three message roles in a prompt?
The three roles are system, user, and assistant. The system role sets the model's identity, behavior, and constraints. The user role is the human's message or question. The assistant role contains the model's previous responses, used in multi-turn conversations.
What is temperature in an LLM?
Temperature controls how random the model's output is. A value of 0 produces near-deterministic, consistent results — best for factual tasks. Values of 0.3–0.7 balance creativity and consistency. Values above 0.8 produce more creative but less predictable output.
What is the difference between temperature and top-p?
Both temperature and top-p control output randomness, but in different ways. Temperature scales the probability distribution over all tokens. Top-p (nucleus sampling) limits sampling to the smallest set of tokens whose cumulative probability reaches p. Most practitioners use temperature and leave top-p at 1.0 — avoid tuning both simultaneously.
What is a context window and why does it matter?
The context window is the maximum number of tokens the model can process at once — covering both input and output. It ranges from 8k tokens (smaller models) to 200k tokens (Claude). When the window fills up, the model either truncates earlier content or fails. Instructions at the start and end of the context are most reliably attended to.
How do I reduce token costs in my prompts?
Count tokens before sending using a tokenizer like tiktoken (for OpenAI models). Remove unnecessary preamble, avoid repeating context, and use structured formats that are token-efficient. Remember that 1 token is roughly 4 characters or 0.75 words in English — code and JSON can be 2–4x more expensive per character.
Quick summary
systemsets identity and rules;useris the human turn;assistantis prior model output- A strong system prompt is specific: identity + rules + scope + fallback + format
- Temperature 0 for factual tasks; 0.3–0.7 for balanced; higher only for creative work
- Use temperature or top-p, not both
- Instructions at the start and end of context are attended to most reliably