What is Chain-of-Thought?
Chain-of-Thought (CoT) prompting asks the model to show its reasoning before giving the final answer. Instead of jumping straight to an answer, it works through intermediate steps — and those steps typically lead to more accurate conclusions.
Without CoT
Q: Roger has 5 balls. He buys
2 more cans of 3 balls each.
How many balls does he have?
A: 11Wrong. The model guesses.
With CoT
Q: Roger has 5 balls... Let's
think step by step.
A: Roger starts with 5 balls.
2 cans × 3 balls = 6 balls.
5 + 6 = 11. The answer is 11.Correct — and verifiable.
Zero-shot CoT
The simplest form — just add a reasoning trigger phrase to your prompt. No examples needed.
Common trigger phrases
Let's think step by step.Think through this carefully.First, let me reason about this.Work through this problem before answering.
These phrases are effective on larger models (GPT-4, Claude 3+). On smaller models they are less reliable — use few-shot CoT instead.
Few-shot CoT
Provide examples that include the reasoning trace, not just the final answer. The model learns to mirror the reasoning pattern.
Q: A store has 24 apples. They sell 9.
How many are left?
A: They started with 24 apples.
They sold 9. 24 - 9 = 15.
The answer is 15.
Q: A box holds 8 items. There are 5 boxes.
How many items total?
A: Each box holds 8 items. There are 5 boxes.
8 × 5 = 40. The answer is 40.
Q: [your question here]
A:When CoT helps
Multi-step arithmetic & logic
Word problems, tax calculations, date arithmetic. CoT forces the model to carry intermediate values correctly instead of pattern-matching to a wrong answer.
Complex reasoning tasks
Legal analysis, medical differential diagnosis, code debugging where multiple steps depend on each other.
Auditing model decisions
When you need to verify why the model reached a conclusion, explicit reasoning makes the output inspectable and debuggable.
When CoT hurts
Latency & cost
CoT produces significantly more output tokens. A prompt that normally returns 20 tokens might return 200. This means 10× higher output cost and 10× slower time-to-first-answer. For high-volume, simple tasks this adds up fast.
Simple classification tasks
Asking "Is this spam? Think step by step" for a spam classifier is waste. The model can classify directly. CoT adds tokens without improving accuracy for tasks that don't require multi-step reasoning.
CoT can produce convincing wrong answers
The model can reason fluently to an incorrect conclusion. Explicit reasoning steps look authoritative but may contain logical errors. CoT does not guarantee correctness — it just makes the failure more visible.
Hidden reasoning (extended thinking)
Some modern models — Claude 3.7 Sonnet's "extended thinking", OpenAI o1/o3 — perform CoT internally before generating the visible response. The reasoning tokens are computed but not shown to the user.
Explicit CoT
- Reasoning is in the output
- Inspectable and debuggable
- Costs output tokens (billed)
- You control the reasoning format
Hidden reasoning (o1/extended thinking)
- Reasoning is internal, not shown
- Cleaner final output
- Thinking tokens billed separately
- Better on hard reasoning benchmarks
For production tasks requiring deep reasoning (maths, logic, code), hidden-reasoning models often outperform explicit CoT prompting — but at higher cost per call.
Alternatives to CoT
Programmatic reasoning — use code, not CoT
For arithmetic or logic, ask the model to write and run code instead of reasoning in natural language. Code execution is deterministic; CoT is not. This is what OpenAI's Code Interpreter does.
Write a Python function to calculate this, then call it with the given values.
Tool-based reasoning — use tools, not thought
Give the model a calculator, a search tool, or a database query tool. Factual or computational steps are handled by the tool — which is always correct — while the model handles language and coordination.
Self-consistency — majority vote over multiple CoT runs
Run the same CoT prompt 5× at temperature > 0 and take the most common answer. More expensive, but significantly more accurate on hard reasoning tasks than a single CoT pass.
Notes
Trigger phrase wording matters more on smaller models
On frontier models most trigger phrases work equally well. On smaller models (7B–13B), "Let's think step by step" consistently outperforms less specific triggers in research benchmarks. Use the canonical phrasing when targeting smaller open-source models rather than improvising alternatives.
Calculate self-consistency cost before enabling it
Self-consistency runs the same CoT prompt 5× and takes a majority vote. At GPT-4o output pricing ($0.60/M tokens) with 500 output tokens per pass, five runs cost $0.0015 per query. At 10k queries per day that is $15/day of added cost — calculate your actual numbers before enabling in production.
Extended thinking tokens are billed at a premium rate
Claude's extended thinking and OpenAI's o1/o3 models bill reasoning tokens separately and at a higher rate than regular output tokens. A task costing $0.01 with standard prompting may cost $0.08 with extended thinking enabled. Check the provider's current pricing page before enabling for high-volume tasks.
CoT and structured JSON output conflict in a single call
Asking for both chain-of-thought reasoning and valid JSON output in one prompt often causes the model to embed reasoning prose inside the JSON or produce malformed output. Separate the concerns: first call for CoT reasoning, second call to extract structured output from the reasoning text.
Chain-of-Thought Prompting FAQ
What is chain-of-thought prompting?
Chain-of-thought (CoT) prompting asks the model to show its reasoning before giving the final answer. Instead of jumping straight to an answer, it works through intermediate steps — which typically leads to more accurate conclusions on complex tasks.
What is zero-shot chain-of-thought prompting?
Zero-shot CoT adds a reasoning trigger phrase — like "Let's think step by step" — to the prompt without providing any examples. It works well on large models such as GPT-4 and Claude 3+, but is less reliable on smaller models.
When should I use chain-of-thought prompting?
Use CoT for multi-step arithmetic, logic, legal analysis, code debugging, and any task where intermediate reasoning matters. Avoid it for simple classification tasks where it adds tokens without improving accuracy.
Does chain-of-thought always improve accuracy?
No. CoT can produce convincing but wrong answers — the model may reason fluently to an incorrect conclusion. It makes failures more visible but does not guarantee correctness. It also increases latency and cost significantly.
What are alternatives to chain-of-thought prompting?
Alternatives include programmatic reasoning (asking the model to write and run code), tool-based reasoning (giving the model a calculator or search tool), and self-consistency (running the same CoT prompt multiple times and taking the majority answer).
Quick summary
- CoT: ask the model to reason step-by-step before answering — improves multi-step tasks
- Zero-shot CoT: add "Let's think step by step" — works well on large models
- CoT hurts latency, cost, and is overkill for simple classification — use it selectively
- CoT can reason fluently to a wrong answer — it improves accuracy, not guarantees it
- For arithmetic/logic, programmatic reasoning (code execution) is more reliable than CoT
- Hidden reasoning models (o1, extended thinking) often outperform explicit CoT on hard tasks