Chain of Thought Prompting Explained

Chain of thought prompting means asking a model to work through a problem in visible steps before giving a final answer, instead of jumping straight to the answer. The technique comes from a 2022 Google Research paper, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, which showed that models score meaningfully higher on arithmetic and logic problems when prompted to reason step by step rather than answer directly.

The mechanism is straightforward once you see it: a model generates its response one token at a time, and each token it writes becomes part of the context for the next one. If the first thing it writes is the final answer, there is no intermediate reasoning to build on. If the first thing it writes is a reasoning step, that step becomes evidence the next token can use. Asking for steps is not decoration. It changes what the model has available to work with.

What it looks like in practice

The right column does not ask for more effort in the abstract. It asks the model to externalize the intermediate work, which is exactly the part a direct prompt skips.

Where it helps and where it does not

Chain of thought reliably helps on tasks with real intermediate steps: arithmetic, multi-condition logic, code tracing, anything with a policy or rule set to apply against specific facts. The 2022 paper's own results were strongest on math word problems and symbolic reasoning, tasks where a wrong shortcut is easy to take and a written-out path makes the shortcut visible.

It helps far less on tasks that are closer to lookup or classification than reasoning. Asking a model to "think step by step" before naming the capital of a country adds nothing, because there are no real intermediate steps to expose. Recent research, including a 2025 analysis titled The Decreasing Value of Chain of Thought in Prompting, has found the gains from chain of thought narrowing on newer, larger models that already do more implicit reasoning by default, and on tasks where the reasoning was never the bottleneck to begin with. The technique is not obsolete, but it is not universal either, and treating it as a default for every prompt wastes tokens on problems that did not need the ceremony.

Two ways to trigger it

1. Zero-shot: add an instruction like "reason through this step by step before answering" directly to the prompt. Works on most current models with no other setup.

2. Few-shot: show one or two worked examples in the prompt that demonstrate the step-by-step reasoning pattern you want, then ask the actual question. More reliable for unusual or domain-specific reasoning patterns a generic instruction will not reliably produce on its own.

Zero-shot is the right default. Reach for few-shot when the reasoning pattern is specific to your domain, like a particular way of evaluating loan applications or triaging support tickets, where a generic "think step by step" does not tell the model which steps actually matter.

A worked example

Prompt: "A customer wants a refund on a $120 order placed 35 days ago. The policy allows refunds within 30 days, or within 45 days if the item was defective. The customer says the item arrived damaged. Is the refund valid?"

Direct answer, no reasoning requested: a model will often just say "No, because it's past the 30-day window," missing the defective-item exception entirely. Asked to reason step by step first, a typical response walks through: order was placed 35 days ago, standard window is 30 days so this alone would fail, but the customer reports damage, the defective-item exception extends the window to 45 days, 35 days is within that window, so the refund is valid. Same model, same facts, different outcome, because the exception only gets applied when the reasoning is forced into the open where it has to be checked against every condition rather than pattern-matched against the common case.

Reasoning models change the calculus

Purpose-built reasoning models generate an internal chain of thought automatically before producing a final answer, without needing to be asked. For those models, explicitly instructing "think step by step" is often redundant, since the behavior is already built into how they respond. The technique still matters for general-purpose models used without a reasoning mode, and for cases where you want the reasoning itself visible in the output, not hidden, so you can check it rather than trust it. For more on that distinction, see what is a reasoning model.

Either way, the underlying discipline is the same one covered in how long should a prompt be: give the model room to do the work it actually needs to do, and no more than that.

This is one entry in a larger toolkit. For the rest of it, see prompt engineering.

FAQ

Does chain of thought cost more tokens?

Yes, since the model generates the reasoning text in addition to the final answer. For tasks where it changes the answer, that cost is worth paying. For simple lookups, it is pure waste.

Can I hide the reasoning from the end user but still benefit from it?

Yes. Ask the model to reason step by step, then give a final answer on its own line or in a structured field, and only display that field. The reasoning still improves the answer even if nobody reads it.

Is chain of thought the same as few-shot prompting?

No. Chain of thought is about requesting step-by-step reasoning; few-shot is about providing worked examples. They combine well, but you can use either alone, and see few-shot vs zero-shot prompting for how they compare directly.

Does it work for creative writing tasks?

Marginally, and differently. Asking a model to plan a story's structure before writing it can help pacing, but that is closer to outlining than to the step-by-step logical reasoning the original technique targets.