How Chain-of-Thought Works
Instead of asking "What is 17 * 24?", chain-of-thought prompting adds "Think step by step" or provides an example showing intermediate reasoning. The model then outputs: "17 * 24 = 17 * 20 + 17 * 4 = 340 + 68 = 408." The explicit steps reduce arithmetic and logical errors.
Google Research published the original CoT paper in January 2022, showing that simply adding "Let's think step by step" to prompts improved GSM8K math accuracy from 17.7% to 78.7% on PaLM 540B. The technique works because it forces the model to allocate compute tokens to intermediate reasoning instead of jumping directly to an answer.
When CoT Helps (and When It Does Not)
CoT is most effective on multi-step reasoning: math, logic puzzles, code debugging, and structured analysis. It adds little value for simple factual lookups ("What is the capital of France?") or creative generation where there is no single correct answer.
For Claude, the thinking block (extended thinking) is a built-in CoT mechanism — the model reasons internally before responding. For ChatGPT, manual CoT prompting ("explain your reasoning before answering") achieves a similar effect.
Implementing CoT in Production
In production systems, CoT has a cost: more output tokens = higher API bills. The trick is selective application. Use CoT for complex queries (detected via query classification) and skip it for simple ones. Many teams use a lightweight model to classify query complexity, then route hard questions to a CoT pipeline and easy ones to direct generation.