In January 2022, Jason Wei and colleagues at Google Brain published the paper that launched the reasoning revolution. The core idea is deceptively simple: instead of asking a model to jump directly from question to answer, provide exemplars that include intermediate reasoning steps. The model then learns to generate its own reasoning steps for new problems. Testing on Google’s PaLM (540B parameter) model, chain-of-thought prompting with just 8 exemplars achieved state-of-the-art on the GSM8K math benchmark — surpassing a fine-tuned GPT-3 with a verifier. The improvement was dramatic across three categories: arithmetic reasoning (GSM8K, SVAMP, MultiArith), commonsense reasoning (StrategyQA, CSQA), and symbolic reasoning (last letter concatenation, coin flip). Crucially, CoT only helps with large models (100B+ parameters). Smaller models generate incoherent chains that hurt performance.

Just two months after Wei et al., Kojima et al. made a surprising discovery: you don’t need hand-crafted exemplars at all. Simply appending “Let’s think step by step” to the prompt triggers chain-of-thought reasoning in large language models. This zero-shot CoT approach is remarkably effective: on MultiArith, accuracy jumped from 17.7% to 78.7% with InstructGPT. On GSM8K, it went from 10.4% to 40.7%. The technique works in two stages: (1) append “Let’s think step by step” and let the model generate reasoning, then (2) append “Therefore, the answer is” to extract the final answer. While zero-shot CoT doesn’t match few-shot CoT performance (no exemplars to guide the reasoning format), it’s dramatically simpler to use. No need to craft domain-specific exemplars — one magic phrase works across tasks.

Standard CoT uses greedy decoding: the model generates one reasoning path and returns one answer. But complex problems often have multiple valid approaches. What if the model takes a wrong turn in its reasoning? Self-consistency (Wang et al., 2022, published at ICLR 2023) addresses this by: (1) Sampling multiple reasoning paths using temperature > 0 (e.g., temperature = 0.7, sample 40 paths). (2) Extracting the final answer from each path. (3) Taking the majority vote — the most common answer across all paths wins. The intuition: if a problem has a correct answer, multiple different reasoning approaches should converge on it. Wrong answers are more random and spread across different values. Results were striking: GSM8K improved by +17.9%, SVAMP by +11.0%, AQuA by +12.2%. Self-consistency is now the standard approach for any reasoning task where you can afford the extra inference cost.

Why does generating intermediate text improve reasoning? Several theories: External working memory — the chain of text serves as scratch space. Each token generated becomes part of the context for the next token, effectively giving the model more “memory” to work with. Problem decomposition — CoT breaks one hard problem into many easy sub-problems. Each step is simple enough for the model to handle in a single forward pass. Distributional shift — training data contains many examples of step-by-step reasoning (textbooks, tutorials, Stack Overflow). CoT prompts the model to access this “reasoning mode” of its training distribution. Attention allocation — intermediate steps help the model attend to the right parts of the problem at each stage, rather than trying to attend to everything at once. Error localization — when reasoning fails, you can identify which step went wrong, enabling targeted correction.

Researchers have developed many CoT variants: Least-to-Most Prompting (Zhou et al., 2022) — first decompose the problem into sub-problems, then solve each sub-problem sequentially, using previous answers as context. Particularly effective for problems requiring compositional generalization. Complexity-Based Prompting (Fu et al., 2023) — among multiple sampled reasoning paths, select the ones with the most reasoning steps (highest complexity). The intuition: longer reasoning chains tend to be more thorough. Active Prompting (Diao et al., 2023) — automatically select the most informative exemplars for CoT by identifying questions where the model is most uncertain. Auto-CoT (Zhang et al., 2022) — automatically generate CoT exemplars using zero-shot CoT, eliminating the need for manual exemplar creation. Clusters questions by similarity and generates demonstrations for each cluster.

Implementing CoT in production requires attention to several details: Prompt structure — for few-shot CoT, include 4–8 exemplars that match your task domain. Each exemplar should show the question, step-by-step reasoning, and a clearly marked final answer. Answer extraction — use a consistent format like “The answer is X” or “Therefore: X” to make parsing reliable. Regex extraction is common. Temperature — for greedy CoT, use temperature = 0. For self-consistency, use temperature = 0.5–0.8 to get diverse paths. Cost management — CoT generates more tokens (reasoning + answer), increasing cost. Self-consistency multiplies this by N samples. Budget accordingly. Streaming — for user-facing applications, you can stream the reasoning steps to show “thinking” progress, or hide them and only show the final answer.

CoT is powerful but not a silver bullet: Faithfulness problem — the generated reasoning may not reflect the model’s actual computation. The model might arrive at the right answer for wrong reasons, or generate plausible-looking reasoning that leads to a wrong answer. The chain is a rationalization, not necessarily the true reasoning process. Error propagation — if an early step is wrong, all subsequent steps build on that error. Unlike humans who can catch and correct mistakes, standard CoT has no self-correction mechanism. Model size dependency — CoT only helps models with 100B+ parameters. Smaller models generate incoherent chains that actually hurt performance (an “inverse scaling” effect). Computational overhead — generating reasoning tokens is expensive. For simple questions, CoT wastes compute. Not true reasoning — CoT improves pattern matching over reasoning-like text, but the model still doesn’t truly “understand” logic.

Chain-of-thought prompting was the spark that ignited the reasoning revolution. Its impact extends far beyond the original paper: It proved reasoning is elicitable — LLMs have latent reasoning capabilities that can be activated through prompting. This changed how we think about model capabilities. It established the paradigm — “think before you answer” is now the default for any complex task. Every modern reasoning technique builds on this idea. It inspired training approaches — OpenAI’s o1/o3 models are trained to generate “thinking tokens” — essentially CoT that’s been baked into the model through reinforcement learning. It changed benchmarking — reasoning benchmarks (GSM8K, MATH) became the primary measure of model capability, shifting focus from perplexity to problem-solving. Looking ahead: CoT is the foundation. Tree-of-Thought (next chapter) extends it with search. Test-time compute (Chapter 4) trains it into the model. Verification (Chapter 5) validates it.