Since GPT-3, AI progress has been driven by train-time scaling: bigger models, more data, more training compute. This is captured by Chinchilla scaling laws (Hoffmann et al., 2022): performance improves predictably with model size and training tokens. But train-time scaling is hitting diminishing returns. Training GPT-4 cost an estimated $100M+. Each generation requires roughly 10x more compute for modest gains. Enter test-time compute scaling: instead of making the model bigger, let it think longer on each problem. A smaller model that spends 10 minutes reasoning can outperform a larger model that answers instantly. This is the insight behind OpenAI’s o1 (September 2024): the model generates extended internal reasoning chains — sometimes thousands of tokens — before producing a final answer. More thinking tokens = better answers. Performance scales with inference compute, not just model size.

OpenAI released o1 in September 2024 with the tagline “Learning to Reason with LLMs.” The core mechanism: o1 is trained via large-scale reinforcement learning to generate an extended chain of reasoning (“thinking tokens”) before producing a final answer. Unlike CoT prompting (where we ask the model to reason), o1 has been trained to reason. The RL training teaches it: when to explore alternative approaches, how to recognize and recover from errors, when to try a different strategy, and how to break complex problems into sub-problems. The thinking tokens are hidden from the user — you only see a summary and the final answer. Internally, o1 may generate hundreds to thousands of thinking tokens. On the AIME 2024 math competition, o1 scored 83.3% (top 500 US students), compared to GPT-4o’s 13.4%. On GPQA Diamond (PhD-level science), o1 achieved 77.3%, surpassing human PhD experts. On Codeforces, o1 reached the 89th percentile.

Thinking tokens are the internal reasoning tokens that o1 generates before producing a visible answer. They serve as the model’s “scratch paper.” Key characteristics: Variable length — simple questions may generate 50 thinking tokens; hard problems can generate 10,000+. The model learns to allocate compute proportional to difficulty. Hidden from users — OpenAI shows a summary (“Thinking for 12 seconds...”) but not the full chain. This is partly for IP protection and partly because raw thinking can be messy. Self-correction — unlike standard CoT, thinking tokens include self-correction: “Wait, that’s wrong. Let me try a different approach...” The model has learned to catch its own mistakes. Strategy switching — the model may try one approach, realize it’s not working, and switch to a completely different strategy. This is the backtracking behavior that ToT achieves externally, but o1 does internally. Cost implication — you pay for thinking tokens. A problem that generates 5,000 thinking tokens costs significantly more than a standard GPT-4o call.

How do you train a model to reason? OpenAI uses large-scale reinforcement learning. The process (simplified): Step 1: Base model — start with a strong pre-trained LLM (likely GPT-4 class). Step 2: Generate reasoning traces — for math/code/science problems with known answers, have the model generate many reasoning attempts. Step 3: Reward signal — correct final answers get positive reward; incorrect get negative. Crucially, you can also reward good reasoning steps (process rewards) not just final answers (outcome rewards). Step 4: Policy optimization — use RL algorithms (likely PPO or similar) to update the model to generate reasoning traces that lead to correct answers. Over many iterations, the model learns: which reasoning strategies work, when to explore alternatives, how to self-correct, and how much to think for different difficulty levels. OpenAI reported that o1’s performance improves consistently with both more training compute AND more test-time compute — two independent scaling axes.

In January 2025, OpenAI released o3-mini with a key innovation: configurable reasoning effort. Users can choose low, medium, or high reasoning effort: Low — minimal thinking tokens, fast responses, cheap. Good for simple questions that don’t need deep reasoning. Medium — moderate thinking, balanced cost/quality. Matches o1’s performance on most tasks at lower cost. High — extensive thinking, maximum accuracy. Best for competition-level math and hard coding problems. This is compute-optimal inference: allocate exactly as much compute as the problem needs. Not every question deserves 10,000 thinking tokens. The practical implication: you can use the same model for both quick Q&A and deep reasoning, just by adjusting the effort level. o3-mini with medium effort matches o1 on math, coding, and science benchmarks while being significantly faster and cheaper. With high effort, it exceeds o1 on many benchmarks.

Reasoning models show dramatic improvements on tasks requiring multi-step reasoning: Mathematics — AIME 2024: o1 scored 83.3% vs GPT-4o’s 13.4%. MATH benchmark: o1 achieved 94.8%. These are competition-level problems requiring multi-step proofs. Science — GPQA Diamond (PhD-level): o1 scored 77.3%, surpassing human PhD experts (69.7%). The model can reason through complex physics and chemistry problems. Coding — Codeforces: o1 reached 89th percentile. SWE-bench Verified: o3-mini solved 49.3% of real-world GitHub issues. Novel reasoning — ARC-AGI: o3 scored 87.5%, a benchmark designed to test genuine reasoning on novel tasks. Where they DON’T help much: simple factual questions, creative writing, summarization, translation. These tasks don’t benefit from extended reasoning. Using o1 for “What year was the Eiffel Tower built?” wastes compute.

In January 2025, Chinese AI lab DeepSeek released DeepSeek-R1, an open-source reasoning model that matches o1’s performance on many benchmarks. This was a watershed moment: Performance — R1 matches or approaches o1 on AIME (79.8%), MATH (97.3%), and coding benchmarks. On some benchmarks it exceeds o1. Open weights — unlike o1, R1’s weights are publicly available. Researchers can study, fine-tune, and deploy it. Training recipe revealed — DeepSeek published their training approach: start with a base model, apply RL (using GRPO — Group Relative Policy Optimization), and the model learns to reason. Remarkably, they showed that RL alone (without supervised fine-tuning on reasoning traces) can produce reasoning behavior. Distillation — DeepSeek also released smaller distilled versions (1.5B, 7B, 14B, 32B, 70B) that retain much of R1’s reasoning ability. The 32B distilled model outperforms o1-mini on several benchmarks. Cost — R1 API pricing is roughly 90% cheaper than o1.

Test-time compute scaling is still in its early days. Key directions: Compute-optimal routing — automatically decide how much thinking each question needs. Route simple questions to fast models, hard questions to reasoning models. This is already happening with o3-mini’s effort levels, but will become more granular. Specialized reasoning — models trained for specific domains: legal reasoning, medical diagnosis, financial analysis. Domain-specific RL training on domain-specific problems. Longer horizons — current reasoning models think for seconds to minutes. Future models may reason for hours on very hard problems (research-level math, complex engineering). Multi-modal reasoning — reasoning over images, diagrams, and code simultaneously. Thinking tokens that reference visual elements. Verification integration — tighter integration of reasoning and verification. The model generates a proof, then verifies each step, then revises. This is where process reward models (next chapter) become critical. Efficiency — making thinking tokens cheaper through distillation, quantization, and speculative decoding for reasoning chains.