Chain-of-thought and test-time scaling let models generate reasoning. But how do you know the reasoning is correct? This is the verification problem. Consider: a model generates a 10-step math proof. Steps 1–7 are correct, step 8 has a subtle error, and steps 9–10 build on that error. The final answer is wrong, but the reasoning looks plausible. Self-consistency (majority vote) helps but is crude: it only checks the final answer, not the reasoning process. If most paths make the same error, majority vote fails. We need verification: a way to check each step of the reasoning, identify errors, and select the best reasoning path. This is where reward models come in. A reward model is a separate neural network trained to evaluate the quality of reasoning — either the final outcome (ORM) or each step (PRM). OpenAI’s 2023 paper “Let’s Verify Step by Step” showed that step-level verification dramatically outperforms outcome-only verification.

An Outcome Reward Model (ORM) is the simpler approach: it evaluates only the final answer of a reasoning chain. Training: collect many (problem, solution) pairs. For each, label whether the final answer is correct or incorrect. Train a neural network to predict correctness from the full solution text. Usage: generate N reasoning paths, score each with the ORM, select the highest-scored path. Advantages: easy to train (just need final answer labels), cheap to annotate (automated verification for math/code). Disadvantages: provides only a sparse reward signal — one score for the entire solution. Can’t pinpoint where errors occur. A solution with 9 correct steps and 1 wrong step gets the same “wrong” label as a completely wrong solution. This makes RL training inefficient: the model doesn’t know which steps to improve. ORMs also suffer from reward hacking: models learn to produce outputs that score well without actually reasoning correctly.

A Process Reward Model (PRM) evaluates each step of a reasoning chain independently. OpenAI’s 2023 paper “Let’s Verify Step by Step” (Lightman et al.) demonstrated the power of this approach. Training: human annotators label each step of a math solution as “correct,” “incorrect,” or “neutral.” OpenAI collected 800,000 step-level labels across 75,000 solutions (the PRM800K dataset). The PRM is trained to predict the correctness of each step given all previous steps. Results on MATH benchmark: using a PRM to select among solutions solved 78.2% of problems, significantly outperforming ORM-based selection. The PRM catches errors at the exact step where they occur, enabling: better solution selection, more targeted RL training (reward good steps, penalize bad ones), and interpretable feedback (you can see which step the model flagged).

Reward signal density: ORM gives one score per solution (sparse). PRM gives one score per step (dense). Dense signals enable faster RL convergence and better credit assignment. Error localization: ORM says “wrong.” PRM says “wrong at step 4.” This enables targeted correction and interpretable feedback. Training cost: ORMs are cheap to train — just need (solution, correct/incorrect) pairs, which can be auto-generated for math and code. PRMs are expensive — need step-level annotations. OpenAI used human annotators for PRM800K. Annotation scalability: ORM labels can be automated (check final answer). PRM labels traditionally required humans, but recent work uses Monte Carlo estimation: for each step, complete the solution many times and check what fraction reach the correct answer. This automates PRM training. Alignment benefits: PRMs reward correct reasoning processes, not just correct answers. This mitigates “reward hacking” where models reach right answers via wrong reasoning. Compute efficiency: recent research (2024) shows PRMs achieve 1.5–5x better compute efficiency than ORMs when used for test-time search.

The biggest bottleneck for PRMs is collecting step-level labels. Human annotation is expensive and slow (OpenAI’s PRM800K required significant annotation effort). Monte Carlo estimation (MCE) automates this: for each step in a solution, complete the remaining solution many times (e.g., 64 completions) using the base model. The step’s score = fraction of completions that reach the correct final answer. Intuition: if a step is correct, many completions from that point will succeed. If a step is wrong, few completions will succeed (they’re building on a bad foundation). Advantages: fully automated, no human annotators needed. Scales to millions of labels. Limitations: labels are policy-dependent — they reflect the base model’s ability, not ground truth. A step might be correct but the model can’t solve the rest, giving a low score (false negative). Or a step might be wrong but the model “recovers” in later steps (false positive). Despite these limitations, MCE-trained PRMs approach human-annotated PRM performance in practice.

Reward models aren’t just for training — they’re powerful at inference time too. The combination of search + verification is the core of modern reasoning systems: Best-of-N with PRM — generate N solutions, score each step with the PRM, select the solution with the highest minimum step score. More reliable than majority vote because it checks reasoning quality, not just answer agreement. Beam search with PRM — at each reasoning step, generate multiple candidates, score them with the PRM, keep the top-k. This is ToT (Chapter 3) but with a learned verifier instead of LLM-as-judge. Process Advantage Verifiers (PAVs) — a 2024 advance that scores each step relative to alternatives. Instead of “is this step correct?” it asks “is this step better than alternatives?” PAVs achieve 8%+ greater accuracy and 1.5–5x better compute efficiency than ORMs. MCTS + PRM — use the PRM as the value function in Monte Carlo Tree Search. The PRM evaluates partial solutions, guiding the search toward promising branches.

PRMs play a dual role: inference-time verification AND training-time reward shaping. In RL training for reasoning (as in o1/R1), the reward signal determines what the model learns: Outcome supervision (ORM) — reward only the final answer. The model must figure out which steps were good or bad from a single signal. This is the credit assignment problem: with a 10-step solution, which steps deserve credit for the correct answer? RL with sparse rewards is notoriously slow. Process supervision (PRM) — reward each step individually. The model gets immediate feedback on every reasoning step. This solves credit assignment: good steps get positive reward, bad steps get negative reward. RL converges much faster. OpenAI’s research showed process supervision is 5–6x more sample-efficient than outcome supervision. The model learns correct reasoning patterns faster because it knows exactly which steps to reinforce. This is believed to be a key component of o1’s training: RL with process-level rewards, not just outcome rewards.

Verification is powerful but far from solved: Beyond math and code — PRMs work well for math (verifiable answers) and code (test cases). But how do you verify reasoning about ethics, strategy, or creative problems? There’s no ground truth to check against. Verifier reliability — the verifier itself can be wrong. If the PRM has a blind spot, it will consistently miss certain errors. Quis custodiet ipsos custodes? (Who watches the watchmen?) Reward hacking — models can learn to produce outputs that score well on the PRM without actually reasoning correctly. The model “games” the verifier. This is an ongoing arms race. Generalization — PRMs trained on math problems may not generalize to science or coding. Domain-specific PRMs are needed, multiplying training costs. Step boundary detection — PRMs assume clear step boundaries. But natural reasoning doesn’t always decompose into discrete steps. Scaling verification — as reasoning chains get longer (thousands of steps), verification becomes harder. Each step depends on all previous steps, creating compounding uncertainty.