Classic LM metrics (perplexity, BLEU) poorly capture multi-step correctness. Reasoning benchmarks provide tasks with reference answers (or verifiers) so you can compute accuracy, pass@k, or partial credit. Good benchmarks stress compositionality (combining skills), generalization (held-out templates), and robustness (paraphrases, distractors). Bad benchmarks are saturated (everyone scores ~100%), leaky (answers in common crawl), or gameable (format hacks). Your job as a practitioner is not just to read leaderboard numbers but to know what capability each benchmark isolates and what setup was used (CoT? tools? self-consistency? test-time compute?).

GSM8K (Cobbe et al., 2021) is a widely used set of ~8K grade-school math word problems with natural language explanations and numeric final answers. It became the flagship benchmark for chain-of-thought improvements because it rewards multi-step arithmetic and planning. MATH (Hendrycks et al.) is a harder corpus of competition-style problems spanning algebra, geometry, calculus, and more, often requiring longer derivations. Together they anchor math reasoning evaluation. Typical reporting includes: accuracy with CoT, pass@k with sampling, and comparisons with/without tools (Python). When models approach ceiling on GSM8K, researchers emphasize MATH and harder subsets to separate models.

HumanEval (Chen et al., OpenAI) popularized pass@k on Python docstring-to-code problems with hidden unit tests. It measures whether the model can produce code that executes correctly — a crisp verifier. Limitations: small size, risk of contamination, and narrow task format. The field has expanded to multi-file benchmarks (e.g., repository-level tasks) where models must navigate context, edit files, and pass broader test suites. For reasoning evaluation, coding benchmarks matter because they combine planning, API knowledge, and debugging loops (especially when paired with sandboxes).

The Abstraction and Reasoning Corpus (ARC), introduced by François Chollet, presents small grid transformation puzzles: a few input/output examples define a latent rule; the model must predict the output for a new input. ARC is designed to stress few-shot generalization and core knowledge priors rather than memorized trivia. ARC-AGI is a related benchmark framing aimed at measuring progress toward more general reasoning; public reporting varies by model generation and evaluation setup. Treat ARC scores as indicators of progress on a particular style of puzzle generalization — not as a complete picture of “AGI.” It remains a valuable counterweight to benchmarks that large models can brute-force via pretraining scale.

GPQA (Rein et al.) is a multiple-choice benchmark of difficult questions in biology, physics, and chemistry, written to be challenging for non-experts and validated by domain experts. GPQA Diamond is a filtered subset often used for cleaner evaluation. Strong scores indicate a combination of factual depth and discriminative reasoning (eliminating distractors). GPQA is frequently cited alongside reasoning models (e.g., OpenAI o-series reporting) because it remains difficult for models that only excel at surface pattern tasks. Like all MC benchmarks, watch for guessing baselines and calibration issues.

BIG-Bench is a large collaborative benchmark spanning diverse tasks. BIG-Bench Hard (BBH) focuses on tasks that remained difficult for early instruction-tuned models, including multi-step reasoning, symbolic manipulation, and weird compositional puzzles. Suites like BBH are useful for regression testing across capabilities after you change a model or a prompt. They also appear in tool-learning papers (e.g., PAL reported gains on selected BBH tasks). The trade-off is complexity: aggregation can hide task-specific regressions, so teams often drill into per-task dashboards.

If benchmark examples or solutions appear in pretraining data, models can succeed via memorization instead of reasoning. Mitigations include: n-gram overlap checks, training-time exclusion experiments, held-out rewrites of questions, and dynamic benchmarks (freshly generated instances). Another subtle issue is answer leakage through style: models learn dataset quirks. For chain-of-thought evaluation, also watch training on CoT traces from the same distribution. Contamination science is imperfect — treat it as risk management: assume some leakage unless you control data; prefer evaluations with private test sets for high-stakes decisions.

Layer evaluations: (1) Core reasoning — GSM8K/MATH + a small private set of your own word problems. (2) Code — HumanEval-style plus internal integration tests. (3) Knowledge-heavy reasoning — GPQA or domain exams. (4) Generalization probes — ARC-style or custom puzzles. (5) Operational metrics — latency, cost, tool failure rates, escalation to human. Automate runs in CI with pinned prompts; store traces (CoT, tool calls) for debugging. For product teams, add human rubrics on a sample of production tasks — benchmarks alone rarely capture UX.