MMLU tests broad academic knowledge across 57 subjects — from abstract algebra to world religions. Each question is multiple-choice (A/B/C/D) with a single correct answer. It was designed to measure how much a model “knows” across the full breadth of human knowledge.

14,042 questions across 57 tasks. Random baseline: 25% (4 choices). Human expert baseline: ~89.8%. Current frontier models cluster above 88%, making MMLU essentially saturated for differentiating top models.

MMLU was the first widely-adopted general benchmark and remains a standard reporting metric. But its saturation means it can no longer differentiate frontier models. A model scoring 87% vs 89% on MMLU tells you almost nothing about which is better for your use case.

164 Python programming problems with function signatures and docstrings. The model generates the function body, which is tested against unit tests. Metric: pass@k (probability of at least one correct solution in k attempts). Most frontier models now score above 85%.

• HumanEval+: Same problems, 80x more test cases to catch edge cases
• MBPP: 974 crowd-sourced Python problems (easier)
• LiveCodeBench: Fresh problems from competitive programming sites, updated monthly to resist contamination

HumanEval problems are now in most training datasets. LiveCodeBench showed models dropping 20–30% on problems released after their training cutoff — proving that high HumanEval scores partly reflect memorization, not genuine coding ability.

SWE-bench presents real GitHub issues from popular Python repositories (Django, Flask, scikit-learn, etc.) and asks models to generate patches that pass the repository’s test suite. This tests end-to-end software engineering: reading code, understanding context, writing fixes, and passing tests.

• SWE-bench Lite: 300 curated, well-specified issues (easier subset)
• SWE-bench Verified: Human-validated subset with clear specifications
• SWE-bench Pro: Private, harder subset — GPT-5 and Claude Opus 4.1 both score only ~23%

448 expert-crafted questions in biology, physics, and chemistry. Questions are designed to be “Google-proof” — you can’t find the answer by searching. Even domain experts with internet access only score ~65%. Current leader: Gemini 3 Pro at 92.6%.

GSM8K: 8,500 grade-school math word problems. Status: saturated (frontier models score 95%+). Replaced by MATH (competition-level problems) and AIME (American Invitational Mathematics Exam) for harder reasoning tests.

Reasoning benchmarks test whether models can chain logical steps, not just recall facts. A model that scores well on GPQA demonstrates genuine multi-step reasoning ability — the kind needed for complex real-world tasks like research, analysis, and problem-solving.

LMSYS Chatbot Arena presents users with two anonymous model responses side by side. Users pick which they prefer. Preferences are aggregated into ELO ratings (like chess rankings). As of January 2026: nearly 5 million votes across 296 models.

Arena avoids the contamination problem because every query is unique. It measures what users actually care about — helpfulness, clarity, accuracy — rather than performance on artificial tasks. It’s become the most trusted ranking for comparing frontier models.

• Style bias: Users prefer longer, more verbose answers even when shorter ones are more accurate
• Demographic skew: Voters are mostly tech-savvy English speakers
• Task distribution: Heavily weighted toward general chat, not domain-specific tasks
• No granularity: “Which is better?” doesn’t tell you why or at what

Three forces drive saturation:

1. Data contamination: Benchmark questions leak into training data
2. Optimization pressure: Labs specifically optimize for benchmark scores
3. Capability ceiling: Models genuinely improve and exceed the benchmark’s difficulty level

• Cherry-picked benchmarks: Only showing scores where the model excels
• No error bars: Single-run scores without confidence intervals
• Old benchmarks only: Reporting MMLU/GSM8K but not harder alternatives
• No comparison to baselines: Scores without context are meaningless
• Benchmark-specific fine-tuning: Models optimized for the test, not the task

1. Build your own eval set from real production data (50–200 examples)
2. Test on multiple benchmarks across different capabilities
3. Use contamination-resistant benchmarks (LiveCodeBench, SWE-bench Pro)
4. Check Chatbot Arena for general quality ranking
5. Run your own A/B tests with real users

Benchmarks are necessary but not sufficient. They give you a starting point for model selection and a common language for comparison. But the only benchmark that truly matters is performance on your specific task with your specific data. Everything else is a proxy.