57 subjects, 16,000 multiple-choice questions covering STEM, humanities, social sciences, and professional areas like law and medicine. Think of it as a college entrance exam for AI. Scoring guide: 25–40% = barely above random guessing (25% is chance). 60–80% = strong. 80–90% = approaching human expert level. 90%+ = exceeds average human performance. Leading models now score 91–94%, making MMLU “saturated” — it can no longer distinguish between frontier models.

MMLU-Pro: A harder successor with 10 answer choices instead of 4, requiring chain-of-thought reasoning. Scores are typically 20–30 points lower than MMLU, making it better for distinguishing current models.

GPQA (Graduate-Level Problem QA): PhD-level questions in physics, chemistry, and biology. Even domain experts only score ~65%. Model scores in the 40–60% range indicate strong reasoning. This is the “hard” knowledge benchmark.

164 Python programming problems where the model writes a function and it’s tested against unit tests. Measures pass@1 (does the first attempt pass?). Scores: 30–50% = basic coding. 50–70% = competent. 70–90% = strong. 90%+ = frontier (and the benchmark is becoming saturated). Note: HumanEval tests isolated function writing, not real-world software engineering.

SWE-bench Verified tests whether a model can solve real GitHub issues from popular open-source projects — understanding codebases, writing patches, and passing test suites. This is much harder than HumanEval. Scores of 30–50% are currently state-of-the-art. SWE-bench is the closest benchmark to “can this model actually do engineering work?”

GSM8K: 8,500 grade-school math word problems. Tests basic multi-step arithmetic reasoning. Frontier models now score 95%+, so it’s becoming saturated. Still useful for testing smaller models.

MATH: Competition-level math problems (AMC, AIME, Olympiad). Much harder. Scores of 50–70% are strong; 70%+ is exceptional. This benchmark genuinely tests mathematical reasoning.

BBH (BIG-Bench Hard): 23 challenging tasks from the BIG-Bench suite where models previously lagged behind humans. Tests logical reasoning, causal understanding, and multi-step inference.

IFEval (Instruction Following Eval): Tests whether the model follows specific formatting instructions (“respond in JSON,” “use exactly 3 paragraphs”). Critical for production use where output format matters.

ARC (AI2 Reasoning Challenge): Science questions from grades 3–9. The “Challenge Set” requires complex reasoning beyond fact retrieval.

Chatbot Arena (by LMSYS) is a platform where users chat with two anonymous models side-by-side and vote for the better response. Over millions of human votes, models receive an Elo rating (like chess). This is widely considered the most meaningful benchmark for real-world model quality because it reflects actual human preferences, not synthetic test questions.

Arena Elo is a relative ranking, not an absolute score. A model with Elo 1250 will beat a model with Elo 1150 roughly 64% of the time. The top 5 models are typically within 30–50 Elo points of each other. Arena scores are separated by category: overall, coding, math, hard prompts, creative writing — a model might rank #1 in coding but #10 in creative writing.

Zero-shot: The model gets only the question. “What is the capital of France?” No examples, no hints.

Few-shot (e.g., 5-shot): The model gets 5 example question-answer pairs before the actual question. This “primes” the model and typically produces higher scores because the model learns the expected format from the examples.

The difference can be 5–15 percentage points on the same benchmark.

If Model A reports “MMLU: 85% (5-shot)” and Model B reports “MMLU: 80% (0-shot),” you cannot directly compare them. Model B might actually be better — it just wasn’t given examples. Always check whether the card specifies the shot count. When comparing models on a leaderboard, make sure all models were tested under the same conditions (same shot count, same prompt template).

The original Open LLM Leaderboard used: ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K. These benchmarks became saturated — top models scored 85–95% on most of them, making it impossible to distinguish between frontier models. Worse, some models showed signs of data contamination (training on benchmark test data), inflating scores artificially.

The updated leaderboard uses harder alternatives: MMLU-Pro, IFEval, BBH, MATH, GPQA, MuSR. These span a wider score range (40–85%), are less contaminated, and test genuine reasoning rather than pattern matching. If a model card shows v1 benchmarks (ARC, HellaSwag), the scores may be outdated or inflated. Look for v2 benchmarks for more reliable comparisons.

Cherry-picking: The card only shows benchmarks where the model wins and omits standard ones where it underperforms. A card that shows 5 custom benchmarks but no MMLU, no HumanEval, and no Arena score is likely hiding something.

No comparison baseline: Showing a score without context. “85% on our internal eval” means nothing if you don’t know what other models score on the same eval.

Contamination signals: A small model matching or beating much larger models on specific benchmarks. A 7B model scoring 92% on MMLU should make you skeptical.

A trustworthy model card: shows standard benchmarks (at least MMLU/MMLU-Pro, one coding benchmark, one reasoning benchmark), includes comparison with the base model and competing models of similar size, specifies testing conditions (shot count, prompt template), and acknowledges weaknesses (“this model underperforms on multilingual tasks”).

Step 1: Identify which benchmarks are reported. Are they standard (MMLU, HumanEval) or custom?
Step 2: Check the testing conditions (shot count, v1 vs v2 benchmarks).
Step 3: Compare against models of the same size class, not just the leaderboard top.
Step 4: Check for Arena Elo if available — it’s the most holistic measure.
Step 5: Note what’s missing. No coding benchmark? The model may not be good at code.