Human evaluation is the gold standard, but it’s expensive ($5–25 per evaluation), slow (hours to days), and doesn’t scale. If you need to evaluate 10,000 responses across 20 dimensions, human review would cost $50K–$500K and take weeks.

Use a strong LLM as an automated judge. Give it a rubric, the prompt, and the response, and ask it to score the output. Research shows LLM judges achieve 80–90% agreement with human evaluators at 500–5000x lower cost. The same 10,000 evaluations cost $10–$100 and finish in minutes.

Rate a single response on a rubric (1–5 scale). Simplest pattern. Works well for absolute quality assessment. Risk: scores can drift without a reference point.

Show the judge two responses and ask which is better. More reliable than pointwise because relative judgments are easier than absolute ones. This is how Chatbot Arena works.

Provide a gold-standard answer and ask the judge how close the response is. Best for factual tasks where a correct answer exists. Requires maintaining a reference dataset.

Score on multiple dimensions separately (accuracy, tone, completeness, safety). More expensive per evaluation but gives granular insight into where a model fails, not just that it fails.

In pairwise comparisons, judges tend to prefer the first response (or sometimes the second, depending on the model). Mitigation: run each comparison twice with swapped positions and average the results.

LLM judges prefer longer, more detailed responses even when shorter answers are more accurate and appropriate. A concise correct answer often scores lower than a verbose partially-correct one.

Models tend to rate their own outputs higher than outputs from other models. GPT-4 rates GPT-4 outputs more favorably than Claude outputs, and vice versa. Mitigation: use a different model family as judge than the one being evaluated.

JudgeBench research found that even advanced judges perform only slightly better than random on tasks requiring factual verification, logical reasoning, and mathematical correctness. LLM judges are better at style than substance.

A judge prompt needs four components:

1. Role: “You are an expert evaluator for customer support responses”
2. Criteria: Specific, measurable dimensions to score
3. Scale: Clear definitions for each score level (what does a 3 vs 4 look like?)
4. Output format: Structured JSON for automated parsing

1. Create a calibration set of 50–100 examples with human labels
2. Run the LLM judge on the same examples
3. Measure Cohen’s Kappa (agreement beyond chance) — aim for >0.6
4. Analyze disagreements to refine the rubric
5. Repeat until judge-human agreement stabilizes

Don’t use simple correlation (Pearson’s r) — a judge can show perfect correlation while exhibiting systematic bias. Cohen’s Kappa measures true agreement, accounting for chance. Research on 54 LLMs found 23 models achieved “human-like” judgment patterns.

Using GPT-4o as a judge costs roughly $0.005–$0.02 per evaluation (depending on prompt length). Claude Sonnet is similar. Smaller models (GPT-4o-mini, Claude Haiku) cost 10–20x less but with lower reliability on complex judgments.

Use a cheap model for easy cases and an expensive model for hard ones:

1. Run a fast/cheap judge on all responses
2. Flag low-confidence scores (near decision boundaries)
3. Re-evaluate flagged items with a stronger judge
4. This cuts costs by 60–80% with minimal accuracy loss

Recent research (2026) proposes dynamically allocating judge queries based on estimated score variance. Items with high variance (ambiguous quality) get more judge evaluations; clear-cut items get fewer. This achieves significantly better accuracy under fixed budgets.

• Helpfulness & relevance: Is the response useful?
• Tone & style: Does it match brand guidelines?
• Completeness: Did it address all parts of the question?
• Safety screening: Is the content appropriate?
• Coherence: Does the response make logical sense?
• Comparative ranking: Which of two responses is better?

• Factual accuracy: Use retrieval + verification instead
• Mathematical correctness: Use code execution
• Code correctness: Run the code against tests
• Format compliance: Use regex/schema validation
• Latency measurement: Use instrumentation
• High-stakes decisions: Use human review

1. Define your evaluation criteria (3–5 dimensions)
2. Write detailed rubrics with score-level definitions
3. Create a calibration set with human labels (50–100 examples)
4. Test judge agreement (target Cohen’s Kappa > 0.6)
5. Implement tiered judging for cost efficiency
6. Run calibration checks weekly to detect judge drift