Evaluating NLP systems is fundamentally harder than evaluating other ML tasks. In image classification, a cat is a cat. In NLP, there are many correct ways to express the same meaning. "The cat sat on the mat" and "A feline rested upon the rug" mean the same thing but share almost no words. This means simple word-overlap metrics miss correct answers that use different vocabulary. Fluency, coherence, factuality, and relevance are separate dimensions of quality that no single metric captures. A summary can be fluent but factually wrong. A translation can be accurate but awkward. Human evaluation remains the gold standard, but it's expensive, slow, and subjective — annotators disagree 10–30% of the time on quality judgments. The field has developed dozens of metrics, each capturing a different aspect of quality, and the art of NLP evaluation is knowing which metrics matter for your task.

For classification tasks, the core metrics are precision (of all items predicted positive, how many are actually positive?), recall (of all actually positive items, how many did we find?), and F1 score (harmonic mean of precision and recall). Accuracy is misleading with imbalanced classes — a spam detector that never flags spam gets 95% accuracy if only 5% of emails are spam. F1 balances precision and recall: high F1 requires both to be high. For multi-class problems, there are three averaging strategies: macro F1 (average F1 across classes, treating all classes equally), micro F1 (aggregate TP/FP/FN across classes, equivalent to accuracy), and weighted F1 (weight by class frequency). Macro F1 is preferred when all classes matter equally; weighted F1 when you care more about frequent classes.

BLEU (Bilingual Evaluation Understudy, Papineni et al., 2002) measures n-gram overlap between a generated translation and one or more reference translations. It computes modified precision for 1-grams through 4-grams, clips counts to avoid gaming by repetition, and applies a brevity penalty for translations that are too short. BLEU scores range from 0 to 1 (often reported as 0–100). A BLEU score of 30–40 is considered good for machine translation. BLEU became the standard MT metric because it correlates reasonably with human judgment at the corpus level. But it has significant limitations: it penalizes valid synonyms ("big" vs "large"), ignores word order beyond n-grams, and correlates poorly with human judgment at the sentence level. Despite these flaws, BLEU remains widely used because it's fast, reproducible, and well-understood.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is the standard metric for summarization. While BLEU focuses on precision (how much of the output matches the reference), ROUGE focuses on recall (how much of the reference is captured in the output). ROUGE-1 measures unigram recall, ROUGE-2 measures bigram recall, and ROUGE-L measures the longest common subsequence. For summarization, recall matters more than precision: you want to capture all the important information, even if you include some extra words. METEOR improves on both BLEU and ROUGE by incorporating stemming (matching "running" with "ran"), synonym matching (matching "big" with "large"), and word order penalties. METEOR correlates better with human judgment than BLEU, especially at the sentence level.

BERTScore (Zhang et al., 2020) addresses the fundamental limitation of n-gram metrics: they can't recognize paraphrases. Instead of matching exact words, BERTScore computes cosine similarity between contextual embeddings of tokens in the prediction and reference. "Big" and "large" get high similarity because BERT knows they're synonymous. BERTScore computes precision (each predicted token's best match in the reference), recall (each reference token's best match in the prediction), and F1. It correlates significantly better with human judgment than BLEU or ROUGE, especially for creative and paraphrastic text. Other learned metrics include COMET (trained on human translation quality judgments), BLEURT (fine-tuned BERT for quality estimation), and UniEval (multi-dimensional evaluation). The trend is clear: learned metrics are replacing hand-crafted ones.

Human evaluation remains the gold standard for NLP because automated metrics can't fully capture fluency, coherence, factuality, and usefulness. Common human evaluation protocols include: Likert scale rating (rate quality 1–5), pairwise comparison (which output is better, A or B?), error annotation (mark specific errors in the output), and adequacy/fluency (separate scores for meaning preservation and grammatical quality). The challenges are significant: human evaluation is expensive ($0.10–$1.00 per judgment), slow (days to weeks for large evaluations), and subjective (inter-annotator agreement is typically 60–80%). Pairwise comparison is the most reliable protocol because it's easier for humans to compare two outputs than to assign absolute scores. This is why LLM leaderboards like Chatbot Arena use pairwise human preferences.

NLP benchmarks provide standardized test sets for comparing models. GLUE (General Language Understanding Evaluation) and its harder successor SuperGLUE test models on a suite of understanding tasks: sentiment, textual entailment, paraphrase detection, and question answering. Models are scored on each task and given an aggregate score. BERT achieved 80.5 on GLUE; human performance is 87.1. Modern models have saturated both benchmarks, scoring above human performance. This led to harder benchmarks: MMLU (massive multitask language understanding, 57 subjects), BIG-Bench (204 diverse tasks), and Chatbot Arena (live human pairwise comparisons). The benchmark treadmill — models saturate benchmarks faster than new ones are created — is a persistent challenge. Benchmarks also suffer from data contamination: test data leaking into training corpora.

A robust NLP evaluation strategy combines multiple approaches. During development: use automated metrics (F1, BLEU, ROUGE, BERTScore) for fast iteration. Run evaluations after every experiment. Track metrics over time to catch regressions. Before deployment: conduct human evaluation on a representative sample. Use pairwise comparison against the current system. Test on edge cases, adversarial inputs, and out-of-distribution data. After deployment: monitor real-world performance through user feedback, implicit signals (click-through, task completion), and periodic human evaluation. The most important practice is error analysis: manually examine failures to understand systematic patterns. A confusion matrix shows what the model gets wrong; error analysis shows why. The second most important practice: evaluate on data that looks like production, not on clean academic benchmarks.