Single-model evaluation is already challenging; multi-agent systems add interaction effects, non-determinism from multiple LLM calls, and emergent behaviors that don’t appear in unit tests. A system can pass every agent-level test and still fail at the system level (deadlocks, redundant work, cost explosions). You need evaluation at three layers: individual agent capability, pairwise interaction quality, and end-to-end task success.

Research benchmarks include collaborative coding tasks (SWE-bench with agent teams), debate and persuasion arenas, negotiation games, and multi-agent reasoning challenges. For production, build internal benchmarks from real task logs: sample completed tasks, replay them with the new system, and compare outcomes. Benchmarks should cover happy paths, edge cases (tool failures, ambiguous instructions), and adversarial inputs (prompt injection across agents).

Task success rate alone is insufficient. A system that solves 95% of tasks but costs 10× more tokens, takes 5× longer, or requires 3× more human interventions may be worse than a simpler baseline. Track: success rate, total tokens (input + output across all agents), wall-clock latency, tool call count, human escalation rate, and dollar cost. Report these as a Pareto frontier: which configurations dominate on both quality and cost?

Multi-agent conversations span many turns. Evaluate: conversation coherence (do agents stay on topic?), information flow (does relevant info reach the right agent?), turn efficiency (how many turns to reach a decision?), and termination quality (did the conversation end at the right time?). Use LLM-as-judge on sampled transcripts with rubrics for each dimension. Compare against golden transcripts from expert-annotated examples.

Every prompt change, model upgrade, or tool update can break multi-agent behavior. Build a regression suite: a set of deterministic-seed replays (where possible) or statistical tests (run N trials, compare distributions). Track behavioral fingerprints: tool call sequences, message counts, and decision patterns. Alert when fingerprints shift beyond a threshold. Store golden outputs and diff against them in CI.

Ablation studies: remove one agent at a time and measure impact on task success and cost. If removing an agent has no effect, it’s dead weight. If removing it crashes the system, it’s a single point of failure that needs redundancy. Also measure marginal contribution: what does adding a third reviewer agent improve over two? Diminishing returns are common — more agents often means more tokens for marginal gains.

Production monitoring extends evaluation into real time. Key signals: success rate (rolling window), p95 latency, token burn rate, error rate by agent, human escalation trend, and conversation length distribution. Set alerts for: success rate drop > 5%, cost spike > 2× baseline, any agent error rate > 10%, and conversation length > 2× median. Build runbooks for each alert with diagnosis steps and rollback procedures.

Start with end-to-end task tests from real logs. Add agent-level unit tests for each role. Build regression suites in CI. Run ablation studies before adding agents. Deploy with production monitoring and runbooks. Review metrics weekly. Next chapter: safety, control, and failure modes — what happens when evaluation misses something and the system goes wrong.