LLM-powered agents fail differently than traditional software. A 2026 study found agents achieve only 60% success on single runs, dropping to 25% across eight consecutive runs without resilience engineering. The failure modes are unique: partial/malformed responses (the model returns half an answer), model version drift (behavior changes silently when the provider updates), timeout/latency spikes (P95 latency can be 10x P50), content policy rejections (valid business queries blocked by safety filters), context window overflow (conversation exceeds token limits), and rate limiting (HTTP 429 under load). Top models now achieve hallucination rates below 1%, but agents still face infinite loops, context drift, and cascading tool failures that traditional monitoring won't catch.

Agent observability requires five pillars beyond traditional application monitoring. Traces: capture every step, prompt, tool call, and model invocation as a connected trace. Metrics: monitor latency (P50/P95/P99), token usage, cost per interaction, and throughput tied to SLAs. Logs & payloads: persist raw prompts, completions, and tool responses for debugging and audit. Online evaluations: run real-time automated evaluators for faithfulness, safety, and PII leakage. Human review loops: incorporate subject matter experts for risky outputs. Organizations with mature monitoring report 80% faster incident resolution, 50% reduction in production issues, and 30% cost savings. By 2026, 89% of organizations have implemented observability, with quality issues as the top production barrier at 32%.

Circuit breakers prevent cascading failures by monitoring error rates and temporarily stopping requests to failing services. The pattern uses three states: CLOSED (normal operation, requests pass through), OPEN (too many failures detected, requests are immediately rejected with a fallback response), and HALF-OPEN (after a recovery timeout, a limited number of test requests are allowed through to check if the service has recovered). Configure with a failure threshold (e.g., 5 failures in 60 seconds triggers OPEN), a recovery timeout (e.g., 30 seconds before trying HALF-OPEN), and a success threshold (e.g., 3 consecutive successes in HALF-OPEN returns to CLOSED). Without circuit breakers, a failing LLM API will consume your entire request budget while returning errors.

Not all errors deserve retries. Transient failures (rate limits, timeouts, API connection errors) warrant retries with exponential backoff. Permanent failures (authentication errors, content policy rejections, invalid inputs) require fast failure — retrying will never succeed. The retry formula: delay = base_delay × 2^attempt + jitter, with a maximum of 3–5 retries. Beyond retries, implement fallback chains: multi-tier model degradation from full functionality (primary model) to core functionality (smaller/cheaper model) to basic responses (cached/template responses). This ensures the user always gets something useful, even when the primary model is unavailable.

Production guardrails operate at three layers. Input guardrails: validate and sanitize all inputs before they reach the model — check for prompt injection, PII in queries that shouldn't contain it, and inputs that exceed context windows. Output guardrails: validate all model outputs before they reach the user — check for hallucinated data, PII leakage, off-topic responses, and outputs that violate business rules. Execution guardrails: limit what the agent can do — restrict tool access by role, enforce spending limits per action, set maximum loop iterations, and require approval for irreversible actions. These three layers create defense in depth: if one layer fails, the others catch the problem. No single guardrail is sufficient; the combination is what makes the system safe.

With 46% of AI budgets spent on inference, cost control is a production requirement, not an optimization. Implement controls at four levels. Per-request budgets: set maximum token limits per agent action to prevent runaway costs from verbose prompts or infinite loops. Per-user budgets: cap daily/monthly usage per user to prevent abuse and ensure fair distribution. Per-agent budgets: set monthly spending limits per agent with alerts at 80% and hard stops at 100%. Model routing: use cheaper models for simple tasks and expensive models only when needed — a well-designed router can cut inference costs by 40–60% with minimal quality impact. Provisioned throughput from cloud providers offers 30–50% savings for predictable workloads.

Scaling from pilot to enterprise-wide deployment requires a deliberate strategy, not just "turn it on for everyone." The scaling playbook has four phases. Single workflow: prove the agent works for one team on one process with full monitoring. Team-wide: expand to the full team, adding load testing and performance baselines. Department-wide: expand across the department, adding role-based access, cost allocation, and cross-team monitoring. Enterprise-wide: full rollout with centralized governance, federated management, and executive dashboards. At each phase, validate that latency stays within SLA, costs scale linearly (not exponentially), error rates don't increase, and human oversight capacity matches demand. The most common scaling failure is overwhelming the human review pipeline.

Before any agent goes to production, it must pass a production readiness review covering all the patterns from this chapter. This isn't a formality — it's the gate that separates pilots that impress demos from agents that survive real-world conditions. The checklist covers observability (all five pillars implemented), resilience (circuit breakers, retries, fallbacks tested), guardrails (input, output, and execution guardrails active), cost controls (per-request, per-user, per-agent budgets set), scaling plan (load tested at 3x expected peak), compliance (audit trails, documentation, human oversight), and runbook (incident response procedures documented and rehearsed). An agent that passes all seven categories is production-ready. Anything less is a risk.