The Amplification Problem
A chatbot that hallucinates gives you a wrong answer. An agent that hallucinates takes a wrong action. It might send an incorrect email to a client, execute a flawed database query that corrupts data, approve a transaction that violates policy, or book a flight to the wrong city. The stakes are fundamentally higher because agents operate in the real world, not just in conversation.
The Klarna Lesson
Klarna’s aggressive automation delivered impressive cost savings but compromised service quality in complex interactions. Customers experienced lack of empathy and contextual understanding when situations deviated from standard patterns. Klarna pivoted to a hybrid model, reintroducing human agents for nuanced support. The lesson: not every interaction should be automated, even if it technically can be.
Specific Risk Categories
Runaway actions — An agent in a loop that keeps executing actions without stopping. Requires hard limits on iterations, cost, and scope.
Permission escalation — An agent that accesses data or systems beyond its intended scope. Requires strict tool-level access controls.
Prompt injection — Malicious inputs that hijack the agent’s behavior (Chapter 26). Agents are more vulnerable because they take actions, not just generate text.
Cascading errors — One wrong step early in a workflow corrupts every subsequent step.
Critical for leaders: Agents require a fundamentally different risk framework than traditional AI. You need action-level permissions (what can the agent do?), scope boundaries (what data can it access?), cost limits (how much can it spend per task?), human checkpoints (which actions require approval?), and kill switches (how do you stop a runaway agent?). Design these controls before deployment, not after an incident.