An agent’s failure isn’t expensive because it failed—it’s expensive because you found out about it from a customer complaint instead of an alert. Detection latency multiplies the cleanup cost by an order of magnitude. This post explores the arithmetic of failure-finding, why automated detection is worth the infrastructure cost, and how to choose detection strategies.
You can’t tune a retry strategy you can’t see. This is how to instrument retries so you know whether each one succeeds (saving you money) or fails permanently (burning it). Three structured metrics turn a blind spot into actionable signal.
The retry decision (when to give up, how long to wait) only works if you can see inside it. Idempotency keys, retry-attempt correlation, and cost accounting are the foundation. Without them, your retry logic is flying blind — you can’t tell if you’re fixing a transient failure or burning your budget on a permanent one.
The cheap part of a failed agent run is running it again. The expensive part is that you can’t — the failure was non-deterministic, so the run that broke is gone, and you pay to summon it back. A cost model shows why reproduction, not repair, dominates your debugging bill, and why always-on tracing is almost always cheaper than the alternative it replaces.
Most agent failures don’t throw. The run returns a result, exit code zero, and the result is wrong — or it burns an hour and quietly gives up. If your monitoring only counts exceptions, you’re blind to the failures that actually cost you. A taxonomy of agent failure modes and the specific instrumentation that catches each one before your users or your bill do.