Observability

Every coordination failure in a multi-agent system is a diff between two agents — and per-agent monitoring cannot express a diff. You can have complete, exception-free telemetry on every agent in a crew and still be structurally blind to split-brain, goal drift, and context drift, because each one lives in the comparison your telemetry has no join key for. What to actually instrument for each of the three, which probes are free and which cost a model call, and how to sample a crew for belief reconciliation without paying O(n) extra calls at every checkpoint.

An agent’s failure isn’t expensive because it failed—it’s expensive because you found out about it from a customer complaint instead of an alert. Detection latency multiplies the cleanup cost by an order of magnitude. This post explores the arithmetic of failure-finding, why automated detection is worth the infrastructure cost, and how to choose detection strategies.

You can’t tune a retry strategy you can’t see. This is how to instrument retries so you know whether each one succeeds (saving you money) or fails permanently (burning it). Three structured metrics turn a blind spot into actionable signal.

The retry decision (when to give up, how long to wait) only works if you can see inside it. Idempotency keys, retry-attempt correlation, and cost accounting are the foundation. Without them, your retry logic is flying blind — you can’t tell if you’re fixing a transient failure or burning your budget on a permanent one.

The cheap part of a failed agent run is running it again. The expensive part is that you can’t — the failure was non-deterministic, so the run that broke is gone, and you pay to summon it back. A cost model shows why reproduction, not repair, dominates your debugging bill, and why always-on tracing is almost always cheaper than the alternative it replaces.

Most agent failures don’t throw. The run returns a result, exit code zero, and the result is wrong — or it burns an hour and quietly gives up. If your monitoring only counts exceptions, you’re blind to the failures that actually cost you. A taxonomy of agent failure modes and the specific instrumentation that catches each one before your users or your bill do.