Failure-Modes

Every coordination failure in a multi-agent system is a diff between two agents — and per-agent monitoring cannot express a diff. You can have complete, exception-free telemetry on every agent in a crew and still be structurally blind to split-brain, goal drift, and context drift, because each one lives in the comparison your telemetry has no join key for. What to actually instrument for each of the three, which probes are free and which cost a model call, and how to sample a crew for belief reconciliation without paying O(n) extra calls at every checkpoint.

A cache that’s never invalidated doesn’t know it’s stopped being correct — it just keeps serving the value it read at write time. Give a multi-agent crew a shared objective instead of a shared cache and you get the identical failure with none of the tooling that usually catches it: every agent read the goal once, at spawn, and nothing tells it when that copy goes stale. This is multi-agent context drift — distinct from goal drift (the target itself moving) and split-brain (the crew’s facts about state disagreeing) — a concrete policy-escalation failure, the mechanism, and what to instrument before a customer hears a promise nobody’s operating on anymore.

Loop drift is one agent staying busy without getting closer to done. Hand the same failure to a crew and every agent stays busy, produces defensible work, and the team’s collective objective still ends up somewhere your original request never sent it — a genuine multi-agent goal drift failure mode, not a mistake in any single agent’s reasoning. A concrete feature-build failure, the four things that make a goal decay one handoff at a time, and what to measure so a ‘done’ check compares against the original ask instead of the nearest upstream spec.

In distributed systems, split-brain is what happens when a partition convinces two halves of a cluster that each is the sole leader, and both start accepting writes. Multi-agent systems build the identical topology every time more than one agent reads and writes a ‘shared’ memory store with no lock, no leader, and no version check — and the failure that follows is a genuine multi-agent memory consensus problem, not a bug in any one agent. The mechanism, a concrete incident-response failure, and what to instrument before it costs you a duplicate action.

An agent writes to three systems, then fails on the fourth. The first three writes are now orphaned in a partially-succeeded state. Rolling them back costs more than the original operation — not in tokens, but in human time and coordination overhead. This is the cleanup bill: the hidden cost of partial failures that retry logic doesn’t touch.

An agent’s failure isn’t expensive because it failed—it’s expensive because you found out about it from a customer complaint instead of an alert. Detection latency multiplies the cleanup cost by an order of magnitude. This post explores the arithmetic of failure-finding, why automated detection is worth the infrastructure cost, and how to choose detection strategies.

An agent called an API, got a syntactically valid response, trusted it, and built three wrong decisions on top of it. No error was thrown. Every step succeeded locally. The cascade cost was 15× the original bad call — and all of it was preventable by a two-line validation check. This is validation debt: paying the cost of skipped checks in compounding failure downstream.

A user closes the tab. An upstream request times out. A parent agent gets cancelled by its own budget. None of that reliably reaches the retry loop three calls deep, so the retry keeps going — burning tokens and rate-limit headroom for a result nobody will ever read. Cancellation is the one signal every fleet retry pattern assumes exists and almost none actually propagate.

A 429 and a 500 both land in the same except block, so most retry budgets treat them the same: one bucket, one backoff curve, one circuit breaker. That conflation is wrong in both directions — it makes you wait too little for the failure that isn’t yours, and panic too much over the one that is. Two error classes, two buckets, and the Retry-After header everyone reads and no one obeys.

You can’t tune a retry strategy you can’t see. This is how to instrument retries so you know whether each one succeeds (saving you money) or fails permanently (burning it). Three structured metrics turn a blind spot into actionable signal.