Failure-Modes

A cache that’s never invalidated doesn’t know it’s stopped being correct — it just keeps serving the value it read at write time. Give a multi-agent crew a shared objective instead of a shared cache and you get the identical failure with none of the tooling that usually catches it: every agent read the goal once, at spawn, and nothing tells it when that copy goes stale. This is multi-agent context drift — distinct from goal drift (the target itself moving) and split-brain (the crew’s facts about state disagreeing) — a concrete policy-escalation failure, the mechanism, and what to instrument before a customer hears a promise nobody’s operating on anymore.

Loop drift is one agent staying busy without getting closer to done. Hand the same failure to a crew and every agent stays busy, produces defensible work, and the team’s collective objective still ends up somewhere your original request never sent it — a genuine multi-agent goal drift failure mode, not a mistake in any single agent’s reasoning. A concrete feature-build failure, the four things that make a goal decay one handoff at a time, and what to measure so a ‘done’ check compares against the original ask instead of the nearest upstream spec.

In distributed systems, split-brain is what happens when a partition convinces two halves of a cluster that each is the sole leader, and both start accepting writes. Multi-agent systems build the identical topology every time more than one agent reads and writes a ‘shared’ memory store with no lock, no leader, and no version check — and the failure that follows is a genuine multi-agent memory consensus problem, not a bug in any one agent. The mechanism, a concrete incident-response failure, and what to instrument before it costs you a duplicate action.

An agent writes to three systems, then fails on the fourth. The first three writes are now orphaned in a partially-succeeded state. Rolling them back costs more than the original operation — not in tokens, but in human time and coordination overhead. This is the cleanup bill: the hidden cost of partial failures that retry logic doesn’t touch.

An agent’s failure isn’t expensive because it failed—it’s expensive because you found out about it from a customer complaint instead of an alert. Detection latency multiplies the cleanup cost by an order of magnitude. This post explores the arithmetic of failure-finding, why automated detection is worth the infrastructure cost, and how to choose detection strategies.

An agent called an API, got a syntactically valid response, trusted it, and built three wrong decisions on top of it. No error was thrown. Every step succeeded locally. The cascade cost was 15× the original bad call — and all of it was preventable by a two-line validation check. This is validation debt: paying the cost of skipped checks in compounding failure downstream.

Launching N attempts at once and keeping the first success feels like a free win over sequential retries — you trade money for tail latency, and money is supposedly the thing you have more of. It isn’t free, and it isn’t even always a trade: for the correlated failures that dominate real production incidents, best-of-N pays for N guaranteed-identical failures up front instead of stopping at one.

A user closes the tab. An upstream request times out. A parent agent gets cancelled by its own budget. None of that reliably reaches the retry loop three calls deep, so the retry keeps going — burning tokens and rate-limit headroom for a result nobody will ever read. Cancellation is the one signal every fleet retry pattern assumes exists and almost none actually propagate.

A 429 and a 500 both land in the same except block, so most retry budgets treat them the same: one bucket, one backoff curve, one circuit breaker. That conflation is wrong in both directions — it makes you wait too little for the failure that isn’t yours, and panic too much over the one that is. Two error classes, two buckets, and the Retry-After header everyone reads and no one obeys.

Retry budgets cap HOW MUCH you retry; this is about WHEN to retry at all. The decision isn’t uniform: user-facing operations, background jobs, and fleet-wide cascades each have different failure costs, different retry ceilings, and different layers where the decision lives. A cost model for when to fail fast instead.