Distributed-Systems

In distributed systems, split-brain is what happens when a partition convinces two halves of a cluster that each is the sole leader, and both start accepting writes. Multi-agent systems build the identical topology every time more than one agent reads and writes a ‘shared’ memory store with no lock, no leader, and no version check — and the failure that follows is a genuine multi-agent memory consensus problem, not a bug in any one agent. The mechanism, a concrete incident-response failure, and what to instrument before it costs you a duplicate action.

Idempotent means doing something twice has the same effect as doing it once. This is essential for APIs, agents, and any system where retries can happen. Here’s why, and how to get it right.

An agent writes to three systems, then fails on the fourth. The first three writes are now orphaned in a partially-succeeded state. Rolling them back costs more than the original operation — not in tokens, but in human time and coordination overhead. This is the cleanup bill: the hidden cost of partial failures that retry logic doesn’t touch.

A user closes the tab. An upstream request times out. A parent agent gets cancelled by its own budget. None of that reliably reaches the retry loop three calls deep, so the retry keeps going — burning tokens and rate-limit headroom for a result nobody will ever read. Cancellation is the one signal every fleet retry pattern assumes exists and almost none actually propagate.

A per-step retry cap bounds a step. It never bounds a run, and it never bounds a fleet — twelve workers each retrying ‘reasonably’ is how you turn one bad deploy into a bill. The four patterns that actually put a ceiling on what a fleet of agents can spend recovering from a failure: shared retry budgets, circuit breakers, decorrelated backoff, and poison quarantine.