The lesson from the $200 postmortem that generalizes past that one incident: a per-step retry cap bounds a step, never a run and never a fleet. Every layer retried politely, within its own limit, and the limits multiplied into a bill because nothing bounded the total. Local caps compose into a global disaster.

That post promised a circuit breaker. This is the full set — the patterns that put a ceiling on what a fleet of agents can collectively spend clawing at a failure that isn’t going away. They come from distributed-systems practice, but agents make them urgent because each retry isn’t a cheap HTTP replay: it’s a full transcript re-read plus a model call, so the unit of waste is dollars, not milliseconds.

Pattern 1: a shared retry budget, not a per-call cap

The failure mode is that N workers each get their own retry allowance, so the fleet’s total retry capacity is N × (per-worker cap) — unbounded in practice. The fix is a shared budget: a token bucket the whole fleet draws from, refilled slowly, that caps retries as a fraction of successful work rather than an absolute count per call.

import time, threading

class RetryBudget:
    """Fleet-wide: allow retries only while they're a small fraction of real traffic."""
    def __init__(self, ratio=0.1, min_per_sec=1.0):
        self.ratio, self.min = ratio, min_per_sec
        self.tokens, self.last = 0.0, time.monotonic()
        self.lock = threading.Lock()

    def on_success(self):
        with self.lock:
            self.tokens += self.ratio      # each success earns partial retry credit

    def allow_retry(self):
        with self.lock:
            now = time.monotonic()
            self.tokens += self.min * (now - self.last)   # slow ambient refill
            self.last = now
            if self.tokens >= 1.0:
                self.tokens -= 1.0
                return True
            return False

The property that matters: when the downstream is healthy, successes keep refilling the bucket and retries flow freely. When it’s broken, successes stop, the bucket drains, and retries choke off automatically — no human, no alert, no config change. The fleet’s retry rate is pinned to its success rate. This is the pattern that would have capped the $200 incident at pennies: once 40% of calls were permanently failing, the budget would have starved within minutes because those failures never produced the successes that refill it.

Pattern 2: a circuit breaker per dependency

The retry budget throttles; the circuit breaker stops. When a specific dependency crosses a failure threshold, open the circuit: fail fast without even attempting the call, for a cool-down window, then let a single probe test whether it’s back.

class CircuitBreaker:
    def __init__(self, threshold=5, cooldown=30):
        self.threshold, self.cooldown = threshold, cooldown
        self.fails, self.opened_at, self.state = 0, None, "closed"

    def call(self, fn):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.cooldown:
                raise CircuitOpen()            # fail fast — no attempt, no spend
            self.state = "half_open"           # let ONE probe through
        try:
            result = fn()
        except Exception:
            self.fails += 1
            if self.fails >= self.threshold or self.state == "half_open":
                self.state, self.opened_at = "open", time.monotonic()
            raise
        self.fails, self.state = 0, "closed"    # recovered
        return result

The key move for agents: the circuit is keyed per dependency (per tool, per API), and it’s shared across the fleet, not per-worker. In the postmortem, twelve workers each independently rediscovered that the enrichment API was down. A shared breaker means the first few failures open it once, and the other eleven workers fail fast for free instead of each paying to relearn the same fact. Fail-fast is a feature: a run that gives up in 50ms costs nothing, and ends in a labeled hard_error outcome you can actually see.

Pattern 3: decorrelated jitter, or the herd stampedes

Plain exponential backoff has a subtle fleet bug: if all N workers fail at the same moment (a dependency blip), they all back off by the same schedule and retry in synchronized waves. The recovering service gets slammed by N simultaneous retries, falls over again, and you get a self-sustaining thundering herd — the retries become the outage.

The fix is jitter, and the good variant is decorrelated jitter:

import random
def next_delay(prev, base=0.5, cap=30):
    return min(cap, random.uniform(base, prev * 3))

Each worker’s next delay is randomized against its own previous delay, which spreads retries across a smooth band instead of stacking them on tick boundaries. This costs nothing and removes an entire class of “our retries caused the second outage” incident. Full backoff with jitter is table stakes; the reason it’s worth naming here is that agent fleets are small enough that people skip it — and twelve workers is plenty to stampede a recovering internal API.

Pattern 4: quarantine the poison, don’t recirculate it

Some failures are permanent — the 400 that will always be a 400, the record that will always be malformed. Retrying those is pure waste no matter how well you throttle it, and if they sit at the head of a queue they block the good work behind them. The pattern is a dead-letter queue: after a bounded number of attempts, move the item out of the live queue into a quarantine for later inspection, and keep processing.

The discipline this enforces is the one the postmortem was really about: classify the error before you retry it. Retryability is a property of the specific error, not a default. A dead-letter queue is where you put the errors you’ve correctly classified as not retryable here — and a growing DLQ is itself a signal, a labeled failure count you can alert on, instead of an invisible retry storm.

How they compose

These aren’t alternatives; they’re layers, and each catches what the one before it misses:

  • Decorrelated backoff makes each individual retry non-synchronized — so a healthy blip recovers instead of stampeding.
  • Circuit breaker stops attempts against a dependency that’s currently down — cheap fail-fast for the whole fleet at once.
  • Retry budget caps total retry volume as a fraction of success — the global ceiling that holds even when many small things fail at once.
  • Dead-letter quarantine removes permanent failures from circulation entirely — so you never pay to retry the unretryable.

The first three bound the cost of transient failure. The fourth bounds the cost of permanent failure. The $200 incident was a permanent failure with only per-step transient-failure controls, which is why the controls that existed did nothing.

The one-line version

Local retry caps compose into global waste: twelve workers each retrying reasonably is how one bad deploy becomes a bill. To bound a fleet, you need controls that live above the individual call — a shared retry budget pinned to success rate, a per-dependency circuit breaker shared across workers, decorrelated jitter so recovery doesn’t stampede, and a dead-letter queue so permanent failures leave the loop. Ask of your system: what is the maximum a full fleet of my agents can spend retrying a dependency that will never recover? If the answer isn’t a number you can state, it’s whatever your provider will bill you before someone wakes up.