https://loopandretry.github.io/pillars/failure-modes/ Recent content in Failure-Modes on Loop & Retry https://loopandretry.github.io/images/og-default.png https://loopandretry.github.io/images/og-default.png Hugo en-us Wed, 15 Jul 2026 09:00:00 -0400 https://loopandretry.github.io/posts/retry-budgets-by-language/ Wed, 15 Jul 2026 09:00:00 -0400 https://loopandretry.github.io/posts/retry-budgets-by-language/ A retry budget is a language-agnostic idea, but the place you enforce it is not. Python’s tenacity decorators, Go’s context-plus-backoff, and JavaScript’s promise chains each make a different mistake easy and a different guarantee hard. Where the shared budget lives, and the per-language trap that leaks it. https://loopandretry.github.io/posts/predicting-agent-failure-before-release/ Tue, 14 Jul 2026 05:02:00 -0400 https://loopandretry.github.io/posts/predicting-agent-failure-before-release/ A demo proves an agent can succeed once. It says almost nothing about how often it will fail under real load, real input distributions, and real adversarial garbage. The failures that cost you in production are predictable before release — but only if you test the things that actually shift between the demo and the deployment. Four pre-release signals that forecast production failure, and the ones that don’t. https://loopandretry.github.io/posts/how-agent-failures-cascade/ Tue, 14 Jul 2026 05:01:00 -0400 https://loopandretry.github.io/posts/how-agent-failures-cascade/ A single agent error rarely stays a single error. The bad output goes into the context, the next step reasons on top of it, and the mistake compounds down the trajectory — one wrong step becoming N wrong steps. This is the cascade, why it’s structurally different from a fleet-wide blast radius, and the three interruption points that stop a local mistake from eating the whole run. https://loopandretry.github.io/posts/fleet-retry-patterns/ Mon, 13 Jul 2026 18:15:00 -0400 https://loopandretry.github.io/posts/fleet-retry-patterns/ A per-step retry cap bounds a step. It never bounds a run, and it never bounds a fleet — twelve workers each retrying ‘reasonably’ is how you turn one bad deploy into a bill. The four patterns that actually put a ceiling on what a fleet of agents can spend recovering from a failure: shared retry budgets, circuit breakers, decorrelated backoff, and poison quarantine. https://loopandretry.github.io/posts/measuring-agent-failure-in-production/ Mon, 13 Jul 2026 18:10:00 -0400 https://loopandretry.github.io/posts/measuring-agent-failure-in-production/ Most agent failures don’t throw. The run returns a result, exit code zero, and the result is wrong — or it burns an hour and quietly gives up. If your monitoring only counts exceptions, you’re blind to the failures that actually cost you. A taxonomy of agent failure modes and the specific instrumentation that catches each one before your users or your bill do. https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/ Sun, 12 Jul 2026 09:00:00 -0400 https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/ An agent burned ~$200 overnight retrying an HTTP 400 — a request that was defined to fail. No component was buggy; each layer retried “reasonably.” The teardown: why retryability is a property of the error and not a default, how three nested retry caps multiply into 75 doomed attempts per item, and why per-step caps never bound a bill. With the two-line fix and a circuit breaker. https://loopandretry.github.io/posts/loop-drift/ Tue, 07 Jul 2026 10:00:00 -0400 https://loopandretry.github.io/posts/loop-drift/ The worst agent failures don’t crash — they keep working. A postmortem on loop drift: an agent that stayed busy for 40 steps without getting closer to done, why the model’s own progress reports can’t catch it, and the external signals and evals that can.