https://loopandretry.github.io/tags/failure-modes/ Recent content in Failure-Modes on Loop & Retry https://loopandretry.github.io/images/og-default.png https://loopandretry.github.io/images/og-default.png Hugo en-us Tue, 14 Jul 2026 13:00:00 -0400 https://loopandretry.github.io/posts/debugging-a-failed-run-costs-more/ Tue, 14 Jul 2026 13:00:00 -0400 https://loopandretry.github.io/posts/debugging-a-failed-run-costs-more/ The cheap part of a failed agent run is running it again. The expensive part is that you can’t — the failure was non-deterministic, so the run that broke is gone, and you pay to summon it back. A cost model shows why reproduction, not repair, dominates your debugging bill, and why always-on tracing is almost always cheaper than the alternative it replaces. https://loopandretry.github.io/posts/predicting-agent-failure-before-release/ Tue, 14 Jul 2026 05:02:00 -0400 https://loopandretry.github.io/posts/predicting-agent-failure-before-release/ A demo proves an agent can succeed once. It says almost nothing about how often it will fail under real load, real input distributions, and real adversarial garbage. The failures that cost you in production are predictable before release — but only if you test the things that actually shift between the demo and the deployment. Four pre-release signals that forecast production failure, and the ones that don’t. https://loopandretry.github.io/posts/how-agent-failures-cascade/ Tue, 14 Jul 2026 05:01:00 -0400 https://loopandretry.github.io/posts/how-agent-failures-cascade/ A single agent error rarely stays a single error. The bad output goes into the context, the next step reasons on top of it, and the mistake compounds down the trajectory — one wrong step becoming N wrong steps. This is the cascade, why it’s structurally different from a fleet-wide blast radius, and the three interruption points that stop a local mistake from eating the whole run. https://loopandretry.github.io/posts/fleet-retry-patterns/ Mon, 13 Jul 2026 18:15:00 -0400 https://loopandretry.github.io/posts/fleet-retry-patterns/ A per-step retry cap bounds a step. It never bounds a run, and it never bounds a fleet — twelve workers each retrying ‘reasonably’ is how you turn one bad deploy into a bill. The four patterns that actually put a ceiling on what a fleet of agents can spend recovering from a failure: shared retry budgets, circuit breakers, decorrelated backoff, and poison quarantine. https://loopandretry.github.io/posts/measuring-agent-failure-in-production/ Mon, 13 Jul 2026 18:10:00 -0400 https://loopandretry.github.io/posts/measuring-agent-failure-in-production/ Most agent failures don’t throw. The run returns a result, exit code zero, and the result is wrong — or it burns an hour and quietly gives up. If your monitoring only counts exceptions, you’re blind to the failures that actually cost you. A taxonomy of agent failure modes and the specific instrumentation that catches each one before your users or your bill do. https://loopandretry.github.io/posts/idempotency-keys-for-agents/ Mon, 13 Jul 2026 09:00:00 -0400 https://loopandretry.github.io/posts/idempotency-keys-for-agents/ Retrying a read is free. Retrying a write can charge a card twice, send two emails, or book two rooms — and the model has no idea it happened. Retry safety is a property you build into the tool, not a flag you set on the loop. Here’s why at-least-once delivery is the default you’re actually running, how to derive a stable idempotency key from an agent’s intent, and a dedup wrapper that makes any write safe to retry. https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/ Sun, 12 Jul 2026 09:00:00 -0400 https://loopandretry.github.io/posts/postmortem-200-dollars-retrying-a-400/ An agent burned ~$200 overnight retrying an HTTP 400 — a request that was defined to fail. No component was buggy; each layer retried “reasonably.” The teardown: why retryability is a property of the error and not a default, how three nested retry caps multiply into 75 doomed attempts per item, and why per-step caps never bound a bill. With the two-line fix and a circuit breaker. https://loopandretry.github.io/posts/loop-drift/ Tue, 07 Jul 2026 10:00:00 -0400 https://loopandretry.github.io/posts/loop-drift/ The worst agent failures don’t crash — they keep working. A postmortem on loop drift: an agent that stayed busy for 40 steps without getting closer to done, why the model’s own progress reports can’t catch it, and the external signals and evals that can.