A user closes the tab. An upstream request times out. A parent agent gets cancelled by its own budget. None of that reliably reaches the retry loop three calls deep, so the retry keeps going — burning tokens and rate-limit headroom for a result nobody will ever read. Cancellation is the one signal every fleet retry pattern assumes exists and almost none actually propagate.
A 429 and a 500 both land in the same except block, so most retry budgets treat them the same: one bucket, one backoff curve, one circuit breaker. That conflation is wrong in both directions — it makes you wait too little for the failure that isn’t yours, and panic too much over the one that is. Two error classes, two buckets, and the Retry-After header everyone reads and no one obeys.
You built an LLM judge to grade your agent. What grades the judge? An eval you never validated is a ruler you never checked against a meter stick — and a biased judge doesn’t just add noise, it moves your headline number in a consistent direction. How to meta-evaluate a judge: the labeled set, the agreement metric that isn’t accuracy, and the drift check.
‘When not to build an agent’ made the case against the loop in the abstract — quadratic cost, serial latency, an untestable failure surface. This is the concrete follow-on: three fixed pipeline shapes (linear chain, router-plus-handlers, fan-out/fan-in) that cover most of what people default to a loop for, why each one is cheaper and more testable, and the one test for when a real loop actually earns its cost.
Retry budgets cap HOW MUCH you retry; this is about WHEN to retry at all. The decision isn’t uniform: user-facing operations, background jobs, and fleet-wide cascades each have different failure costs, different retry ceilings, and different layers where the decision lives. A cost model for when to fail fast instead.
A single agent fails by getting the task wrong. A team of agents fails in ways no single agent can: correlated collapse, diffused responsibility, context fragmentation, and consensus that converges on nothing. The four failure modes that only exist once you have more than one agent — and why adding agents can lower reliability.
Prompt caching can cut your input-token bill by 90% on the reused part of a request — or do nothing at all, with no error to tell you which. It’s an exact-prefix match with a short TTL, and small, common mistakes in how agents build requests break it silently. Here’s what actually gets cached, the five ways real agents lose the discount without noticing, and how to check whether yours is.
The default retry pattern — catch the failure, append ’that didn’t work, try again,’ resend the same messages — doesn’t give the model a clean second attempt. It gives the model a context window containing its own wrong answer, which is exactly the thing most likely to make the second attempt rhyme with the first. Why retries poison the window, how to tell a poisoning retry from a safe one, and a scrub step that keeps the constraints without keeping the wrong path.
Every per-step timeout is a number someone typed in without a model behind it — too short and you kill real work in flight, too long and you pay to sit idle waiting on a hang. Both mistakes are failure modes with a price tag. A small cost model finds the number that actually minimizes total cost instead of the one that felt safe.
Streaming a text response is easy: print tokens as they arrive, order doesn’t matter to the reader. Streaming a tool call is not, because the payload is JSON, and partial JSON is not valid JSON. The three ways people handle that mismatch, why two of them break in production, and the pattern that lets you show progress without ever executing on a half-formed argument.