Field notes on building LLM agents that survive production. Concrete code, real failure modes, numbers over adjectives — context, tools, evals, cost, and the ways agents break.
What an MCP server actually is, and the tool-design mistakes that break it
MCP gives you a standard wire format for connecting an agent to tools — it doesn’t make the tools on the other end safe to call. Most MCP servers are thin wrappers around an existing REST API, which quietly imports every assumption that API made about having a human developer as its client. Here’s what MCP actually is, how it differs from the API you already know, and the specific tool-design mistakes that turn a working MCP server into a flaky agent.
What Idempotent Actually Means: Why Retries Are Safe (and When They Aren't)
Idempotent means doing something twice has the same effect as doing it once. This is essential for APIs, agents, and any system where retries can happen. Here’s why, and how to get it right.
The cost of undoing: partial failures and the cleanup bill
An agent writes to three systems, then fails on the fourth. The first three writes are now orphaned in a partially-succeeded state. Rolling them back costs more than the original operation — not in tokens, but in human time and coordination overhead. This is the cleanup bill: the hidden cost of partial failures that retry logic doesn’t touch.
What actually drives your Claude bill: cache misses, quadratic context, and prepaid retries
The per-token rate on Anthropic’s pricing page is real, but it’s the least useful number for predicting your actual invoice. Three multipliers move the bill far more than model choice does — a cache that silently misses, a transcript that grows quadratically with run length, and a best-of-N pattern that prepays for latency you may not need. Here’s the arithmetic on all three, combined, with real numbers.
The cost of finding a failure after the customer finds it
An agent’s failure isn’t expensive because it failed—it’s expensive because you found out about it from a customer complaint instead of an alert. Detection latency multiplies the cleanup cost by an order of magnitude. This post explores the arithmetic of failure-finding, why automated detection is worth the infrastructure cost, and how to choose detection strategies.
The agent that trusted a bad API: silent failures and validation debt
An agent called an API, got a syntactically valid response, trusted it, and built three wrong decisions on top of it. No error was thrown. Every step succeeded locally. The cascade cost was 15× the original bad call — and all of it was preventable by a two-line validation check. This is validation debt: paying the cost of skipped checks in compounding failure downstream.
Best-of-N is prepaid retries: the cost math of racing parallel attempts
Launching N attempts at once and keeping the first success feels like a free win over sequential retries — you trade money for tail latency, and money is supposedly the thing you have more of. It isn’t free, and it isn’t even always a trade: for the correlated failures that dominate real production incidents, best-of-N pays for N guaranteed-identical failures up front instead of stopping at one.
The transcript is a log, not an index: retrieval for long-running agents
Eviction and summarization keep a long-running agent’s window small, but the summaries themselves still accumulate in the linear transcript — and on a run long enough, that accumulation becomes the new quadratic. Past a few hundred steps, the fix isn’t a better compaction policy inside the window. It’s moving history out of the window entirely and querying it like a database instead of replaying it like a log.
The caller gave up ten minutes ago: orphaned retries in agent fleets
A user closes the tab. An upstream request times out. A parent agent gets cancelled by its own budget. None of that reliably reaches the retry loop three calls deep, so the retry keeps going — burning tokens and rate-limit headroom for a result nobody will ever read. Cancellation is the one signal every fleet retry pattern assumes exists and almost none actually propagate.
429 is not a timeout: why rate limits need their own retry budget
A 429 and a 500 both land in the same except block, so most retry budgets treat them the same: one bucket, one backoff curve, one circuit breaker. That conflation is wrong in both directions — it makes you wait too little for the failure that isn’t yours, and panic too much over the one that is. Two error classes, two buckets, and the Retry-After header everyone reads and no one obeys.