Here is the intuition to kill: a failed run is cheap because you can just run it again. A deterministic system earns you that assumption — the bug is sitting there, reproducible on demand, and debugging is a bounded search through code you control. An agent is not that system. When a run fails, the run that failed is gone. What you re-run is a different run that happens to share a prompt, and it may well succeed. The token cost of the failure was never the expensive part. The expensive part is that you have to pay, over and over, to make the failure happen again in front of you.
This post is about that second bill — the cost of reproduction — and why it dominates the cost of the actual fix. It’s the cost twin of the measuring-failure-in-production post: that one was about noticing silent failures; this one is about what it costs to understand one once you’ve noticed it.
The failed run is a crime scene with no recording
A traditional bug report comes with a stack trace, an input, and a promise: feed the input back in and you’ll see the trace again. That promise is what makes debugging tractable. You bisect, you add a log line, you re-run, you narrow. Every re-run is free information because every re-run is the same run.
An agent breaks that chain in two places. First, the model call is stochastic — same prompt, different sampled tokens, different tool calls, different path. Second, the world moved: the API the agent called returns different data now, the row it read got updated, the rate limiter is in a different state. So the failed trajectory isn’t a function of inputs you still have. It was a function of inputs plus two sources of entropy you didn’t record. Re-running is not re-observing. It’s rolling the dice again and hoping for the same bad number.
That changes the unit economics of debugging completely. In a deterministic system, reproduction cost is ~zero and all your money goes to the fix. In an agentic system, you pay a reproduction tax before you can even begin the fix — and if the failure is rare, that tax is enormous.
A cost model for reproducing a failure
Say a failure mode shows up with probability p per run. To debug it the classic way — re-run until it happens, then inspect — you need, in expectation, 1/p runs to see it once. Each run costs tokens and wall-clock. And an engineer is sitting there through all of it, which is the most expensive axis in the whole system.
# What it costs to reproduce one instance of a p-probability failure
# by re-running until it recurs. All rates illustrative — swap in yours.
p = 0.02 # failure probability per run (a 2%-of-runs bug)
RUN_COST = 0.05 # $ in tokens + tool calls per full run
RUN_SECONDS = 40 # wall-clock per run
ENG_RATE = 120 / 3600 # $/second for the engineer waiting on it
expected_runs = 1 / p # ~50 runs to see it once
token_cost = expected_runs * RUN_COST # $2.50 in runs
wall_seconds = expected_runs * RUN_SECONDS # ~33 minutes of re-running
eng_cost = wall_seconds * ENG_RATE # ~$67 of engineer time waiting
print(f"runs to reproduce : {expected_runs:.0f}")
print(f"token cost : ${token_cost:.2f}")
print(f"engineer time : ${eng_cost:.2f}")
print(f"total to reproduce: ${token_cost + eng_cost:.2f} (before any fix)")
runs to reproduce : 50
token cost : $2.50
engineer time : $66.67
total to reproduce: $69.17 (before any fix)
Look at the ratio. The tokens burned re-running — the line that shows up on an invoice — are $2.50. The reproduction tax, mostly a human waiting for a dice roll to come up bad again, is nearly $70. And every bit of that is spent before anyone has written a single character of the fix. This is the same shape as the $200 postmortem, except the money isn’t being burned by the agent — it’s being burned by you, trying to make the agent misbehave on command.
And 1/p is the optimistic case, because it assumes each re-run is an independent draw from the same distribution. It isn’t. The world moved, so some of your re-runs can’t reproduce the bug at any p — the state that triggered it is gone. For those, expected reproduction cost isn’t high, it’s infinite. You will never see it again by re-running, and you’ll spend the afternoon proving that.
The fix is to make reproduction free
Notice what the model is actually charging you for: the cost per reproduction times the number of reproductions. Repair cost — the engineer’s time once they can see the failed trajectory — barely moved between the deterministic and agentic worlds. The entire blowup is in the reproduction term. So that’s the term to attack.
You attack it by recording the run instead of re-rolling it. If you capture, for every step, the exact prompt sent, the exact sampled response, and the exact tool inputs and outputs, then a failed trajectory becomes a replay rather than a re-run. Reproduction cost drops from 1/p × RUN_COST to approximately zero — you open the trace of the run that already failed. The two sources of entropy that made re-running useless (model sampling, world state) are now bytes on disk.
# The always-on trace turns reproduction into a file read.
def step(agent, state, trace):
prompt = render_prompt(state)
resp = model.call(prompt) # the stochastic part...
tool_out = run_tool(resp.tool, resp.args) # ...and the moved-world part
trace.append({ # ...both pinned to disk, per step
"prompt": prompt,
"response": resp.raw,
"tool": resp.tool, "args": resp.args, "tool_out": tool_out,
})
return apply(resp, state)
# Debugging a failure is now: load the trace, look. No re-running, no dice.
def reproduce(run_id):
return load_trace(run_id) # cost: one file read, p irrelevant
The economic decision is now a one-liner. Tracing has a standing cost — storage, a little latency, some plumbing — that you pay on every run, the vast majority of which succeed. Call it TRACE_COST per run. It’s worth it exactly when
TRACE_COST < p × (reproduction_tax_you_avoid)
Plug the numbers in: even if tracing costs a full cent per run, and the failure is rare at 2%, the avoided reproduction tax is ~$70. The break-even failure rate is absurdly low — you’d keep tracing on for a bug that shows up in one run in seven thousand. This is why “just turn on tracing” is nearly always right for agents and merely nice-to-have for deterministic services: the value of a trace scales with how expensive reproduction is without one, and for a non-deterministic system that cost is unbounded.
The one-line version
For a deterministic system, reproduction is free and debugging cost is repair cost. For an agent, the run that failed is a non-deterministic event you did not record, so reproduction costs 1/p re-runs of tokens and — mostly — engineer time waiting, and for world-dependent failures it’s not reproducible at all. That reproduction tax, not the tokens and not the fix, is where your debugging bill goes. Record every run’s prompts, samples, and tool I/O so a failure becomes a replay instead of a re-roll, and the whole tax collapses to a file read. Tracing looks like overhead until you price the afternoon you’ll otherwise spend trying to make a 2% bug happen on demand.