Ask an engineer why their agent’s per-step timeout is set to 30 seconds and the honest answer is usually “it felt long enough.” That number is a bet, placed without odds, against a distribution nobody looked at. Set it too low and you cut off calls that were about to succeed — a real result, discarded, paid for in tokens already spent and now retried from scratch. Set it too high and every genuine hang sits there burning wall-clock and holding a worker while nothing happens. Both directions cost money. The number that “feels long enough” is very rarely the number that minimizes either.

This is a failure mode wearing a config value’s clothes. A timeout firing early looks identical to a real failure downstream — it’s counted the same way in your logs, it triggers the same retry, and it can trip the same circuit breaker as an actual outage. Silent failures already hide inside your outcome taxonomy; a false timeout is one you manufactured with a bad guess at a number.

Two ways to be wrong, priced differently

A call’s true completion time is a distribution, not a constant — and it has a long right tail. Pick a timeout T and you split that distribution into two buckets, each with its own cost:

  • False timeout (call would have finished, just after T): you paid for the work done so far, threw it away, and now pay again for a retry — plus whatever the retry’s own chance of also timing out costs, compounding.
  • True timeout (call was actually hung): you paid to sit idle for the full T before finding out, holding a worker the whole time.

Push T up and false timeouts get rarer but true ones get more expensive to detect. Push it down and detection gets cheap but you manufacture false timeouts on calls that were simply a bit slow. There’s a number in between that minimizes the sum — and it’s a calculation, not a feeling.

# Cost of a chosen timeout T, given the true call-duration distribution.
# Illustrative rates — swap in your own percentile curve and prices.
import random

IDLE_RATE   = 0.002        # $/second the worker burns just waiting
CALL_COST   = 0.10         # $ in tokens/compute already spent when it's cut off
RETRY_MULT  = 1.2          # a retry isn't a clean redo — some work must repeat

def sample_duration(rng):
    # long-tailed: most calls finish fast, a minority run long, a few truly hang.
    r = rng.random()
    if r < 0.85:  return rng.uniform(1, 6)     # normal
    if r < 0.95:  return rng.uniform(6, 30)    # slow but real
    return rng.uniform(120, 300)                # actually hung — never finishes

def cost_for_timeout(T, trials=20_000, seed=7):
    rng = random.Random(seed)
    total = 0.0
    for _ in range(trials):
        d = sample_duration(rng)
        if d <= T:
            total += IDLE_RATE * d                  # succeeded, just paid to wait
        else:
            total += IDLE_RATE * T + CALL_COST * RETRY_MULT  # cut off + retried
    return total / trials

for T in (5, 10, 15, 20, 30, 45, 60):
    print(f"T={T:3d}s  avg cost/call=${cost_for_timeout(T):.4f}")

T=  5s  avg cost/call=$0.0457
T= 10s  avg cost/call=$0.0247
T= 15s  avg cost/call=$0.0235
T= 20s  avg cost/call=$0.0222
T= 30s  avg cost/call=$0.0184
T= 45s  avg cost/call=$0.0199
T= 60s  avg cost/call=$0.0214

The minimum sits at 30 seconds here — not at either end, and not where intuition points either. T=5 is nearly 2.5× the minimum: it fires constantly on the “slow but real” bucket, paying the retry tax on calls that would have finished on their own in ten more seconds. T=60 avoids almost all of those false timeouts, but now it’s paying full idle price on every call in the 30–60s range that would have been caught and retried earlier for less, plus the same fixed cost on the truly hung 5% either way — that bucket never finishes under any of these values, so a longer T only makes detecting it more expensive, never less. The curve is shallow on the right and steep on the left, which is the useful finding: overshooting the minimum is mildly wasteful, undershooting it is expensive, and “it felt long enough” tells you nothing about which side you’re on.

The number moves under you

That minimum isn’t a constant you set once. It shifts with three things you should actually be watching instead of the timeout value itself:

The shape of the tail. If a dependency’s p99 creeps up — more common than people expect — the “slow but real” bucket gets fatter and the same T starts manufacturing false timeouts it didn’t before. A timeout tuned against last quarter’s latency distribution is tuned against a distribution you no longer have.

The retry multiplier. If your retries are cheap and idempotent, cutting things off early costs less, so a lower T looks better. If a retry means re-doing expensive, non-idempotent work — the exact problem idempotency keys exist to solve — a false timeout is much more expensive than the model above assumes, and the optimum shifts higher.

Idle cost relative to compute cost. A worker sitting idle for 60 seconds is nearly free on a cheap always-on box and genuinely expensive on a metered, per-second-billed one. IDLE_RATE isn’t a universal constant; it’s your infrastructure’s pricing, and it changes the optimum’s location, sometimes by a lot.

None of these are things you set once and forget. They’re things you monitor and re-tune, the same way you’d re-tune a retry budget as failure rates drift.

Detecting when you have it wrong

You don’t need the full distribution to know you’re miscalibrated — two counters tell you which side of the plateau you’ve fallen off:

def outcome(duration, T):
    if duration <= T:
        return "completed"
    return "timed_out"

# Over a window of calls, watch this ratio:
def timeout_health(log):
    timed_out = sum(1 for d in log if outcome(d, T=30) == "timed_out")
    # if a large share of timeouts were "barely" over T, you're cutting real work
    near_miss = sum(1 for d in log if 30 < d <= 30 * 1.3)
    return timed_out / len(log), near_miss / max(timed_out, 1)

If near_miss is a large fraction of your timeouts, most of them were calls that would have finished a few seconds later — that’s the false-timeout bucket, and it means T is too low for the current distribution. If timeouts are rare but the ones that happen run to the full T with nothing near-miss, you’re probably fine on precision but paying more idle cost than you need to on true hangs — worth checking whether T can come down without the near-miss ratio climbing.

The one-line version

A timeout isn’t a safety margin, it’s a bet with two ways to lose: too short and you pay to redo real work that was about to finish; too long and you pay to sit idle on work that was never going to. Both losses have a price, the price depends on your actual latency tail and retry cost — not on what feels safe — and the number that minimizes total cost is a calculation you can run, not a constant you inherit from whoever set it first. Compute it, then watch the near-miss ratio to know when the distribution has moved out from under you.