The fleet-patterns post explained the patterns that keep a fleet from drowning in its own retries. The when-to-give-up post showed when to retry at all — which layer gets to make the decision, and why most code doesn’t ask the right questions. (Both of those build on the cost model and a real incident that motivated it.) This post is about what you need to know to make that decision trustworthy in the first place.
The core problem: a retry attempt looks the same whether it’s recovering from a transient network hiccup or whether you’re trapped in a loop burning your budget on a permanent failure. Without visibility into what’s happening, your retry logic is guessing.
The foundation: idempotency keys
An idempotency key is a unique token attached to a request that tells the system “if you’ve seen this before, return the cached result instead of replaying the work.” It’s not directly a retry concern — it’s a consequence of a deeper rule: retries are only safe when the operation is idempotent, and idempotency is only verifiable if the operation is labeled with a stable identity.
import uuid, time
class RetryableCall:
def __init__(self, operation_name, user_id, resource_id):
self.idempotency_key = f"{user_id}:{resource_id}:{operation_name}:{int(time.time() * 1000)}"
# ^ stable within a retry window (e.g., one second), fresh across retries >1s apart
self.attempt = 0
def call(self, client):
self.attempt += 1
headers = {"X-Idempotency-Key": self.idempotency_key}
return client.do_work(headers=headers)
The key construction matters. If your idempotency key is just a UUID per task (not per attempt), all retries of the same task share it — which is what you want. If it’s per-millisecond, two retries 100ms apart get different keys, and the server will double-process. The stability window should match your retry window: if you retry up to 5 seconds, the key should be stable for 5+ seconds.
This is how you tell the server “this is attempt 3, but if you cached the result from attempt 1, use that.” Without it, your retry is a replay: the server genuinely executes the operation twice.
Retry-attempt correlation: the chain of custody
Once you own idempotency, you need to track which attempt is which. This is where retry-attempt correlation comes in — a log trace that chains a series of attempts together so a human (or a monitoring system) can follow the story of a single logical operation through its retries.
class CorrelatedRetry:
def __init__(self, logical_op_id):
self.logical_op_id = logical_op_id # stable across all attempts of this op
self.attempt_sequence = []
def log_attempt(self, attempt_num, latency_ms, status, error=None):
entry = {
"logical_op_id": self.logical_op_id,
"attempt": attempt_num,
"latency_ms": latency_ms,
"status": status,
"error": error,
"timestamp": time.time(),
}
self.attempt_sequence.append(entry)
# emit to logs / traces
logger.info("retry_attempt", extra=entry)
# Example usage:
logical_id = str(uuid.uuid4())
retry_tracer = CorrelatedRetry(logical_id)
for attempt in range(1, max_attempts + 1):
try:
start = time.monotonic()
result = call_downstream()
latency = (time.monotonic() - start) * 1000
retry_tracer.log_attempt(attempt, latency, "success")
return result
except Exception as e:
latency = (time.monotonic() - start) * 1000
retry_tracer.log_attempt(attempt, latency, "failed", str(e))
if attempt == max_attempts:
raise
What does this buy you? When a request fails after 3 retries, you can ask: “Did each attempt fail for the same reason, or did they fail differently?” If all three attempts get the same error (e.g., “rate limit: retry after 60s”), you know the failure is not transient — it’s a real limit you’ve hit. If the first fails with a timeout and the second succeeds, you have evidence the transient was actually transient and the retry worked.
The logs become your debugging surface. When ops calls and says “this user’s transaction failed,” you trace by logical_op_id and see not just “failed: 500” but “attempt 1: timeout after 3.2s; attempt 2: timeout after 2.8s; attempt 3: failed upstream rate limit.”
Cost accounting: measuring what you’re actually spending
The retry budget bounds HOW MUCH you retry. Cost accounting tells you WHAT you’re spending — and whether the budget is actually protecting you or you’re gaming it.
class RetryBudgetObserver:
def __init__(self, budget_name):
self.budget_name = budget_name
self.total_attempts = 0
self.successful_retries = 0 # retries that led to success
self.failed_retries = 0 # retries that failed
self.abandoned_retries = 0 # retries we didn't attempt (budget exhausted)
self.total_cost_usd = 0.0
def on_retry_attempt(self, cost_usd, eventual_success=None):
self.total_attempts += 1
self.total_cost_usd += cost_usd
if eventual_success is None:
self.abandoned_retries += 1 # budget said no
elif eventual_success:
self.successful_retries += 1
else:
self.failed_retries += 1
def report(self):
roi = (self.successful_retries / max(1, self.successful_retries + self.failed_retries)) if self.successful_retries + self.failed_retries > 0 else 0
return {
"budget_name": self.budget_name,
"total_attempts": self.total_attempts,
"successful_retries": self.successful_retries,
"failed_retries": self.failed_retries,
"abandoned": self.abandoned_retries,
"roi": roi, # success rate of attempts we made
"cost_usd": self.total_cost_usd,
"cost_per_success": self.total_cost_usd / max(1, self.successful_retries),
}
ROI of 30% means one in three retry attempts recovered the operation. ROI of 5% means your budget is burning on dead ends — either your budget is too generous, or you’re retrying things that won’t ever succeed.
The cost-per-success metric is the one that matters most in production. If your cost per successful retry is $0.001 and your success rate is 95%, the budget is working. If it’s $1.00 per success (because you’re retrying expensive operations), the question becomes “is that cost cheaper than the user’s alternative?” — a circuit breaker may make more sense than a retry budget at that scale.
Putting it together: the trace becomes the decision
When all three pieces are wired up, the observability surface becomes active. You don’t just log “retry happened” — you emit a structured record:
{
"logical_op_id": "550e8400-e29b-41d4-a716-446655440000",
"operation": "process_transaction",
"idempotency_key": "user:12345:transaction:1721779200000",
"attempt_sequence": [
{
"attempt": 1,
"status": "timeout",
"latency_ms": 30001,
"downstream": "payment_svc",
"error": "read timeout after 30s"
},
{
"attempt": 2,
"status": "timeout",
"latency_ms": 30002,
"downstream": "payment_svc",
"error": "read timeout after 30s"
},
{
"attempt": 3,
"status": "rate_limit",
"latency_ms": 245,
"downstream": "payment_svc",
"error": "429: too many requests"
}
],
"budget_name": "user_transactions",
"budget_decision": "abandon_further_retries",
"cost_usd": 0.0023,
"eventual_outcome": "failed"
}
This record tells a story: “The same operation hit two different failures — first timeouts (transient?), then a rate limit (hard limit). We stopped retrying. Cost: $0.0023, outcome: failed.” A human reading this can decide: “The rate limit kicked in after two timeouts — if we’d backed off longer, we might have succeeded. Or: the timeouts aren’t transient, they’re a symptom of cascade — backing off won’t help.”
The metrics that flow from this (success_rate per retry budget, cost_per_success, time_to_abandon) become the steering signal. When cost_per_success climbs above your threshold, it means your budget is chasing failures that won’t resolve — time to tighten it. When success_rate drops, it means your transient assumptions are wrong — the errors you thought would pass aren’t.
That’s how you make a retry decision trustworthy: you instrument it so thoroughly that the logs themselves tell you whether the decision is working or not.