The fleet-patterns post explained how to cap retry costs across a fleet. The observability post covered idempotency keys and correlation. But you still can’t answer the question that matters in production: are retries working, or are they just making things slower and more expensive?
A retry either recovers a transient failure or burns budget on a permanent one. Without visibility into which is happening, you’re flying blind — you can’t tell if your retry budget is too generous (wasting money) or too stingy (giving up too early), and you can’t tell if a retry storm is fixing an outage or amplifying it.
This post is about three metrics that make retries visible, and the instrumentation pattern that produces them.
The blind spot
Most production systems log retry attempts, but they log them as events: “attempt 1 failed, attempt 2 succeeded” or “attempt 3: gave up.” That’s useful for reconstructing a specific failure after the fact, but it doesn’t tell you about the pattern across a fleet.
Consider a real scenario: one service starts failing. Its clients see errors and begin retrying. Some of those retries succeed — the service recovers momentarily — but then it fails again. One client retrying a few times is reasonable. Twelve clients each retrying a few times is a retry storm. Hundreds of clients each retrying is a cascade.
How do you know which one is happening? You count. But most monitoring systems don’t emit “how many retries happened, and how many of them succeeded” as a metric — they log individual events that aggregation tools have to scrape together later. By then it’s too late to alert on it in real-time.
Three metrics that matter
Metric 1: retry success rate (the one that steers everything else)
This is the fraction of retried operations that ultimately succeeded, measured per operation type (or per dependency, depending on your architecture). High success rate (>70%) means retries are earning their cost — most of the time they’re fixing transient failures, not burning budget. Low success rate (<20%) means you’re likely hitting permanent failures repeatedly.
import dataclasses
from collections import defaultdict
@dataclasses.dataclass
class RetryMetrics:
operation_name: str
total_retried: int = 0 # operations we retried at least once
retried_succeeded: int = 0 # of those, how many eventually succeeded
retried_failed: int = 0 # of those, how many exhausted retries
total_attempts_consumed: int = 0 # across all retried ops
def success_rate(metrics: RetryMetrics) -> float | None:
if metrics.total_retried == 0:
return None
return metrics.retried_succeeded / metrics.total_retried
# Example from production:
auth_retries = RetryMetrics(
operation_name="token_refresh",
total_retried=145,
retried_succeeded=102,
retried_failed=43,
total_attempts_consumed=312 # 102*2 (mostly 1 retry each) + 43*3 (gave up after 3)
)
print(f"Token refresh success rate: {success_rate(auth_retries):.1%}")
# Output: Token refresh success rate: 70.3%
This single number steers everything downstream. If it’s high, your retry logic is working as intended. If it drops suddenly, something structural has broken (a service misconfiguration, a network partition, a permanent bug). If it’s stubbornly low, you’re retrying operations you shouldn’t be retrying — time to audit your error classification.
Metric 2: retry amplification factor
This is the ratio of total attempts made to unique operations attempted, measured per class of failure. A factor of 1.0 means no retries happened (all operations succeeded on the first try). A factor of 1.5 means on average each operation was attempted 1.5 times. A factor of 3.0 means each operation was attempted three times on average — a strong signal of a retry storm.
def amplification_factor(metrics: RetryMetrics) -> float:
if metrics.total_retried == 0:
return 1.0 # no retries = factor of 1
# Average attempts per retried operation
return metrics.total_attempts_consumed / metrics.total_retried
# Continuing the token refresh example:
print(f"Amplification factor: {amplification_factor(auth_retries):.2f}x")
# Output: Amplification factor: 2.15x
# Interpretation: 70% succeeded on second try, 30% needed 3+ attempts before failing
The amplification factor is your early warning system. When it spikes without a corresponding success-rate drop, it means a failure is either:
- Becoming steadily permanent (more retries needed per operation) — suggests a degraded dependency
- Becoming intermittent in a pathological way (lots of oscillation) — suggests a resource exhaustion or cascading failure
Combine it with the success rate: high amplification + low success = major incident. High amplification + high success = your retries are working hard but earning it. Low amplification + high success = boring and healthy.
Metric 3: retry latency percentile impact
This is the p50, p95, p99 increase in operation latency for operations that were retried versus those that succeeded on the first attempt. It tells you the time cost of retrying, which is often invisible in cost accounting (you count the token cost but forget that the user was waiting).
import statistics
@dataclasses.dataclass
class PercentileMetrics:
operation_name: str
first_try_latencies_ms: list[float] # operations that succeeded immediately
retried_latencies_ms: list[float] # operations that needed retries
def latency_percentile_cost(pctile_metrics: PercentileMetrics):
for p in [50, 95, 99]:
first_try = statistics.quantiles(pctile_metrics.first_try_latencies_ms, n=100)[p-1]
retried = statistics.quantiles(pctile_metrics.retried_latencies_ms, n=100)[p-1]
overhead = retried - first_try
print(f"p{p} latency: +{overhead:.0f}ms for retried operations ({retried:.0f}ms vs {first_try:.0f}ms)")
# Example:
# p50 latency: +120ms for retried operations (350ms vs 230ms)
# p95 latency: +890ms for retried operations (2100ms vs 1210ms)
# p99 latency: +3200ms for retried operations (4800ms vs 1600ms)
This matters because latency is a user-facing cost. A user’s request hanging for 5 seconds instead of 2 seconds due to retries isn’t just a token cost — it’s potential abandonment, and each 100ms of latency costs you revenue. If your retry strategy is adding 3+ seconds to p99, you might be better off failing fast and letting the user retry from their side.
The instrumentation pattern
These three metrics don’t come from logs — they come from instrumentation you add to your retry code. Here’s the pattern that produces all three without requiring an external tracing system:
import time
from dataclasses import dataclass
import json
@dataclass
class RetryAttempt:
operation_id: str
operation_name: str
attempt_num: int
start_ms: float
status: str # "success", "transient_error", "permanent_error"
error_code: str | None = None
result: str | None = None # "succeeded_on_retry" or "exhausted" or "succeeded_first_try"
class RetryInstrument:
def __init__(self):
self.attempts = [] # emit to metrics sink periodically
def execute_with_retries(self, op_id, op_name, fn, max_retries=3, backoff_base=1.0):
"""
Execute fn with exponential backoff retries.
Emit structured metrics for every attempt.
"""
start_ms = time.monotonic() * 1000
last_error = None
for attempt_num in range(1, max_retries + 1):
attempt_start = time.monotonic() * 1000
try:
result = fn()
latency = time.monotonic() * 1000 - attempt_start
# Record: this attempt succeeded
self.attempts.append(RetryAttempt(
operation_id=op_id,
operation_name=op_name,
attempt_num=attempt_num,
start_ms=attempt_start,
status="success",
result="succeeded_first_try" if attempt_num == 1 else "succeeded_on_retry",
))
# Emit the outcome
self._emit({
"operation_id": op_id,
"operation_name": op_name,
"total_attempts": attempt_num,
"latency_ms": time.monotonic() * 1000 - start_ms,
"success": True,
"result": "succeeded_on_retry" if attempt_num > 1 else "first_try",
})
return result
except Exception as e:
latency = time.monotonic() * 1000 - attempt_start
last_error = e
# Classify the error
is_permanent = self._is_permanent_error(e)
status = "permanent_error" if is_permanent else "transient_error"
self.attempts.append(RetryAttempt(
operation_id=op_id,
operation_name=op_name,
attempt_num=attempt_num,
start_ms=attempt_start,
status=status,
error_code=self._error_code(e),
))
if is_permanent or attempt_num == max_retries:
# Give up: emit failure
self._emit({
"operation_id": op_id,
"operation_name": op_name,
"total_attempts": attempt_num,
"latency_ms": time.monotonic() * 1000 - start_ms,
"success": False,
"error_code": self._error_code(e),
"reason": "permanent_error" if is_permanent else "retries_exhausted",
})
raise
# Retry with exponential backoff
wait_s = min(backoff_base * (2 ** (attempt_num - 1)), 30)
time.sleep(wait_s)
def _is_permanent_error(self, e: Exception) -> bool:
# Your error classification logic from the postmortem
# 4xx errors (except 429, 503 on client-side) are permanent
# Timeouts, 5xx are transient
if hasattr(e, 'status_code'):
return 400 <= e.status_code < 500 and e.status_code not in [429, 503]
return False
def _error_code(self, e: Exception) -> str:
if hasattr(e, 'status_code'):
return f"http_{e.status_code}"
return type(e).__name__
def _emit(self, event: dict):
# Send to your metrics collector (Datadog, Prometheus, CloudWatch, etc.)
# In production, batch these and send periodically
print(json.dumps({"retry_outcome": event}))
def aggregate_metrics(self) -> dict:
"""Compute the three metrics from buffered attempts."""
by_operation = {}
for attempt in self.attempts:
key = attempt.operation_name
if key not in by_operation:
by_operation[key] = {
"total_ops": 0,
"succeeded_ops": 0,
"failed_ops": 0,
"total_attempts": 0,
"latencies": []
}
# Build chains: group attempts by operation_id to see each operation's journey
by_op_id = {}
for attempt in self.attempts:
if attempt.operation_id not in by_op_id:
by_op_id[attempt.operation_id] = []
by_op_id[attempt.operation_id].append(attempt)
for op_id, chain in by_op_id.items():
last_attempt = chain[-1]
op_name = last_attempt.operation_name
by_operation[op_name]["total_ops"] += 1
by_operation[op_name]["total_attempts"] += len(chain)
if last_attempt.status == "success":
by_operation[op_name]["succeeded_ops"] += 1
else:
by_operation[op_name]["failed_ops"] += 1
metrics = {}
for op_name, agg in by_operation.items():
metrics[op_name] = {
"success_rate": (agg["succeeded_ops"] / agg["total_ops"]) if agg["total_ops"] > 0 else 0,
"amplification_factor": (agg["total_attempts"] / agg["total_ops"]) if agg["total_ops"] > 0 else 1.0,
"total_operations": agg["total_ops"],
"total_attempts": agg["total_attempts"],
}
return metrics
# Usage in a real agent loop:
instrument = RetryInstrument()
def fetch_enrichment_data(user_id):
def call():
# Your actual API call here
response = requests.get(f"https://api.example.com/enrich/{user_id}")
response.raise_for_status()
return response.json()
return instrument.execute_with_retries(
op_id=f"enrich_{user_id}_{time.time()}",
op_name="enrichment_api",
fn=call,
max_retries=3,
)
# Periodically aggregate and emit the three metrics:
metrics = instrument.aggregate_metrics()
for op_name, m in metrics.items():
print(f"{op_name}: success_rate={m['success_rate']:.1%}, amplification={m['amplification_factor']:.2f}x")
When to act on these metrics
- Success rate drops below 50%: Immediate investigation. Either your error classification is wrong (you’re retrying permanent failures) or a dependency is genuinely broken (time to page someone). Either way, the retry logic isn’t helping — consider failing fast.
- Amplification factor spikes above 2.5x: A specific operation class is struggling. Not necessarily bad (if success rate is still >70%), but it’s burning budget. Flag it for the oncall engineer to watch.
- P99 latency for retried operations exceeds user timeout: Your retries are slower than the timeout a user-facing client uses to give up. You’re making the user’s experience worse. Reduce per-step retry caps or fail faster.
These three metrics, emitted continuously into your observability platform and wired into dashboards and alerts, turn retries from an invisible implementation detail into a visible, steerable system. They’re the bridge between the theory (retry budgets, fleet patterns) and production operations — the metrics that let you ask “is this working?” and get an answer.