I’ve written twice already about what retries cost you. This post is about something worse than cost: the retry that succeeds twice. Your agent calls charge_card, the network hiccups on the way back, the response never arrives, the loop retries — and now the customer is charged twice. Nothing errored. No exception was swallowed. The bill is correct on your side and wrong on theirs, and the model that “did the work” has no idea it happened.

The failure here isn’t the retry. Retrying is right — a timed-out request genuinely might not have landed. The failure is that the write had no way to recognize it had already run. That’s a property of the tool, not the loop, and it’s the single most under-built property in agent tooling I see.

At-least-once is the default you’re already running

Distributed systems people have a name for this. When a caller can’t tell whether a request succeeded — because the failure happened after the work but before the acknowledgment — it has three choices. Retry and maybe do it twice (at-least-once). Don’t retry and maybe do it zero times (at-most-once). Or do the engineering to make retries safe and get exactly-once effects.

Almost every agent loop I’ve read is at-least-once by construction and at-most-once by hope. It retries on error (at-least-once), and it assumes each tool runs at most once (at-most-once), and those two assumptions are contradictory. The gap between them is a duplicated side effect waiting for the first flaky network call.

And agents make it worse than a normal retry loop, for a reason specific to how they work: the model retries too, on its own, above your retry logic. A tool returns a timeout observation, the model reads it, decides the action didn’t go through, and calls the tool again — a second retry stacked on whatever your orchestration already did. You can cap your own retries. You cannot cap the model’s judgment. So even a single-retry policy can fire a write two or three times, and no amount of tuning max_retries closes it. The only close is making the write itself idempotent.

What idempotent actually has to mean here

Idempotent means calling it twice has the same effect as calling it once. The web loves to illustrate this with PUT versus POST, which is nearly useless for agents, because the interesting writes an agent makes — charge this card, send this email, create this ticket, book this room — are all POST-shaped: each call is meant to create a new thing. You can’t make “create a charge” idempotent by relabeling it. You make it idempotent by giving each intended charge a stable identity, so the second call carrying the same identity is recognized as the same charge and collapses into the first.

That identity is an idempotency key: a token that names the intent, generated once, sent with every attempt. Stripe, and most serious payment and messaging APIs, take one directly as a header. The whole trick is deriving a key that’s stable across retries but distinct across genuinely-different actions — and for an agent, that derivation is the part everyone gets wrong.

Deriving a key from intent, not from the moment

The naive key is a fresh UUID. It’s also wrong, because a fresh UUID is regenerated on every attempt, so the retry carries a different key and the dedup never triggers. The key has to be a deterministic function of what the agent is trying to do, computed once and reused for every attempt of that same intent.

import hashlib, json

def idempotency_key(tool: str, args: dict, scope: str) -> str:
    """A stable key for one *intended* effect.

    Same (tool, args, scope) -> same key -> retries collapse.
    Different intent -> different key -> genuinely-new actions still go through.
    """
    # Canonicalize args so key ordering / whitespace can't split one intent
    # into two keys.
    canonical = json.dumps(args, sort_keys=True, separators=(",", ":"))
    material = f"{scope}\x00{tool}\x00{canonical}"
    return hashlib.sha256(material.encode()).hexdigest()[:32]

The subtle field is scope. It’s what makes two legitimately distinct calls with identical arguments get distinct keys — so it decides your dedup window, and getting it wrong breaks in one of two directions. Too broad a scope (say, the whole mission ID) and the agent’s second, genuinely-intended “email the customer” of the day silently vanishes as a “duplicate.” Too narrow (a fresh value per attempt) and nothing dedups at all. The right scope is the unit of work the action belongs to: the specific step, the specific order, the specific approval — the thing that, if repeated, means “the same effect,” and if new, means “a new effect.” Choosing it is a modeling decision, not a default, and it’s the one you should actually think about.

A wrapper that makes any write safe to retry

Given a stable key, dedup is a small amount of boring, essential plumbing: check whether this key already ran, and if so return the stored result instead of running again. The store must be durable (survive a restart — an in-memory dict dedups within one run and forgets across the crash that caused the retry) and atomic on reserve (two concurrent attempts must not both see “not yet run”).

import sqlite3, json, time

class IdempotentWrites:
    """Wrap a side-effecting tool so repeated calls with the same key run once."""

    def __init__(self, db="idem.sqlite"):
        self.db = sqlite3.connect(db, isolation_level=None)  # autocommit
        self.db.execute("""
            CREATE TABLE IF NOT EXISTS effects (
                key     TEXT PRIMARY KEY,
                status  TEXT NOT NULL,          -- 'running' | 'done'
                result  TEXT,
                ts      REAL NOT NULL
            )""")

    def run(self, key: str, fn, *args, **kwargs):
        # Atomic reserve: INSERT fails if the key already exists, so exactly one
        # caller wins the right to execute. No check-then-act race.
        try:
            self.db.execute(
                "INSERT INTO effects(key, status, ts) VALUES (?, 'running', ?)",
                (key, time.time()))
        except sqlite3.IntegrityError:
            return self._await_existing(key)   # someone else owns this effect

        try:
            result = fn(*args, **kwargs)        # the real, un-retryable write
        except Exception:
            # The effect may or may not have landed. Release the reservation so a
            # deliberate retry can try again — do NOT mark 'done'.
            self.db.execute("DELETE FROM effects WHERE key=?", (key,))
            raise

        self.db.execute("UPDATE effects SET status='done', result=? WHERE key=?",
                        (json.dumps(result), key))
        return result

    def _await_existing(self, key: str):
        for _ in range(50):                    # bounded wait for the winner
            row = self.db.execute(
                "SELECT status, result FROM effects WHERE key=?", (key,)).fetchone()
            if row and row[0] == "done":
                return json.loads(row[1])       # replay the first call's result
            time.sleep(0.1)
        raise TimeoutError(f"in-flight effect {key[:8]} did not settle")

Now the loop wraps every write, and retries — yours and the model’s — collapse:

idem = IdempotentWrites()

def charge_card(amount, customer):            # the real, dangerous call
    return payments.charge(amount=amount, customer=customer)  # POST, not safe alone

def tool_charge(args, step_id):
    key = idempotency_key("charge_card", args, scope=step_id)
    return idem.run(key, charge_card, args["amount"], args["customer"])

Call tool_charge five times for the same step and the card is charged once; the other four return the first charge’s result. The model sees a clean success every time and stops retrying, which is exactly the observation you wanted it to have.

Two details that aren’t optional. First, the except branch deletes the reservation rather than marking it done — because a write that threw might have half-landed, and you want a subsequent deliberate retry to be allowed, not silently swallowed as “already done.” (Whether that retry is itself safe loops back to the API supporting real idempotency keys end to end; the wrapper makes your layer honest, not the vendor’s.) Second, _await_existing is bounded. An unbounded wait on an in-flight effect is just loop drift with extra steps.

What I’d actually do

  1. Classify every tool as read or write, and mean it. Reads retry freely. Writes do not retry unless they carry an idempotency key. If you can’t articulate a tool’s dedup scope, you don’t yet understand what retrying it does — that’s the signal to stop and model it.
  2. Derive keys from intent, once. A UUID minted per attempt is the bug that looks like a fix. The key is a function of the action and its scope, computed before the first attempt and carried through every retry.
  3. Push the key to the vendor when they take one. Stripe, SendGrid, and most serious write APIs accept an idempotency key directly; your wrapper is a fallback for the ones that don’t, and belt-and-suspenders for the ones that do.
  4. Store durably, reserve atomically, release on failure. In-memory dedup forgets across exactly the crash that triggers the retry. And mark “done” only after the effect actually completed — a reservation is not a result.

The thing to internalize is that “retry” and “side effect” are two words that should never sit next to each other unqualified. A retried read is a non-event. A retried write is a correctness bug unless you did the work to make it not one — and the model, cheerfully calling your tool a third time, will never do that work for you.

The code here is a minimal illustration, not a payments library — real money handling wants the vendor’s own idempotency support, reconciliation, and an audit trail. The at-least-once / exactly-once framing is standard distributed-systems vocabulary; the agent-specific twist is the second retry loop living inside the model’s own judgment, which no orchestration-level cap can bound.