Exactly-Once Was Hard Before Your Agent Could Retry Itself

We spent two decades teaching services to retry safely. The playbook is well worn: a client generates a unique idempotency key, attaches it to the request, and the server records the key alongside the result inside the same transaction that performs the work. A dropped connection, a timeout, a 500 — the client retries with the same key, the server recognizes it, and returns the recorded result instead of charging the card twice. Stripe shipped this pattern years ago and it became table stakes for any API that touches money.

That entire design rests on one assumption nobody wrote down: the caller repeats its request byte-for-byte. The retry carries the same key because the retry is the same code path re-executing with the same variables. Break that assumption and the whole scheme quietly stops working.

An LLM agent breaks it on the first retry. When a tool call times out, the agent doesn't replay a saved HTTP request — it reasons again, from a context window that now includes the timeout error, and emits a new tool call. Sometimes that call is identical. Often it isn't. The model rewords the argument, reorders JSON keys, rounds a number differently, or "helpfully" adds a field it left out the first time. If your idempotency key is a hash of the request body, congratulations: the retry produces a different key, sails past your dedup check, and executes the side effect a second time. You built exactly-once for a caller that no longer exists.

The caller is non-deterministic now

Classic distributed-systems reliability draws a clean line: the decision about what to do can be messy, but the execution must be deterministic. Agent runtimes shove a language model directly into the decision phase and then let its output drive execution without a translation layer in between. The model's job is to be creative and context-sensitive. That is exactly the property you do not want in the component that generates your dedup key.

Industry reports put the agent retry rate somewhere in the 15–30% range across timeouts, validation failures, and the model simply deciding its last attempt didn't work. So this is not a rare edge case you can defer. Roughly one in four tool calls in a busy agent is a candidate for duplication, and the duplicates don't look like duplicates because a stochastic process generated them.

There's a second, subtler trap. Agents retry for two genuinely different reasons, and they require opposite handling:

• Transient retry: the tool errored or timed out and the agent is re-attempting the same intent. You want strict idempotency — return the cached result, do not re-execute.
• Sampling retry: the agent got a valid result, didn't like it, and wants a genuinely new attempt. You want a fresh execution.

A naive content-hash key can't tell these apart. Two "re-run this query" calls that happen to serialize identically get collapsed into one when the model actually wanted two. Meanwhile two "charge the customer" calls that differ only in a reworded memo field get executed twice when the model meant one. The failure cuts both directions.

Stop deriving the key from the request

The fix is to stop treating the model's output as the source of the idempotency key. The key must come from the workflow's position, not from the bytes the model happened to emit at execution time.

Concretely: a tool call is uniquely identified by where it sits in the agent's plan, not by its arguments. If your runtime assigns a stable workflow_id (or run ID) to each agent execution and a stable step index to each planned action, then the idempotency key becomes something like {workflow_id}:{step_id}:{tool_name}. Retrying step 4 of run abc123 always yields the same key regardless of how the model rephrases the arguments on the second pass. The rule practitioners keep arriving at: your idempotency key must be deterministic from the workflow context, not from the execution moment.

This flips the usual advice. In HTTP land the client generates a UUID and reuses it; the payload can be whatever. In agent land the payload is the untrustworthy part and the position is the anchor. Same (workflow_id, tool, scope) means same effect and returns the recorded result. A different workflow_id with identical arguments means an independent, intentional effect — even if the tool and args are byte-for-byte the same. That's the property content hashing can never give you, because content hashing conflates "same request" with "same intent."

Build the ledger the tool checks first

Position-derived keys need somewhere to live. That's an action ledger — a durable table the tool consults before it touches the outside world. A workable schema is unglamorous:

• idempotency_key as the primary key (derived from workflow position)
• status: PENDING, SUCCESS, or FAILED
• result_data and error_data as JSON
• millisecond timestamps and an index on (agent_id, action_type, created_at)

The one rule that matters is ordering: check the ledger, then execute the side effect, then record the result — in that sequence. Checking after execution is the single most common way teams ship a broken idempotency layer; the duplicate has already happened by the time you look.

Concurrency is where this gets real. Two tool calls with the same key can arrive within milliseconds — an agent that fires parallel actions, or a retry racing the original that never actually died. Don't hand-roll a read-then-write; you'll lose the race. Insert a PENDING row under a unique constraint and let the database arbitrate. The insert that wins executes; the insert that fails with an integrity violation knows another attempt already holds the key and waits for that result rather than duplicating the work. The uniqueness guarantee lives in the storage engine, not in your application logic, because your application logic is racing itself.

Classify failures while you're here, because the agent will retry the ones you'd rather it didn't. Timeouts, connection drops, 429s, and 503s are transient and retry-safe. A 401, 403, or 404 is permanent — retrying just burns tokens and latency. When you genuinely can't tell, default to transient; a safe retry against an idempotent tool is cheap, but silently dropping a legitimate action is not.

When byte-identity isn't enough: semantic dedup

Position-derived keys handle the common case cleanly, but not every duplicate comes from a retry of the same planned step. Sometimes the agent re-derives the same intent from two different reasoning paths — it decides twice, at different points in its plan, to send the same email or file the same ticket. Different step IDs, same real-world effect. Position keys won't catch that because, by design, they treat different positions as independent.

This is where teams add a semantic layer in front of execution. Embed the proposed tool call, compare it against recent calls from the same session, and if cosine similarity crosses a threshold — practitioners cite roughly 0.9 for near-identical and 0.85 for equivalent-intent — treat it as a probable duplicate and require confirmation before executing. It's a soft gate, not a hard key, and you tune it toward false positives (block a real action, ask again) rather than false negatives (let a duplicate through). Use it as a backstop, not as your primary mechanism; an embedding threshold is a heuristic, and heuristics don't belong on the only thing standing between your agent and a double charge.

Durable execution moves the problem, it doesn't dissolve it

The heavier answer is durable execution — Temporal, Restate, DBOS, Inngest, and the checkpointing layers now shipping inside LangGraph and the major agent SDKs. These frameworks persist completed steps and replay the workflow after a crash, skipping anything already done and injecting the stored result. It's a genuinely good pattern and it eliminates a large class of "the worker died mid-plan" duplicates for free.

But read the fine print every one of these systems includes. If a tool call succeeds and the worker crashes before the result is checkpointed, replay will re-issue that call. Durable execution guarantees your workflow code resumes; it does not guarantee the external world only saw your side effect once. The framework docs say it plainly: any tool that writes external state must still carry an idempotency key tied to workflow state, or replay will duplicate the payment, the ticket, the deploy. Durability narrows the window in which a duplicate can occur. It does not close it. The ledger is still load-bearing.

What to actually do

Exactly-once was always a polite fiction — what real systems ship is at-least-once delivery plus idempotent processing, and the effect comes out exactly once. Agents don't change that math. They just retire the assumption that made the client half of it easy, because the client is now a model that rewrites its own requests.

So move the anchor. Derive idempotency keys from the agent's workflow position, not from the tokens it emitted. Put a ledger in front of every side-effecting tool and check it before you act, arbitrating concurrent keys in the database rather than in your code. Distinguish a transient retry from a deliberate re-sample so you don't collapse two intended actions into one. Add semantic dedup as a backstop for cross-path repeats, and lean on durable execution to shrink the crash window — but keep the ledger, because the framework won't cover the last gap.

The uncomfortable takeaway: the reliability primitive you trusted for a decade encoded an assumption about your caller, and your caller changed. Every place your code says "the retry will look the same" is now a latent double-execution bug. Go find those assumptions before your users find them for you.