Retries Without Duplicate Actions: Idempotency for AI Agents

An agent can submit a booking, lose the response, and restart with no reliable way to know whether the booking happened. If the restarted run treats missing information as failure and submits a fresh request, a customer may receive two bookings or a payment may be attempted twice.

Idempotency is the design discipline that lets repeated attempts refer to one intended business operation. It does not make independent systems commit atomically, and it cannot prove an external action happened exactly once. A sound implementation instead preserves the operation’s identity, records what is known, and stops when the outcome is uncertain.

This tutorial builds that behavior for a hypothetical booking-and-payment agent. The same patterns apply to other consequential tool calls, but the fields, confirmation rules and recovery evidence must match the business action. The focus is the tool boundary and its recovery path, not a general guide to deploying agents. For workflow structure and resumable execution, see designing long-running agent workflows.

Prerequisites and setup

Before writing a retry loop, identify the operation you want to protect. For the example, it is one booking request for one tenant and one customer, including a specific room, date range, guest count, and payment instruction. A retry is another attempt to carry out that same request. A new booking—even with identical details—is a new intention and needs a new operation identity.

You need a durable store for operation records, a worker or execution service that can load them after a restart, and a provider that exposes some way to inspect or safely repeat a request. The precise recovery options vary by provider. If an external system offers no reliable lookup and no documented retry behavior, the safe response to an unknown result may be manual reconciliation rather than another call.

You also need a way to identify the authenticated tenant and the actor requesting the action, and to validate the exact action before it reaches the provider. In the example, a booking approval covers the dates, room, guest count, total amount, currency, and recipient. The approval is checked before execution, bound to those values, and expires. Editing any material field changes the action that was approved.

Use separate identifiers for the business operation, a particular execution attempt, and the provider’s reference. They answer different questions: which intention is this, which try produced this log entry, and what record did the external system return? Keeping those distinctions makes a restart trace understandable instead of collapsing it into one ambiguous request ID.

For this walkthrough, assume a relational database and an execution worker. The SQL and pseudocode below are illustrative design examples, not tested drop-in code; adapt transaction syntax, data types, provider calls, and concurrency handling to your stack. The example assumes that workers can persist an operation before any external call and that the provider’s own behavior has been checked before choosing a retry policy.

flowchart TD
  accTitle: Idempotent operation recovery
  accDescr: Reserve a stable business operation before execution. A timeout requires reconciliation, and a confirmed result completes the same operation without inventing a new identity.
  A["Stable tenant and operation ID"] --> B["Reserve and validate payload"]
  B --> C["Verify permission and approval"]
  C --> D["Call provider with stable key"]
  D -->|"Confirmed"| E["Persist business result"]
  D -->|"Unknown"| F["Reconcile provider state"]
  F -->|"Confirmed"| E
  F -->|"Still unknown"| G["Hold for investigation"]

The operation identity survives retries and worker restarts. An uncertain response follows the reconciliation path; it does not authorize a new payment attempt with a different identity. The provider's retention and retry rules bound what recovery can safely do.

Step 1: Define the operation identity

Create an operation ID when the business intention is accepted into your workflow, not whenever a worker starts. Persist it with the tenant ID and any upstream request or workflow ID that helps explain why the operation exists. The worker may run once, restart, move to another machine, or be invoked again; none of those events creates a new intention.

A practical key can combine a tenant scope with that durable operation ID, for example tenant-42:booking-781. The exact format is an implementation choice. Avoid placing customer names, email addresses, payment details, or other sensitive values in a key that may appear in provider logs. Use a stable, opaque identifier and store the association to the business record locally.

Do not derive the identity only from the payload. Two customers can intentionally request identical bookings, and one customer can intentionally make the same request twice. If identical parameters automatically produce an identical key, a legitimate second booking can be mistaken for a retry of the first. Identity should represent intent; a payload digest should detect changes to that intent.

Build a canonical representation of the action and calculate a digest from it. Canonicalization means that equivalent values have one agreed representation: for example, dates use a fixed format, currency uses an agreed code, and optional fields have consistent treatment. Include a schema version so a future change in field interpretation does not silently make old and new requests appear equivalent.

A record might conceptually contain the following fields:

Expected result: a retry can retrieve the same operation record even when it runs in a new process. A second, genuinely new booking receives a different operation ID, even if its payload happens to match the first.

Step 2: Reserve the operation before calling out

Add a uniqueness constraint for the operation identity within its tenant. The database constraint is the final protection against two workers creating separate local records for what should be one operation. A prior “does this key exist?” query is not enough: two workers can both see no row before either inserts one.

An illustrative table definition follows. Adjust it to your database and operational needs; the important properties are the unique identity, durable payload digest, and a state field that can represent uncertainty.

CREATE TABLE agent_operations (
  tenant_id           TEXT NOT NULL,
  operation_id        TEXT NOT NULL,
  payload_hash        TEXT NOT NULL,
  payload_version     INTEGER NOT NULL,
  status              TEXT NOT NULL,
  provider_reference  TEXT,
  approval_reference  TEXT NOT NULL,
  approval_expires_at TIMESTAMP NOT NULL,
  created_at          TIMESTAMP NOT NULL,
  updated_at          TIMESTAMP NOT NULL,
  PRIMARY KEY (tenant_id, operation_id)
);

In one short database transaction, validate the tenant and operation identity, check that the approval still applies, and insert the reservation. If the key already exists, load its record. Compare its payload hash and version with the incoming request. The same key with a different material payload is a conflict to reject or quarantine; it is not permission to overwrite the original action.

Keep the transaction limited to local database work. Do not hold a database transaction open while waiting for an external provider. Network calls can take an unpredictable amount of time or fail without a conclusive result, and a long-held transaction does not make the provider part of the database commit.

Concurrent requests that race to create the same operation should converge on the one row admitted by the uniqueness rule. The losing worker then reads the existing state and follows that state’s recovery path. It must not respond to a uniqueness conflict by generating a different operation ID.

Expected result: before the first external call, one durable row exists for the intended action. A repeated request with the same key and same payload finds that row; a reused key with changed details is visibly rejected.

Step 3: Model states that tell the truth

Choose states for facts the system can establish, not labels that merely make a workflow look complete. A useful starting set is reserved, submitted, succeeded, failed, and reconciliation_required. Define what evidence permits each transition. For example, a timeout alone cannot justify failed, because it says what the worker observed, not what the provider did.

reserved means the local operation exists and is eligible for the next controlled action. submitted means an attempt was made or is being recorded, but it need not prove the provider accepted or completed the action. succeeded should require a confirmed result that can be associated with this operation. failed should mean a definitive failure under the provider’s documented behavior. reconciliation_required means the available evidence is insufficient to decide safely.

Record attempts separately from the operation’s identity. An attempt record can include an attempt ID, start and finish times, worker identity, outcome category, and a short error classification. Avoid copying sensitive payload data into every attempt row or log. The operation record answers what the business intended; the attempt history explains how execution proceeded.

If workers can overlap, use a claim or lease to reduce unnecessary concurrent work. A lease helps one worker coordinate the next attempt, but it is not a substitute for a stable provider idempotency key: a worker can continue after its lease expires, or a process can crash after the provider acts. Treat the lease as local scheduling protection, not proof that only one external call can ever occur.

Make allowed transitions explicit. For instance, a worker should not turn reconciliation_required back into reserved merely because a timer fired. A controlled reconciliation action must first establish why another attempt is appropriate. Similarly, a terminal local success should not be overwritten by a late timeout report from an older worker.

Expected result: an operator can tell the difference between confirmed success, definitive failure, and missing evidence. Dashboards and automated recovery do not disguise uncertainty as failure or success.

Step 4: Bind execution to the approved payload

Immediately before the external action, load the operation and confirm that its payload hash still matches the validated request. Check that the approval is present, unexpired, and bound to that same canonical payload. If the booking dates, amount, recipient, or another material field changed, stop and obtain a fresh decision through the business process.

This check belongs at the execution boundary, not only in the agent’s prompt or an earlier user interface. The model can propose an action, but the execution service should independently verify the tenant, current operation state, payload, and approval. A restart should reload those facts from durable records rather than trust a model-generated summary of what was approved.

Avoid treating an approval reference as blanket permission to execute any later version of an action. Store the digest or an equivalent immutable representation of the approved fields, along with the expiry. Define which fields are material for the operation. A display-only note might not change the action; a different date, amount, currency, or recipient usually does.

If approval expires while the provider outcome is unknown, do not automatically issue a new action. First reconcile the existing operation. If the provider already completed it, the expiry does not erase that fact. If no action occurred and a new execution is needed, revalidate and obtain a current approval before that execution.

Expected result: the provider call can be traced to the exact action that passed the current business check. A changed request cannot inherit an old approval simply because it reused an operation key.

Step 5: Call the provider with a stable retry identity

When the provider supports an idempotency mechanism, derive or store its key from the operation identity and reuse it for attempts that represent that same intended action. Do not include the attempt number in that key. An attempt ID changes each time; the operation key must not.

Provider semantics are bounded and provider-specific. Stripe documents stable idempotency keys for retrying requests under stated constraints, including retention behavior; they are not an indefinite exactly-once guarantee. Check the provider’s current rules for key scope, parameter matching, retention, and which responses are retained before deciding how long automatic retries are safe. Stripe’s idempotent request guidance is a concrete example of why those constraints matter.

Send the minimum required request, the stable provider key when supported, and a correlation reference that does not expose sensitive customer information. Before sending, persist enough local information to establish that an attempt is about to occur. Afterward, record the response classification and provider reference when available. Do not log credentials or full payment data as a shortcut to debugging.

The response handler needs more than a binary success/error branch. Treat a confirmed success as evidence to persist and verify. Treat a definitive rejection according to the provider’s documented semantics and your business rules. Treat a timeout, connection loss, malformed response, or worker crash as potentially unknown unless the provider gives reliable evidence otherwise.

Expected result: retries that truly represent the same operation reuse the same identity, and the attempt history records what each worker observed. The system does not assume that every error means no external action occurred.

Step 6: Recover from a lost response

Here is the central failure trace. It is hypothetical, but represents a common distributed-systems boundary. An agent has a valid, approved booking request with operation ID booking-781. The worker reserves the operation and sends the booking request. The provider creates the booking and returns a confirmation, but the worker crashes before saving that response.

At restart, the workflow loads booking-781 and finds that its local state does not contain a confirmed result. It does not create booking-782, infer that the booking failed, or ask the model to reconstruct a fresh request. It marks the outcome as unresolved if that has not already been done, then starts reconciliation for the original operation.

Reconciliation uses the provider’s documented options. If the provider can look up a booking using a stable reference, use that lookup and verify that the returned booking corresponds to the expected tenant and action. If the provider supports safe retry with the same key, retry only within its documented constraints. A returned prior result may establish what happened; a new key would instead risk expressing a second intention.

If the provider confirms the booking, persist its reference and result, then mark the operation successful. If the provider confirms no booking exists and the action is still approved and valid, follow the defined retry policy using the same operation identity. If evidence remains inconclusive, leave the record in reconciliation_required and route it for controlled follow-up. Do not let an agent invent certainty to clear a blocked workflow.

The recovery decision must account for elapsed time. A provider may retain a key only for a documented interval, or its lookup record may not be immediately available. If that period has passed, do not assume that replaying the same string still deduplicates the action. Consult provider records or a human-controlled reconciliation path before sending another request.

Durable workflow persistence also needs explicit replay boundaries around non-deterministic side effects; persistence alone does not make an external action safe to replay. LangGraph’s persistence guidance is one reference for that distinction. At the workflow level, persist the operation identity and state so a resumed run reaches the same controlled execution boundary instead of silently generating a new action.

Expected result: the restart either establishes the provider outcome, safely retries within known constraints, or leaves an explicit unresolved item. It never converts a missing response into a new business operation by default.

Step 7: Deduplicate delayed and repeated events

A successful provider response may not be the only path by which your system learns an action completed. A later webhook, queue message, or callback may report the same outcome. Store the provider event identity where one is available and enforce uniqueness for processing it. Associate the event with the operation and provider reference before triggering downstream work.

Separate recording an event from acting on it. First persist or claim the event in a local transaction. Then apply the corresponding business update in a way that can itself be retried safely. If event processing crashes after recording but before the update, the worker needs a resumable record. If it crashes after the update but before marking the event complete, the update must not cause a second customer notification or another downstream booking action.

For a downstream action, create its own operation identity. A notification, refund, calendar update, and payment capture are distinct intentions, even when one is triggered by a confirmed booking. Carrying the original booking key into every unrelated action can incorrectly collapse separate work. Instead, retain a causal link to the booking while giving each action its own stable identity and payload digest.

Keep event handling tolerant of duplicates and ordering. A delayed failure event should not blindly overwrite a later, verified success. Define which evidence is authoritative for each state transition and how conflicting records are investigated. Preserve raw event identifiers and timestamps needed for that analysis, subject to your data-retention practices.

Expected result: duplicate delivery of one event does not repeat its business effect, and downstream actions remain independently identifiable. Event replay becomes a recovery mechanism rather than an accidental way to create new work.

Step 8: Run an operational recovery test

Test at the tool boundary and through worker restart, not only by checking that a helper function generates the same string twice. The important property is that the whole path preserves one operation identity and blocks unsafe new side effects when the outcome is uncertain. In a provider test environment, inspect the provider-side result where available; otherwise verify the local operation history and provider reference returned by the test.

Use this failure scenario as a repeatable operational test. Configure a test worker to stop after the provider has accepted a request but before the local success record is written. Restart the workflow from persisted state. The exact mechanism for inducing the interruption depends on your environment; do not claim that a local unit test proves the provider’s behavior.

Capture the operation ID, attempt ID, provider reference if available, state transitions, and relevant timestamps in the test report. The test passes only if those records tell a coherent sequence and the number of intended provider actions is consistent with the scenario. If your test environment cannot expose provider-side counts, state that limit and verify the most authoritative records you do have.

Also test delayed recovery, not just an immediate restart. Let a test operation remain unresolved long enough to exercise the team’s escalation path and provider-key retention boundary. Confirm that an operator can distinguish “not yet checked” from “checked and absent,” and that the workflow does not retry automatically beyond the window your provider documentation supports.

Expected result: an operator can reproduce the crash boundary, show how the system recovered, and identify the evidence used to permit or block the next action. A green unit test without that evidence is not a recovery test.

Step 9: Make the execution trace observable

Log structured events that let an operator follow one operation across worker attempts without placing sensitive payloads in general-purpose logs. Useful fields include tenant scope, operation ID, attempt ID, state before and after, provider reference when known, error category, and reconciliation decision. Redact credentials and avoid logging full payment or customer details.

Track unresolved operations by age and state, not just raw tool-call failures. A rising count of timeouts may be inconvenient; a growing queue of unknown payment outcomes is a business risk that calls for a different response. Define who sees the queue, what evidence they need, and how a resolved item is returned to normal processing.

Build alerts around actionable conditions: an unresolved operation exceeds its permitted waiting window, the same key arrives with a changed payload, a provider reference conflicts with the stored operation, or automatic recovery is approaching a documented boundary. Avoid alerting on every retry as though each attempt were a new business failure; group attempts under the operation while preserving the individual trace.

Agent observability should connect model/tool activity to the durable operation record, so a reviewer can see what was proposed, what the execution service accepted, and what external evidence followed. For more on tracing agent execution, see approaches to agent observability. Reliability targets should measure completed and verified work, rather than treating every tool invocation as a success; see how to define agent SLOs around completed work.

Expected result: an incident responder can answer which business operation was intended, which attempts occurred, whether the provider outcome is known, and why the next action is allowed. If those questions require reconstructing a conversation from scattered text logs, the operation record is not doing enough work.

Step 10: Troubleshoot common implementation failures

A retry gets a different key. Find where the key is created. If it is generated inside the worker invocation, on each model tool call, or from the attempt ID, move identity creation to the durable business-operation boundary. Verify that restarting the same workflow reloads the original key instead of regenerating it.

A changed request reuses an existing key. Compare canonicalization and payload hashes. Check for omitted material fields, inconsistent currency or date normalization, and schema-version changes. Reject mismatches before reaching the provider. Do not “fix” the conflict by replacing the stored hash; that erases the evidence that the action changed.

A timeout is automatically marked failed. Separate transport outcome from business outcome. Add an unknown or reconciliation-required state and require a provider lookup, documented safe retry, or controlled human decision before another action. A client deadline is useful operational data, but it does not establish what the external system committed.

Two workers both call the provider. Inspect the database uniqueness rule, reservation transaction, and worker-claim logic. Confirm the losing worker loads the existing operation rather than creating a new one. Then verify that both calls, if they can overlap, use the same provider key and that the provider’s documented constraints cover the retry pattern.

The local record says success but there is no provider reference. Review what qualifies as success and whether a weak response, queued acknowledgement, or model-generated statement is being treated as business confirmation. Store the evidence that supports the transition. If the provider has only accepted work for later processing, represent that intermediate state rather than claiming the booking is complete.

A repeated event sends another notification. Identify whether deduplication applies only to event ingestion or also to the downstream notification operation. Record a stable identity for each side effect and make the local event transition resumable. A unique event row cannot by itself prevent duplication if the notification is outside that transaction and its own result is uncertain.

An unresolved item remains stuck. Check whether it has an owner, an age threshold, and an allowed next action. “Retry later” is not a complete policy. The recovery record should say what evidence is missing, which source can provide it, and whether the next step is automated lookup, a permitted same-key retry, or operator review.

Printable implementation worksheet

Use this compact worksheet during design review or an incident drill. Fill it with values specific to one consequential action; do not use broad answers such as “the agent retries safely.” Keep the completed record with the service’s operational runbook.

• Business intention: Name the single action represented by one operation, and distinguish it from a genuinely new request.
• Stable identity: State where the operation ID is created, how it survives a restart, and how tenant scope is represented.
• Payload binding: List the material fields, canonicalization rules, payload version, and behavior when the same key arrives with a different digest.
• Approval validity: Record what exact action was approved, how expiry is checked at execution, and what happens when the payload changes.
• Local reservation: Identify the uniqueness constraint and transaction that establish one operation record before the external call.
• Provider behavior: Record the documented key scope, retention, parameter rules, lookup options, and the point at which automatic retry must stop.
• Unknown outcome: Name the state used after a timeout or lost response, the evidence needed to resolve it, and who owns unresolved cases.
• Event deduplication: Identify provider event identity, duplicate-delivery handling, and separate identities for downstream side effects.
• Recovery test: Specify how to interrupt execution after provider acceptance, what restart should load, and the observable pass/fail evidence.
• Operational trace: List the operation, attempt, provider, and state-transition fields an operator can inspect without exposing sensitive payloads.

A design review is complete when each item points to a concrete record, rule, or test—not merely a prompt instruction. If the provider’s behavior is not known, make that an explicit unresolved dependency before enabling automatic retries for the action.

Choose the next implementation step

Start with the single action where duplicate execution would be hardest to undo. Map its operation identity, payload fields, provider evidence, and unresolved state before expanding the pattern to other tools. Then test the crash-after-success boundary and the changed-payload conflict. Those two cases expose whether the system distinguishes a retry from a new intention.

Do not promise exactly-once effects across your database, worker, and provider. Aim for a more defensible operational property: repeated attempts converge on one identifiable business operation when provider semantics permit, conflicts are visible, and no new side effect is issued while the previous outcome remains unknown. Teams that need help designing the durable execution boundary can explore production platform engineering.

WANT TO TALK THROUGH YOUR SITUATION?

Talk it through with us. Thirty minutes, no slides.