What durable execution means
An agent that runs for hours is not simply a model call that takes a long time. It is a workflow that must preserve its progress while waiting on people, external systems, or scheduled work, and then continue after its current process is gone. Durable execution means the workflow’s progress is recorded so it can recover from interruption. The engineering challenge is deciding what to record, when to record it, and which actions are safe to repeat.
A process that keeps its state only in memory is easy to understand while it is running and difficult to trust when it stops unexpectedly. A container may restart after a deployment; a worker may lose its connection while an external service is still processing a request; a person may take a day to respond. If the only record of progress disappears with the process, the system cannot reliably tell what happened or what it should do next.
Persistence alone does not make a workflow safe. Replaying a step can call a service twice, send duplicate instructions, or overwrite a newer decision. The workflow must distinguish durable internal progress from effects in other systems, and it must be designed around uncertainty: after a timeout, the caller may not know whether the other system completed the request. Durable execution therefore depends on explicit state, careful replay boundaries, and operations that can be retried safely. LangGraph’s persistence guidance describes the need to persist progress while treating non-deterministic side effects carefully during replay.
This tutorial builds a framework-neutral design for a hypothetical invoice review workflow. It shows how to define transitions, checkpoint bounded work, suspend and resume at an approval boundary, and test recovery without assuming that an external action happens exactly once. The examples are illustrative, not tested integrations or production-ready code. Adapt the data model and transaction boundaries to the database and services your system actually uses.
Prerequisites and setup
Before implementing the workflow, identify four things: a durable store for workflow records, a worker that can claim and advance work, a way to deliver or poll for new work, and an observable record of each transition. A queue may help dispatch work, but it is not a substitute for the workflow’s authoritative state. The state store must let a worker determine which step is current, whether a lease is active, and whether a previous attempt has already recorded its result.
You also need a clear description of the business action. For the running example, the agent reads an invoice, compares it with an order, prepares a proposed correction, waits for an authorized person to approve that exact correction, and only then asks a downstream system to apply it. The example does not assume a particular invoice platform, approval product, or model provider.
Set up a development environment with a relational database or an equivalent store that supports conditional updates. Prepare a test worker that can be stopped and restarted, and use a fake downstream service that records requests in a table or log. Keep test data separate from live business systems. For initial recovery tests, make the fake service configurable: it should be able to accept a request and then simulate a lost response, reject a request, or return success.
Agree on operational terms before writing code. A run is one instance of the business workflow. A step is a bounded unit with an input, an outcome, and a defined next transition. A checkpoint is the durable record that allows the worker to determine where the run can safely continue. An attempt records one effort to execute a step. These distinctions make incident timelines much easier to interpret than a single mutable blob called agent state.
Step 1: Define states and transitions
Start with the business lifecycle rather than a prompt or an agent loop. For the invoice example, a minimal lifecycle is queued, reviewing, awaiting_approval, applying_change, completed, and needs_attention. Each state needs an entry condition and a permitted way out. awaiting_approval, for example, must not be left by an arbitrary worker deciding that an approval probably arrived. It requires a recorded, valid decision for the pending proposal.
Write transition rules in plain language first. queued may become reviewing when a worker claims the run. reviewing may become awaiting_approval after the proposal and its review inputs are saved. An unambiguous result that needs no human decision may follow a separately defined path, if the business has approved that path. awaiting_approval may become applying_change only when a valid approval matches the pending proposal. applying_change may become completed after the downstream effect is verified. An unresolved timeout or contradictory result moves to needs_attention, not to a falsely successful terminal state.
For each transition, name the record that proves it happened. A state label without supporting details is not enough for recovery. For example, a transition to awaiting_approval should be accompanied by the proposal version, the relevant invoice identifier, a digest or equivalent stable identifier for the proposed change, and the time the request became eligible for a decision. A transition to completed should include evidence that the intended business result was observed, not simply a model-generated statement that the action succeeded.
A useful rule is to keep the workflow’s control state separate from its conversational or reasoning context. The model may help summarize documents or propose a classification, but the application should own the current state, allowed transitions, and side-effect boundary. A more detailed treatment of the surrounding run manifest is available in Build an AI Harness Run Manifest: State, Tools and Stop Conditions. This tutorial focuses on making progress durable and recoverable rather than specifying a complete agent harness.
The flow below shows the central execution path. A worker performs a bounded step, saves its result, and either continues, waits for a decision, or stops for investigation. A resumed run returns to the durable record before it takes further action; it does not assume that the previous process’s memory survived.
flowchart TD
%%{init: {"flowchart": {"htmlLabels": false}}}%%
A["Claim queued run"] --> B["Execute bounded step"]
B --> C["Persist outcome"]
C --> D["Continue or finish"]
C --> E["Wait for valid approval"]
C --> F["Stop for investigation"]
E --> A
accTitle: Durable workflow path
accDescr: A worker claims a run, executes one bounded step, and persists its outcome. The workflow then continues, waits for a valid approval before resuming, or stops for investigation.
The approval path returns to a worker claim only after the application has recorded and checked the decision. The diagram intentionally does not show a direct approval-to-side-effect arrow: the resumed worker must re-read durable state and validate the pending action before execution. A timeout is not itself evidence that a step failed or succeeded, so the investigation path is an important outcome rather than a logging detail.
Step 2: Create a durable record for each run
Choose a record shape that lets a new worker understand the run without relying on local memory. At minimum, store a stable run identifier, current state, state version, current step, relevant business identifiers, timestamps, and a compact reference to the inputs and outputs needed for recovery. Store large documents or generated artifacts separately when appropriate, and refer to them by stable identifiers. Avoid persisting secrets or unnecessary copies of sensitive source material in general-purpose logs.
A relational schema can make the key relationships concrete. The following PostgreSQL-style sketch is illustrative. It omits indexing, retention, tenant isolation, database permissions, and migration details, all of which need decisions in a real system.
CREATE TABLE workflow_run (
run_id TEXT PRIMARY KEY,
business_key TEXT NOT NULL,
state TEXT NOT NULL,
state_version INTEGER NOT NULL DEFAULT 0,
current_step TEXT NOT NULL,
proposal_version INTEGER,
proposal_digest TEXT,
lease_owner TEXT,
lease_expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL,
updated_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE workflow_step (
run_id TEXT NOT NULL REFERENCES workflow_run(run_id),
step_name TEXT NOT NULL,
attempt INTEGER NOT NULL,
status TEXT NOT NULL,
input_ref TEXT,
output_ref TEXT,
started_at TIMESTAMPTZ NOT NULL,
finished_at TIMESTAMPTZ,
error_class TEXT,
PRIMARY KEY (run_id, step_name, attempt)
);
The run row describes the latest control state; step rows preserve attempts and their outcomes. A new attempt should not erase the fact that an earlier attempt timed out. That earlier attempt may be the clue needed to distinguish a failure before a request was sent from an uncertain result after the request left the worker.
Use a conditional state update so two workers cannot silently advance the same version as if each were alone. In database terms, a worker can update a run only when the state and version still match the values it read. If the update affects no row, the worker should reload state and decide whether it lost a race, encountered a changed run, or should stop. Do not respond to a failed conditional update by blindly repeating the whole step.
UPDATE workflow_run
SET state = 'awaiting_approval',
current_step = 'wait_for_approval',
state_version = state_version + 1,
updated_at = CURRENT_TIMESTAMP
WHERE run_id = 'run-example-001'
AND state = 'reviewing'
AND state_version = 4;
An expected result is one updated row. Zero updated rows means the worker’s view was stale or the transition was otherwise not valid; it should fetch the latest record before proceeding. The example identifier and version are placeholders, not a prescribed API or production configuration.
Persist the inputs that determine a step’s meaning. If a later replay reconstructs a request using a different document version, changed configuration, or different business record, it may not be replaying the same work. Record stable references and relevant versions, and make changes to the workflow definition explicit enough that operators can tell which version of the process created a run.
Step 3: Make each step bounded and checkpoint its outcome
A durable step should have a clear start and finish, a defined input, and an outcome that can be saved before another step begins. Avoid one long worker function that reads a file, calls several tools, asks for approval, and updates a system before saving any intermediate progress. If the worker stops in the middle, that design gives recovery little evidence about what has already happened.
For the example, split work into load_invoice, compare_to_order, prepare_proposal, wait_for_approval, apply_approved_change, and verify_result. The model may assist with document interpretation, but each step should return structured information that the application can validate. A comparison result might contain the invoice line, order line, discrepancy amount, source references, and a confidence or review classification selected by your application’s own policy. Do not let unvalidated free-form text decide which business record to mutate.
Use this general execution shape. It is pseudocode rather than a runnable library example; the storage and worker functions must be implemented for your environment.
async def advance(run_id, store, tools):
run = await store.load_run(run_id)
assert_transition_is_allowed(run.state, run.current_step)
attempt = await store.start_attempt(
run_id=run_id,
step=run.current_step,
expected_version=run.state_version,
)
result = await execute_one_step(run, tools)
validated = validate_step_result(run.current_step, result)
await store.commit_step_and_transition(
run_id=run_id,
expected_version=run.state_version,
attempt_id=attempt.id,
result=validated,
next_state=choose_next_state(run, validated),
)
The commit operation should record the step result and advance the run together when they belong to one database transaction. If separate storage systems are involved, define how a partially recorded transition is detected and repaired. The important property is not the particular function name: after a restart, the worker can identify the last committed step and distinguish it from an incomplete attempt.
Checkpointing is not the same as saving every token or every internal thought. Persist business-relevant inputs, outputs, decision references, and execution status. Keep transient reasoning out of the authoritative record unless a deliberate, reviewed need exists to retain it. A concise, structured record makes recovery and investigation more useful than a large, unversioned conversation transcript.
Step 4: Put external side effects behind a safe boundary
An external action is different from a database checkpoint. A local transaction can often commit a result and state change together; a remote service may receive a request even if the worker never receives its response. That uncertainty is the reason not to claim exactly-once external effects. Your application needs a repeat-safe request design or a reconciliation path that can determine what happened before trying again.
For the invoice workflow, do not call the downstream update while the run is still preparing a proposal. First persist the exact proposed change and its stable identity. If a person must approve it, collect the approval against that identity. Only after validation should a worker create an execution record and submit the corresponding action. The approval should be invalid if the proposal changes: an approval for one amount or destination must not silently authorize a later, altered payload.
Where the receiving system supports a deduplication key, derive a stable key from the business action and proposal version, and reuse it when retrying that same action. Do not assume that a header name or behavior is universal; check the actual receiving service’s contract. If the receiver provides no reliable deduplication mechanism, make retries conditional on a read-back or reconciliation process, and route ambiguous outcomes to a person rather than resending automatically.
A useful record for an outbound action includes the run identifier, proposal version, payload reference, request identity, submission time, attempt number, and outcome classification. Distinguish not_submitted, submitted_unknown, confirmed_applied, and confirmed_not_applied. A network timeout after submission belongs in submitted_unknown until the system obtains evidence to resolve it. Treating unknown as failed and immediately resending is a common way to create duplicate business effects.
If the application uses an outbox pattern, write the intent to send alongside the workflow transition in one local transaction, then have a separate sender deliver it. The sender can mark delivery attempts and reconcile responses. An outbox narrows a local consistency gap; it does not make the remote system’s action exactly once. For a deeper implementation discussion of repeat-safe operations and retries, see Idempotent Tools for Agents: Surviving Retries and Restarts.
Step 5: Pause for approval and resume from validated state
A pause should release the worker rather than hold a process open while waiting. Persist the state needed to resume, record what decision is pending, and make the run eligible for later dispatch when a decision arrives. A resumable interrupt is useful only when the application retains control of what the response is allowed to authorize.
For an approval, bind the decision to a specific run, proposal version, and payload digest or equivalent immutable identity. Record who or what submitted the decision, when it arrived, and whether it passed the application’s checks. Give the pending approval an expiry. An expired approval must not be accepted as current merely because it eventually appears in an event stream or a delayed callback. LangGraph’s interrupt guidance emphasizes that the application must enforce the scope and expiry of an approval when pausing and resuming a workflow.
On resumption, reload the run from durable storage, validate the approval against the still-current proposal, and check that the run is in the expected state. Then commit the transition that makes the action eligible for execution. This sequence protects against a stale callback, a duplicate decision, an approval for the wrong invoice, or a proposal that was edited after review.
Do not treat a button click, callback, or message as sufficient proof by itself. It is an input to validate. If the decision does not match the pending proposal, record the mismatch and stop the run for investigation. If the approval has expired, return the run to the appropriate review path rather than extending its authority implicitly. The precise review process depends on the business; the durable workflow should make its state and reason visible.
Step 6: Recover from timeout, restart, and duplicate delivery
A worker restart is usually straightforward when each completed step is committed: claim the run, reload state, and continue from the next permitted step. A timeout around a side effect is harder. The worker may have submitted a request, the receiver may have applied it, and the response may have been lost. Recovery must reason from recorded evidence, not from the worker’s assumption about what probably happened.
Use distinct handling for failure classes. A local validation error is generally a deterministic problem to fix or escalate, not a transient retry. A temporary connection failure before submission may permit a retry under a bounded policy. A timeout after submission is an uncertain outcome and needs reconciliation. A malformed or contradictory response should stop automatic advancement. A process crash after the database commit but before acknowledging a queued message should be safe if redelivery reloads state and sees that the step is already committed.
Set limits on attempts and elapsed time for each step. These are operational controls, not proof that an operation failed. When a limit is reached, record the cause and route the run to an explicit outcome such as needs_attention. Avoid an unbounded retry loop that repeatedly spends compute while the underlying problem remains unchanged. Keep the original error class and relevant request identity so a person can determine whether the next action is a repair, a reconciliation, or a deliberately authorized retry.
Worked recovery trace: invoice correction
Assume, for illustration, that an invoice review has found a mismatch between an invoice line and a purchase order. The agent prepares a proposed correction; an authorized reviewer approves it; a worker submits the change to a fictional downstream ledger. The amounts, identifiers, and timeline below are invented to demonstrate the recovery method, not to describe a real customer or system.
At 09:00, run inv-001 reaches awaiting_approval. The durable record contains proposal version 3, a reference to the invoice and order, and digest proposal-3. At 09:20, the reviewer approves proposal version 3. The application validates the run, proposal identity, and expiry, then records the decision and advances the run. The action record uses the same stable identity for any retry of this proposal.
At 09:21, the worker sends the request. The fictional receiver applies the change, but the response does not reach the worker before its timeout. The worker records submitted_unknown; it does not mark the run complete, and it does not send a second request with a new identity. The worker process then restarts. On recovery, a new worker loads the run and sees that execution is uncertain rather than unstarted.
The recovery path queries the receiver for the action identity or checks the resulting business record, depending on the receiver’s actual capabilities. In this hypothetical trace, a read-back confirms that the exact approved change is present. The application records the observed result and advances to completed. If that read-back had been inconclusive, the correct outcome would have been needs_attention, not an automatic second mutation.
The key design choice is to separate a tool-call attempt from the business result. A successful network response is useful evidence, but a missing response does not prove failure; a model’s summary is not verification. The application should define what observable condition demonstrates that the intended result occurred and keep the run open until that condition is met.
Step 7: Test recovery before trusting the workflow
Test the workflow by injecting failures at boundaries, not only by checking that a successful run reaches its final state. Use a fake downstream service and a disposable database so you can terminate the worker at controlled points. Capture the run record, step attempts, action identity, and resulting business record after each test. These are proposed operational tests; no test execution is implied here.
Start with a crash after a step result is committed but before the worker acknowledges its message. Pass if the redelivered work reloads the run and does not redo the committed step. Fail if it creates a second effect or advances from stale memory. Next, simulate a crash before the step result is committed. Pass if the run remains recoverable and the attempt is visible as incomplete. Fail if the application reports completion without durable evidence.
Then simulate the ambiguous timeout: the fake receiver applies the change and drops the response. Pass if the run records an uncertain outcome, avoids issuing a new action identity, and enters reconciliation. Fail if it marks the action failed solely because the response was lost, sends an unguarded duplicate, or declares success without checking the business result. Repeat with a receiver that rejects the action to confirm that the recovery path can distinguish a confirmed rejection from an unknown outcome.
Finally, test approvals. Try a valid decision for the current proposal, a duplicate decision, an approval for an older proposal version, an expired decision, and a decision for another run. Pass only when the current valid decision can resume the intended run and the other inputs cannot authorize the side effect. Check the resulting records as well as the user-facing state. A useful test is not merely that the worker returns an error; it verifies that no action was submitted.
Failure analysis and troubleshooting
The run repeats a step after restart. Check whether the step result and transition were committed before the queue message was acknowledged. Inspect state versions and step-attempt records. If the step has an external effect, determine whether the receiving service offers a safe way to recognize the same action. Repair the transition logic before increasing retry limits; a higher retry count can magnify duplicate effects.
A run stays in reviewing with no active worker. Inspect the lease and its expiry, then determine whether the worker crashed or another worker owns the run. A stale lease can be reclaimed according to an explicit rule, but reclaiming it must not bypass state-version checks. If the underlying step might have submitted an external request, mark its result uncertain and reconcile before continuing.
An approval arrives, but the run will not resume. Compare the decision’s run identifier, proposal version, payload identity, and expiry with the current durable record. A mismatch should remain visible as a rejected or unresolved event. Do not edit the stored approval to make it fit the current proposal. If the proposal changed, request a new decision through the application’s defined process.
The run says completed, but the business record is unchanged. Find the evidence used to make the terminal transition. If completion was based only on a successful tool return or generated summary, strengthen the verification step to inspect the resulting business state. Reopen or correct the run according to an explicit repair procedure, and preserve the earlier event history so the discrepancy can be understood.
The database shows duplicate attempts but the action is unclear. Compare request identities, timestamps, receiver responses, and read-back results. If the receiver cannot distinguish the attempts, do not assume that the latest attempt won or that duplicates were harmless. Put the run into a human-reviewed reconciliation state and improve the action boundary before enabling more automatic retries.
A practical observability design makes these investigations possible. Record a correlation identifier across the run, step, and outbound action; capture state transitions and attempt outcomes; and alert on growing age in waiting states, repeated uncertain results, and runs that exceed their defined limits. Keep logs useful without copying full documents or sensitive payloads indiscriminately. The related AI Agents in Production: Agent Observability article explores observability as a distinct implementation topic; here, the focus is the evidence needed to recover a specific workflow.
Printable implementation worksheet
Use this worksheet in a design review or as a first-pass specification. Fill in the answers for one business workflow before extending the pattern to more tools or additional agent behavior. If an answer is unknown, record that uncertainty as a design decision rather than relying on an implicit assumption.
- Run identity: What stable identifier distinguishes one business workflow from another? Which business record does it refer to, and how will you prevent accidental cross-run reuse?
- State model: What are the permitted states, and which transition moves each run forward? Which states are terminal, and which require a person or reconciliation before continuing?
- Checkpoint boundary: What exact result must be stored before the next step can run? Can a replacement worker recover from the durable record alone?
- Attempt history: Where are started, completed, failed, and uncertain attempts recorded? Can you distinguish a crash before submission from a lost response after submission?
- Side-effect identity: What identifies one intended external action? How will a retry reuse that identity, or how will an ambiguous result be reconciled if the receiver cannot deduplicate it?
- Approval scope: Which exact proposal does the decision authorize? How are proposal changes, duplicate decisions, wrong-run decisions, and expired decisions rejected?
- Verification: What observable business condition proves that the intended result occurred? What state should the workflow enter if that evidence is missing or contradictory?
- Recovery limits: Which failures may be retried, how many attempts are allowed, and what moves a run to human investigation? Who can see the reason for that transition?
- Failure tests: Have you tested restart after checkpoint, duplicate delivery, approval mismatch, receiver rejection, and timeout after a remote effect may have occurred?
- Operational signals: Can an operator find the run, its current state, last successful checkpoint, latest attempt, and unresolved action without reconstructing the story from scattered logs?
A design review is ready to move into implementation when the team can explain what happens after a crash at every side-effect boundary. It does not need to predict every incident in advance, but it should make unknown outcomes explicit and provide a safe route to resolve them. For teams that need help translating these boundaries into a production implementation, production platform engineering is an appropriate next step.
Put recovery behavior ahead of longer runtime
The practical goal is not to keep one process alive for hours. It is to make the workflow’s progress durable enough that any eligible worker can continue from a known state, while preventing uncertainty from being mistaken for success or failure. Start with one business path, explicit transitions, bounded steps, and a testable recovery route. Add more automation only after the application can explain what happened when a worker stops at the least convenient moment.
A useful counterexample is a long-running loop that saves a transcript every few minutes but performs a business update before recording its intent. It may appear persistent because some context survives, yet a crash around the update still leaves the system unable to tell whether retrying is safe. Durable execution is about the boundary between recorded state and real-world effects, not merely about retaining conversation history. The separate guide Why Long-Running Agents Lose Their Place—and How to Resume Them covers the distinct question of preserving context across sessions, while Agent Harness vs Agent Framework vs Runtime: A Practical Guide helps clarify which layer owns execution responsibilities.