Durable Execution for AI Agents: Checkpointing Workflows That Run for Days
Learn how durable execution and checkpointing keep long-running AI agents resumable, replayable, and cost-safe across failures, restarts, and multi-day runs.
The Step-47 Problem
A research agent has been running for nine hours. It has crawled 40 sources, synthesized three interim briefs, and is waiting on a human approval before publishing. Then a Kubernetes node gets drained during a routine deploy rollover. The process dies at step 47 of 52. When the worker comes back, the agent starts over — re-crawling sources, re-running expensive LLM calls, and this time producing slightly different intermediate results because the model sampled differently.
This is the defining operational problem of long-horizon AI agents. Request/response agents finish in seconds and fail cleanly. Agents that run for hours or days fail partially — and partial failure is where cost, correctness, and trust all break down.
Durable execution is the discipline that fixes this. It turns agent workflows into resumable, replayable state machines, and checkpointing is the mechanism that makes it work. This guide covers the failure modes, the checkpointing model, schema design, idempotency, fault-tolerance patterns, and a decision checklist for when durable execution is — and isn't — worth the complexity.
Why Long-Running AI Agents Break in Production
Short-lived agents live inside a request. If the call fails, the caller retries and nothing is lost. Long-horizon agents are different: they hold state across tool loops, human-in-the-loop pauses, scheduled waits ("check this inbox in six hours"), and external callbacks from webhooks or batch jobs. Their lifetime is measured in hours, and their execution surface spans multiple processes, providers, and networks.
The moment an agent outlives a single process, every assumption about in-memory state becomes a liability.
The Failure Modes Nobody Budgets For
- Process and container restarts. Deploys, OOM kills, node drains, spot-instance reclaims. Routine infrastructure events become data-loss events.
- LLM provider timeouts and rate limits. A 429 mid-loop doesn't just delay the agent; without durable state, it can force a full restart.
- Tool API failures. Third-party APIs fail at the worst moments — after the expensive reasoning step, before the write.
- Non-deterministic drift on retry. Re-running completed LLM steps produces different outputs. The agent may take a divergent path, and any downstream side effects already committed are now inconsistent.
- Cost blowups from re-execution. Each restart pays for the entire completed prefix again. A run that fails three times on a 46-step prefix costs roughly four times that prefix — and the multiplier is unbounded if the failure is intermittent.
- Lost human approvals. An approval granted at 2 p.m. vanishes with the process, and the human is asked again — eroding trust in the system.
Retries alone don't solve this. Retrying from step zero is both expensive and semantically wrong: it can produce a different result than the run that was already half-committed to the world.
What Durable Execution Actually Means
Durable execution is an execution model in which workflow progress is persisted as an event history. On failure, the workflow resumes from its last durable point rather than restarting. The code reads as ordinary sequential logic, but the runtime journals every step so it can reconstruct exactly where execution stopped.
The core primitives:
- Durable step boundaries. Each unit of work (an LLM call, a tool invocation, a write) is wrapped so its inputs and outputs are persisted before proceeding.
- Replay of recorded results. On recovery, completed steps are not re-executed; their recorded results are replayed into the function. This is the crucial distinction: the results are deterministic because they are read from the log, not recomputed. What must remain deterministic is your workflow code's control flow given those results (see "Determinism Constraints" below).
- Persisted state. Variables, loop counters, and branch decisions survive process death.
- Durable timers. A
sleep(6 hours)survives restarts — it's a scheduled wake-up, not a blocked thread. - External signals. Human approvals, webhooks, and callbacks are delivered as durable events that can arrive hours or days later.
Durable Execution vs. Workflow Orchestration vs. Retries
These are often conflated. They solve different problems.
| Approach | What it guarantees | Where it falls short for agents |
|---|---|---|
| Naive retries | The call is attempted again | Restarts from step zero; non-deterministic drift; cost blowups |
| Workflow orchestration (cron/DAG tools) | Tasks are scheduled and dependencies respected | Schedules tasks but doesn't guarantee step-level completion semantics; state lives outside the code |
| Durable execution (event-sourced, replay-based) | Step-level completion, replay of recorded results, durable timers and signals | Requires discipline around determinism, idempotency, and history size |
Orchestration tools schedule; durable execution guarantees. The pattern is well-established in payments and infrastructure and is now being applied directly to agent frameworks. If your payment processor can survive a mid-transaction crash, your agent should survive a mid-loop one.
The engines are not interchangeable, and the constraints matter specifically for multi-day agents:
- Temporal — event-history model; history size is bounded, so long loops require continue-as-new to start a fresh history while carrying state forward. This is a real design constraint for agents that iterate thousands of times.
- AWS Step Functions — state-machine DSL with a maximum execution duration (one year) and a bounded state payload (256 KB). Good for bounded, orchestrated flows; awkward for tight agent loops.
- Azure Durable Functions — orchestrator replay model with explicit determinism rules for orchestrator code; side effects must live in activity functions.
- Restate — a journal-per-service model rather than a single workflow history; different tradeoffs around where state lives.
- Inngest — event-driven step functions with per-step retries; step granularity is the unit of durability.
Pick based on your loop shape and your execution-duration and payload limits, not on a feature checklist.
The mental model: a durable agent workflow is an append-only log of what happened, plus code that can be replayed against that log to reconstruct where it is. The log is the source of truth; memory is a cache.
The Checkpointing Model: State, Events, and Replay
Checkpointing in a durable system is event-sourced: the runtime appends an immutable record of each step's inputs and outputs to a history. Current state is never stored as a mutable blob — it is derived by replaying the history through the workflow code.
This has a profound consequence: you can recover from any point, audit any run, and reproduce any result — as long as the workflow code's control flow is deterministic given the recorded results, and as long as every side effect is idempotent.
Checkpoints come in three granularities:
- Step-level (coarse). One checkpoint per logical step — an LLM call, a tool call, a file write. This is the workhorse granularity and the right default for most agents.
- Decision-level (medium). Checkpoint the tool call and the decision that selected it, so a resumed run doesn't re-derive the plan. Useful when planning is expensive and you want the plan pinned.
- Token-level (fine). Streaming checkpoints within a single LLM call. Almost never worth it: it multiplies history size, complicates replay, and the unit of useful recovery is the completed call, not a partial token stream. Reserve it for very long generations where a mid-call crash is both likely and costly.
Determinism Constraints
Replay works only if the workflow code, given the same recorded results, takes the same path. That imposes hard rules:
- No wall-clock, randomness, or environment reads in workflow code.
now(),random(), UUIDs, and config lookups belong inside a step whose result is recorded — otherwise a replay can branch differently than the original run. - No direct side effects in workflow code. HTTP calls, DB writes, and file operations go inside steps. If they run in the workflow body, they execute again on every replay.
- Version your workflow. Inserting, removing, or reordering steps changes the history shape and breaks replay of in-flight runs. Use the engine's versioning/patching mechanism, or drain in-flight runs before a breaking deploy. This is the most common cause of "worked in staging, corrupted in prod."
- Pin model and prompt versions in the checkpoint. Record the model id, prompt hash, and tool schema version with each step so a replay can prove it is replaying the same logic, not a mutated one.
What to Persist Per Checkpoint
A checkpoint record should carry enough to replay and to audit:
step_idand a monotonic sequence number- input hash (not necessarily the full input, if large) and the full output
- model id, prompt hash, and tool/schema version
- token counts and latency (for cost attribution and SLOs)
- an idempotency key for any step that writes to an external system
- timestamp and the worker/attempt id that produced it
What not to persist: raw secrets and credentials (store references), and unbounded blobs (store a pointer plus a hash). And plan for checkpoint schema evolution — records written by yesterday's code must still replay under today's reader, or you need a migration path.
Idempotency: The Part That Actually Bites
The crash window that matters is after a side effect commits and before the checkpoint records it. On resume, the step is retried and the side effect happens again. Exactly-once across an external API is not achievable; effective-once via idempotency is.
- Idempotency keys on every write. Derive the key from
workflow_id + step_id(or a content hash), and have the downstream API honor it. Stripe-style idempotency keys, dedup tables, or conditional writes all work. - Write-ahead intent. Record "about to call tool X with key K" before the call, then "called, result R" after. On resume, an intent without a result means check the downstream state (did the write land?) rather than blindly retrying.
- Make reads safe to repeat. Re-crawling a source is wasteful but harmless; re-sending an email is not. Classify every step as read (safe to retry) or write (needs a key).
Human-in-the-Loop and Long Waits
Approvals and scheduled waits are the reason agents run for days, and they are where naive systems lose the most trust.
- Approvals are durable signals. The approval event is appended to the history and survives the process. The human is asked once. Record who approved, when, and against which version of the artifact.
- Timers are persisted wake-ups, not threads. A
sleep(6 hours)consumes no worker and survives restarts — but it does consume history, so avoid tight timer loops; batch or continue-as-new instead. - Timeouts are first-class. Every human wait needs a durable timeout branch, or a stuck approval stalls the workflow forever.
Operational Failure Modes at Scale
- Poison checkpoints. A step that deterministically fails on resume (bad recorded input, a now-invalid tool schema) will retry forever. Cap retries per step and route to a dead-letter state that a human can inspect and repair.
- Stuck workflows. A workflow whose worker died and whose timer never fired looks alive in the dashboard. Alert on no history progress over a threshold, not just on process health.
- History explosion. Long loops grow history until it hits engine limits. Continue-as-new (or the equivalent) is mandatory for unbounded agent loops.
When Durable Execution Is — and Isn't — Worth It
Worth it:
- Runs measured in minutes-to-days, or that cross a human approval or scheduled wait
- Workflows with expensive steps (long LLM calls, costly crawls) where a restart is a real cost
- Anything that writes to external systems and must not double-write
- Compliance or audit requirements that need a reproducible run history
Not worth it:
- Request/response agents that finish in seconds — a plain retry loop is simpler and sufficient
- Workflows with no external side effects and cheap steps, where a restart is cheap and harmless
- Teams not prepared to enforce determinism and idempotency discipline — a durable engine used carelessly produces more subtle bugs than a simple retry loop
The decision rule: if a crash at step 47 would cost real money, produce an inconsistent world, or break a human's trust, you need durable execution. If none of those are true, you don't.
The Checklist
Before adopting durable execution for an agent, confirm:
- Every step is wrapped as a durable boundary, with side effects inside steps only.
- Workflow code is deterministic — no clock, randomness, or direct I/O in the body.
- Every write step has an idempotency key honored downstream.
- Model, prompt, and tool versions are recorded per checkpoint.
- A versioning/patching strategy exists for in-flight runs across deploys.
- Long loops use continue-as-new or equivalent to bound history.
- Every human wait has a durable timeout and a dead-letter path.
- You alert on stalled history progress, not just process health.
Get these right and the step-47 crash becomes a non-event: the worker comes back, replays the log, and resumes at step 48 — same inputs, same decisions, no re-crawl, no double-write, no second approval request.