Durable Execution for AI Agents: Checkpoints, Retries and Idempotency

A demo agent runs for 30 seconds. A production agent runs for 20 minutes, makes 40 tool calls, and gets interrupted by a deploy, a rate limit, an OOM kill or a provider timeout somewhere in the middle. What happens next decides whether your agent is a toy or a system.

The usual answer is "retry the whole run." That is wrong in two ways: you pay for every LLM call again, and you re-execute side effects. The agent that already created the Jira ticket, charged the card or sent the email does it a second time.

This post covers the three ideas that fix this: checkpointing, retry classification and idempotency keys. It ends with a small, dependency-free implementation you can adapt.

Why agents are harder than normal jobs

Ordinary background jobs are usually short and mostly deterministic. Agent runs differ in four ways:

Long and multi-step. The chance that something fails grows with every step.
Non-deterministic. Re-running the same prompt can produce a different plan. You cannot "replay from the top" and expect the same tool calls.
Side-effectful. Tools write to the outside world, and some writes cannot be undone.
Expensive. Each replayed LLM call costs real tokens and latency.

The core principle: record the result of every completed step, and on resume, return the recorded result instead of executing again. This is the idea behind durable execution engines such as Temporal, AWS Step Functions and Restate, and behind checkpointers in agent frameworks like LangGraph. You can use those, or build the minimal version yourself to understand what they do.

The three building blocks

1. Checkpoints

After each step (an LLM call, a tool call), persist its input and output keyed by a stable step ID. On restart, the runner walks the same code path; for each step it first checks the store. If a result exists, it returns that without executing.

This works because of one rule: the orchestration code must be deterministic; only the steps may be non-deterministic. The LLM's non-determinism is fine because its output is recorded once and then replayed from the store.

2. Retry classification

Not all failures deserve a retry. Sort them first:

Failure	Example	Action
Transient	429, 503, network timeout	Retry with exponential backoff and jitter
Model-recoverable	Tool returned validation error, malformed JSON	Feed the error back to the model, let it correct
Permanent	401/403, 404 on required resource, schema mismatch	Fail the step, do not retry, surface to a human
Ambiguous	Timeout after the request was sent	Retry only if the call is idempotent

The last row is the dangerous one. A timeout on a "create order" call does not tell you whether the order was created.

3. Idempotency keys

An idempotency key is a unique token sent with a write. The receiving system stores the key with the result; a second request with the same key returns the original result instead of repeating the action. Many payment and messaging APIs support this natively (check the provider's docs for the exact header name and retention window). For your own tools, you implement it yourself.

Derive the key from the run and the step, not from a random value at call time:

idempotency_key = f"{run_id}:{step_id}"

If the key were random per attempt, a retry would look like a brand new request.

A minimal implementation

This uses SQLite from the standard library. The Run object memoizes steps and retries transient failures.

import json
import random
import sqlite3
import time
from typing import Any, Callable


class TransientError(Exception):
    """Safe to retry (429, 503, timeouts on idempotent calls)."""


class PermanentError(Exception):
    """Do not retry."""


class Run:
    def __init__(self, run_id: str, db_path: str = "runs.db"):
        self.run_id = run_id
        self.db = sqlite3.connect(db_path)
        self.db.execute(
            """CREATE TABLE IF NOT EXISTS steps (
                run_id TEXT, step_id TEXT, result TEXT,
                PRIMARY KEY (run_id, step_id))"""
        )

    def step(
        self,
        step_id: str,
        fn: Callable[[str], Any],
        max_attempts: int = 4,
        base_delay: float = 1.0,
    ) -> Any:
        row = self.db.execute(
            "SELECT result FROM steps WHERE run_id=? AND step_id=?",
            (self.run_id, step_id),
        ).fetchone()
        if row:  # already completed in a previous attempt: replay
            return json.loads(row[0])

        idem_key = f"{self.run_id}:{step_id}"
        for attempt in range(1, max_attempts + 1):
            try:
                result = fn(idem_key)
                break
            except TransientError:
                if attempt == max_attempts:
                    raise
                delay = base_delay * 2 ** (attempt - 1)
                time.sleep(delay + random.uniform(0, delay / 2))  # jitter

        self.db.execute(
            "INSERT INTO steps VALUES (?, ?, ?)",
            (self.run_id, step_id, json.dumps(result)),
        )
        self.db.commit()
        return result

Using it in an agent loop:

def agent(run: Run, task: str):
    plan = run.step("plan", lambda key: call_llm_plan(task))

    results = []
    for i, action in enumerate(plan["actions"]):
        # step IDs come from position/identity in the recorded plan,
        # so they are stable across restarts
        out = run.step(
            f"action-{i}-{action['tool']}",
            lambda key, a=action: call_tool(a, idempotency_key=key),
        )
        results.append(out)

    return run.step("summary", lambda key: call_llm_summarize(results))

If the process dies after action 7, calling agent(Run("run-123"), task) again replays plan and actions 0–7 from SQLite for free, then continues at action 8. The plan is not regenerated, which matters: a fresh LLM call could produce a different plan and break step alignment.

There is a known gap in this minimal version. If the process crashes after fn succeeds but before the INSERT, the step re-executes on resume. That is exactly why the idempotency key is passed to the tool: the second execution is de-duplicated by the downstream system. Checkpoints reduce repeated work; idempotency keys make the remaining repeats safe. You need both.

Handling tools that are not idempotent

Some APIs give you no idempotency support. Options, in order of preference:

Wrap with your own dedupe table. Before calling, insert (idem_key, "started"). After, mark "done" with the result. On resume, a started row without done means the outcome is unknown.
Read before write. Query for the resource ("is there already a ticket titled X for run Y?") before creating it. Tag created resources with the run ID so this is possible.
Escalate on ambiguity. If the outcome is unknown and the action is irreversible, stop and ask a human instead of guessing.

Treat side-effect tools as a separate class in your design: mark each tool read_only, idempotent_write or unsafe_write, and let the runner apply different retry policy to each.

Choosing a durability approach

Approach	Good for	Tradeoff
Hand-rolled step store (above)	Small teams, few tools, learning	You own edge cases: concurrency, cleanup, versioning
Framework checkpointers (e.g. LangGraph)	Graph-shaped agents, human-in-the-loop pauses	Coupled to the framework's state model
Workflow engines (Temporal, Restate, Step Functions)	Runs lasting hours or days, strict guarantees	Operational overhead, new programming model
Queue with retries only	Single-step, idempotent tasks	No memory of partial progress

Verify current features and guarantees in each product's documentation; they change quickly.

Common pitfalls

Non-deterministic step IDs. Using timestamps or random UUIDs as step IDs means replay never finds prior results. Derive IDs from position and content in the recorded plan.
Checkpointing only the final answer. The expensive part is the middle. Persist every step.
Retrying inside the model loop and the HTTP client and the queue. Three layers of 3 retries is 27 attempts. Pick one layer per failure type.
No jitter. Synchronized retries after a provider blip create a thundering herd and trigger more 429s.
Ignoring code versions. If you deploy new agent code mid-run, replayed steps may no longer line up. Store a version with each run and let old runs finish on old code, or fail them explicitly.
Storing secrets in checkpoints. Step results are persisted; redact credentials and sensitive PII before writing.
Unbounded runs. Add a max step count and a wall-clock budget. Durable does not mean infinite.

Checklist before you ship

Every LLM and tool call goes through a memoizing step function
Step IDs are deterministic and stable across restarts
Every write tool accepts an idempotency key, or is wrapped with a dedupe table
Failures are classified: transient, model-recoverable, permanent, ambiguous
Backoff has jitter and a maximum attempt count
Runs have step and time budgets
You have a test that kills the process mid-run and asserts no duplicate side effects

That last test is the most valuable. Inject a crash after each step in turn, resume, and compare external side effects against an uninterrupted run. It finds ordering bugs that code review will not.

Next steps

Start small: wrap your existing agent's tool calls in a step() function, add run IDs, and write the kill-and-resume test. Then decide whether the hand-rolled store is enough or whether a workflow engine is justified by your run length and reliability needs. Pair this with tracing so you can see which steps replayed and which executed.

If you want to build agents that survive production, with hands-on labs on reliability, evals and observability, take a look at the Joinloop AI engineering cohort.