Guides  /  Durability and failure

What happens when an agent fails mid-task?

Answer

The answer depends on where the agent's progress is stored. If progress is held in the memory of the process running the agent, that progress is lost when the process dies, and the task restarts from the beginning.

If progress is written to a durable journal as each step completes, the agent resumes from the last completed step, on a different machine if necessary, and the steps that already ran are not repeated.

The mechanism

The location of the state decides the answer.

An agent doing real work is a multi-step process. The agent calls a model, calls a tool, writes a result, calls another model. The process holding those steps can stop at any point: a rescheduled pod, a drained node, a rolling deployment, a dependency that times out.

Most agent frameworks keep the run's progress in the memory of the process executing it. Some add checkpointing, where progress is serialized to an external store between steps. Checkpointing works, and it puts the store in your hands: you operate it, size it, and keep it available. The recovery path is as good as the checkpoint frequency you configured.

Durability can instead be a property of the runtime executing the agent. Every state change is appended to a journal as it happens, so a failure loses no progress.

STATE IN PROCESS MEMORY process dies 123 back to step 1, every completed step runs again and its side effects happen twice STATE IN A DURABLE JOURNAL process dies 123 45 NEW MACHINE forward from step 4 steps 1–3 are not repeated
What it costs

Restarting is not free.

A task that restarts from the beginning pays for its completed steps twice. The model calls already made are billed again. The tools already invoked run again, and any side effect they caused happens a second time: a payment submitted, a ticket opened, a record written.

The cost of repeating a side effect decides whether an agentic system can act on anything that matters. A duplicated payment is an incident. Teams that hit this build idempotency keys, deduplication tables, and reconciliation jobs around the agent, and then maintain them.

The requirement

Storing state is not enough to resume a run.

Durable stateEvery state change is committed before the next step begins, so no completed work exists only in memory.
An event journalThe sequence of what happened is replayable, so a new process can reconstruct exactly where the previous one stopped.
Recorded side effectsThe runtime records that a tool call completed, so a resumed run knows to skip it.
ReschedulingSomething has to notice the failure and place the work on a healthy machine without a human in the loop.
Held executionsA run paused for a human decision survives crashes and deployments for as long as the decision takes.

The Akka SDK provides these as runtime properties. State is durable by default and replayable from its event journal, so an agent that fails mid-task resumes from where it stopped. The same runtime handles clustering, failover and rescheduling.

Questions

Related questions.

Does checkpointing solve this?

Checkpointing solves part of it. Persisting progress between steps means a failure loses at most the work since the last checkpoint. You still operate the checkpoint store, choose the checkpoint frequency, and handle the window between checkpoints. The recovery guarantee is the one you configured.

What happens to a tool call that was in flight when the process died?

The runtime has to record the outcome of the call. Without that record, a resumed run cannot tell a call that never happened from one that succeeded before the failure, and it has to guess whether to repeat the action.

How long can an agent stay paused?

For as long as the thing it is waiting for takes. A run held for a human approval should survive crashes, deployments, and days without polling, replay, or lost state.

Does this matter for a prototype?

A prototype rarely meets it. Prototypes run short tasks, on one machine, with side effects that are safe to repeat. The behaviour starts to matter when tasks get longer, traffic runs concurrently, and the actions touch systems of record.

On Akka. Durable state, event journals, and held executions are described in the Akka SDK documentation.