Guides  /  Streaming and flow control

What happens to queued agent work when traffic spikes?

Answer

Work accumulates for as long as arrivals exceed completions. The wait an arriving request sees is decided by how fast capacity grows and by what holds intake while it grows.

Akka adds nodes to the cluster as load arrives and places the new work across them. Backpressure holds intake to the rate the system sustains during the gap, and scale-to-zero releases the added capacity once the spike passes.

The mechanism

A backlog grows for as long as arrivals exceed completions.

Arrivals above the completion rate leave a difference, and that difference accumulates for as long as the gap is open. Every waiting item is behind the items ahead of it, so the wait grows faster than the backlog does.

Agent work drains slowly for reasons a request-serving system does not have. A single run holds a model call, a tool call, and often a chain of both, so it occupies its slot for seconds or minutes. A backlog of runs clears at that pace.

A fixed pool of capacity turns the spike into a decision an operator makes under pressure. Provisioning for the peak removes that decision and pays for peak capacity in every hour that is not the peak.

What the runtime has to do

Absorbing a spike requires capacity and a hold on intake.

Add capacity as load arrivesNodes join the cluster and take a share of the work, with no operator paged and no ticket raised.
Place work on the nodes it addedSharding assigns arriving work across the cluster, so capacity is used as soon as it is available.
Hold intake to the sustainable rateBackpressure keeps producers at the rate the system completes work, so what is waiting stays bounded and nothing is silently dropped.
Release the capacity afterwardsScale-to-zero returns the added nodes when the spike passes, so peak capacity is paid for during the peak.
What this changes

Peak capacity stops being a standing cost.

Akka Automated Operations delivers elastic scaling to 10 million agentic transactions per second with scale-to-zero, on shared compute that carries the agents, memory, orchestration and endpoints together. A spike raises the cores in use for the length of the spike.

Verizon raised order processing capacity by 750% and cut response times from 6 seconds to 2.4 seconds on Akka.

Questions

Related questions.

Do we still need a queue in front of the agents?

A queue holds work that has not started and gives an operator somewhere to watch the backlog. A queue adds no capacity, so what drains the backlog is the capacity the runtime added.

What happens to a run already in progress when the cluster grows?

Runs in progress continue where they are, and arriving work goes to the added nodes. A run that is moved between nodes resumes from its last completed step.

How fast does new capacity arrive?

New nodes take time to start and join, and backpressure holds intake during that window. The wait is bounded and visible for the length of the gap.

What about a spike no amount of capacity clears?

The system is held to the rate it sustains and the excess waits inside a bound somebody chose. The caller receives a defined wait or a refusal carrying a reason.

What do idle agents cost?

Scale-to-zero means idle capacity is not provisioned. Akka bills on platform tier and service cores, so a system with no traffic is not paying a per-service minimum.

On Akka. Elastic scaling, scale-to-zero and clustering are capabilities of Akka Automated Operations on the Akka SDK runtime. Customer figures. Verizon results are reported by the customer.