Guides  /  Streaming and flow control

What happens when a model provider rate-limits you mid-run?

Answer

The provider refuses the call, and the work behind that call is still outstanding. What happens next is decided by whether the refusal is handled by the code that made the call or by the runtime scheduling the work.

Akka treats the refusal as capacity information and applies it to every caller that shares the limit. The run keeps its state while it waits, and the steps that already completed are not run again.

The mechanism

A rate limit belongs to the account, and every caller shares it.

A provider refuses a call when the account has passed its requests or its tokens per minute. The refusal names the account. Every agent, sub-agent and tool call drawing on that key is counted against the same limit.

Retry logic written around a single call spaces that call's attempts. Ten callers backing off on their own clocks present the provider with ten independent attempt streams, and the refused calls return together to be refused again.

A run that is part way through has already spent tokens and minutes. Discarding it because one call was refused throws away every step that completed before the refusal.

RETRY WRITTEN AROUND EACH CALLcallerseach one backs off on its own clockproviderrefuses againrun lostLIMIT APPLIED TO EVERY CALLERthe refusal arrives herecallersruntimeholds the accepted rateprovideracceptsrun resumes
What absorbing it takes

The refusal reaches the stage that generates the work.

A shared budgetEvery caller in the system is held by the same limit, because the provider counts them together. Per-caller backoff divides one limit into uncoordinated fractions.
Pressure applied upstreamThe stage generating requests is slowed, so calls stop being created faster than the provider will accept them.
Work held while it waitsA run waiting on a refused call keeps its state and its position, and resumes when capacity returns.
A route to another modelA policy that selects the model sends the retry to a provider with capacity left.

Akka provides retries, backpressure, throttling and circuit breakers as runtime behaviour, so a refusal from a provider slows the work that depends on it without each agent carrying its own limiter.

What this changes

The limiter stops being written per service.

Teams that handle this in application code write a backoff helper, a token bucket and a circuit breaker, then copy those helpers into the next service. Each copy carries its own defaults and drifts as the services change.

A refusal is also a routing decision. Akka Optimize routes each task to the best model from any vendor under your own policies, so work meeting a limit at one provider goes to one with capacity.

Questions

Related questions.

Is a retry with backoff wrong?

Backoff spaces the attempts of one caller. A rate limit applies to the whole account, so the number of callers decides whether spacing each of them individually keeps the total under the limit.

Does the run have to start over?

Durable state records each completed step, so the run resumes at the step that was refused. The tokens spent before the refusal are not spent a second time.

What if the limit lasts for hours?

The paused run holds its state and releases the compute it was using, so a long wait costs storage. The deadline set on the run decides what happens if capacity never returns.

Does this apply to tool APIs as well?

Any upstream that refuses work under load produces the same situation. A payment API, a search index and a model endpoint are each absorbed by slowing the work that depends on them.

On Akka. Retries, backpressure, throttling and circuit breakers are properties of the runtime, described in the Akka SDK documentation. Model routing under your own policies is Akka Optimize.