The candidate runs against live traffic without serving it, and every response it produces is graded by the criteria that grade the model in service. The comparison of those grades decides the promotion.
Akka Optimize promotes a candidate when its quality holds at the score the model in service meets and its token use falls. Traffic moves through the routing policy, so promotion is a routing change.
A shadow run sends production requests to the candidate model and discards its responses. The user receives the answer from the model in service, and the candidate answer goes to the graders.
Grading the candidate on live traffic is what makes the comparison usable. A published benchmark measures a model on somebody else's distribution of work. The promotion turns on your own distribution, which is the distribution a shadow run measures.
The promoted model becomes the destination the policy names for that class of work. Agents routing through the policy reach it on the next request, with no release and no code change.
The same mechanism measures a prompt revision. The revised prompt runs against live traffic under the criteria that grade the prompt in service. Traffic reaches the revision once the grades support the change.
Akka Verify writes every shadow run, every grade and every promotion to the interaction record it holds for governance. The decision to promote is evidenced on the record an auditor reads.
A shadow run pays for the candidate responses it grades. Shadowing a sample of the class bounds that cost, and the run ends once the comparison is decided.
Grades are held per class of work, so a subset that fails appears as a class that did not clear its threshold. The routing policy moves the classes that cleared and leaves the rest with the model in service.
Long enough for the candidate to see the range of the class, including the cases that appear rarely. Volume alone does not end it, since a thousand near-identical requests measure one case a thousand times.
The sign-off recipe engine routes the change to the personas the control matrix names, with the evidence dossier attached. A model change is one of the declared change types the review is scoped from.
The gate reads quality and token consumption together, and the threshold for the class decides which one the promotion turns on. A class where accuracy carries the consequence can promote on quality alone.
What decides which class of work is a candidate.
The same gate, used to return the traffic.
Routing traffic to the best open-weight model and training smaller ones on your data.
