Grading is scoring each response against the criteria you set for a good answer. Structured output is graded by a rule: a generated SQL query either returns the rows the task required or it does not. Unstructured output is graded by a second model reading it against criteria written in advance.
Akka grades every run against the interaction record the agent already wrote. Rule-based evaluators and the LLM judge write their verdict onto that record, so the grade stays attached to the run it scored.
A deterministic evaluator is a check that returns the same verdict on the same output every time it runs. Generated SQL executes against a database and the returned rows are compared with the rows the task required. A JSON payload validates against its schema or fails on a named field.
An LLM judge is a second model that reads a response and scores it against criteria written before the run. A support reply has no single correct text, and the criteria name what it had to contain: the account status and the policy that applied.
Evaluations run offline against recorded traffic and online against traffic as it is served. An offline run answers whether a change improved quality before it ships, and an online run answers whether quality fell afterward.
Akka Verify reads the interaction record for governance and Akka Optimize reads it for training, and both read the grades already attached to the run. A stack of separate products copies the trace into an evaluation tool that stores the grade apart from the run.
Grading works against agents running in a third-party coding tool such as Cursor, Claude Code, or Copilot. A grade that stops an action requires the agent to run on Akka.
A unit test asserts one expected output and fails on every other. Two agent responses can differ in wording and both be acceptable, so the evaluator scores properties of the answer: the fields it contains and the policy it applied.
The judge is measured against a reference set graded by people before it grades anything that gates a decision. Disagreement between the judge and that set means the criteria need rewriting.
Online evaluation grades traffic as it is served, so the fall appears on the same record the release was gated on. A model promoted on that record is rolled back at the gate that promoted it.
An evaluation running inline in the transactional path returns its verdict before the response is emitted, and the runtime blocks or escalates on that verdict. An inline check adds its duration to every response it guards, so compliance chooses which controls run inline.
Evaluation, inline enforcement, and the evidence record an auditor reads.
The graded records a smaller model trains on.
Cost per verified task, and what it prices.
