The tests come from the specification that produced the code. Scenarios, evaluation criteria and edge cases each carry the requirement they were derived from, so a failure names the requirement it belongs to.
Akka Specify generates scenarios, evaluation criteria and edge cases as part of the delivered system. Each one measures the system against the requirement that asked for it.
A test written by reading the code encodes what the code does. When the code was generated, that test confirms the choices the generator made and says nothing about the requirement.
The specification is the independent statement. A scenario derived from a functional requirement fails when the system does something the requirement did not ask for, whoever or whatever wrote the code.
When a scenario fails, the trace runs back to the requirement that produced it. The change is made in the code when the requirement was right, and in the specification when the requirement was wrong.
Akka Verify captures every interaction into a non-sampled record, so a failing run is replayed against changed code to show whether the outcome moved. The same record is what the evaluation criteria are scored against once the system is live.
A test derived from the requirement fails when the system does not meet the requirement, whatever produced the code. A mistake the generator introduced shows as a scenario that does not pass.
Deterministic evaluators score structured output against a rule. An LLM judge scores unstructured output against the criteria written in the specification, and both run offline and online against the interaction log.
A build-and-deploy gate runs the evaluation suite and fails the deploy when the signed control set is unmet. The gate produces a versioned, hash-chained report.
A change to the specification regenerates the tests that trace to it. A change to the code is checked against the tests that exist, so a change that breaks a requirement is refused before it merges.
Existing tests keep running. Generated scenarios cover the requirements in the specification, and a suite written earlier covers whatever its authors chose to cover.
