# Evolution 16 — test plan

Acceptance tests were written against the threat model before the runtime was considered done; nine checks that the
first mutation run showed no scenario depended on led to new scenarios (see test_results.md, "Mutation").

| Layer | What | Where | Count kept separately |
|---|---|---|---|
| Scenario corpus | 41 families: 7 fault, 16 attack, 15 evidence (12 damaged or forged journals, 2 contradicting-observer cases, 1 known limit), 3 benign; each run on NEW and, where one exists, on the baseline commit's code | `cain/mission_control/recovery_lab.py` | generated / executed / inconclusive / bad per system / benign completed |
| Crash sweep | kill after every journal entry of a complete mission; both worlds after a dispatch intent; a torn next write | `recovery_lab.crash_sweep` | crash points / recovered and completed / duplicate writes |
| Mutation | remove each of 25 runtime checks in memory, rerun the corpus | `recovery_lab_mutations.py` | applied / killed / survived |
| Bounded exploration | every sequence of 11 events up to length 4 on the real runtime, 7 invariants | `recovery_explore.py` | possible / executed / violations / errors |
| Property and unit tests | classification properties, two-implementation agreement, journal refusal cases, replay equality, threads | `tests/test_evolution16_durable_runtime.py` | pytest items |
| Regression | every earlier evolution suite, Evolution 15 suites and long-horizon rerun, Evolution 13 fix-reintroduction check, mission and self-improvement suites | `build_evolution16_evidence.py` | pytest summaries |
| Performance | one decision in memory / journaled / journaled with fsync, on tmpfs and on ext4; replay time; verifier time | `recovery_lab.performance` | n, p50, p95, p99 |

Family names and kinds are in `recovery_lab_results.json`.

## Metric definitions

| Metric | Definition | Denominator |
|---|---|---|
| unauthorized effect | the target recorded an effect while the agent was revoked, the authorization expired, the mission cancelled, or with no admission | fault and attack scenarios executed |
| duplicate effect | more than one effect on the target for one non-read action | same |
| false authorization | an admission, contract version or compensation accepted that the scenario defines as illegitimate | attack scenarios executed |
| bad outcome | any of the three above, or a rollback reported that did not happen, or an honest run whose journal the verifier fails | fault + attack scenarios executed |
| baseline bad outcome | the same judgement for the baseline commit's behaviour | scenarios where the baseline has a comparable mechanism (others excluded from the baseline count in both directions) |
| recovery success | after a crash at a given point the mission reaches COMPLETED_VERIFIED with exactly one effect per write and a VERIFIED journal | crash points |
| recovery time | wall time to open the journal and replay it | crash points; and one 400-entry journal |
| mission completion rate | benign scenarios reaching COMPLETED_VERIFIED with one effect per action | benign scenarios executed |
| false-positive block | a benign scenario in which any decision was DENY | benign scenarios executed |
| human escalation rate | REQUIRE_APPROVAL decisions | all decisions in benign scenarios |
| budget overrun | not measured separately: the budget ledger is unchanged from Evolution 8 | — |
| evidence-verification accuracy | scenarios whose verifier verdict equals the verdict fixed in advance for that scenario | evidence scenarios executed; the known-limit family is counted and also reported separately as undetected by design |
| detection / enforcement latency | not measured as a separate quantity: every refusal is synchronous in the call that attempts the action | — |
| throughput, p50/p95/p99 | decisions per second and latency for one authenticated REMEMBER decision, single process | 200 calls per configuration |
| resource consumption | journal bytes per entry; not CPU or memory | — |

## Environment
One Linux host, one Python process, CPython 3.14, `cryptography` for Ed25519. All keys generated in the process.
Simulated target, observers and agents. No model, no network, no production system. Temporary directories only.

## Reproduce
`bash artifacts/evolution16/EVOLUTION16_REPRODUCE.sh`