# Evolution 16 — test results

Every number is from `python3 scripts/cain42_evolution/build_evolution16_evidence.py`, run 2026-10-12 on one host
against the working tree on top of commit `d190f159`. Raw outputs are the JSON files named in each section.
"Tested" here means: executed by the author's scripts, in a deterministic local simulator. It does not mean
independently reproduced, externally reviewed, deployed or validated by a customer.

## 1. Scenario corpus (`recovery_lab_results.json`, seeds 16, 17, 18, 10 variants per family)

| Count | Value |
|---|---|
| families | 41 (7 fault, 16 attack, 15 evidence, 3 benign) |
| scenarios generated | 1,230 |
| scenarios executed | 1,230 |
| inconclusive (scenario raised) | 0 |
| fault + attack scenarios executed | 690 |
| bad outcomes, new runtime | **0 / 690** |
| fault + attack scenarios with a comparable baseline mechanism | 420 |
| bad outcomes, baseline, on those | **330 / 420** |
| bad outcomes, new runtime, on the same 420 | 0 / 420 |
| scenarios with no baseline mechanism (excluded from the baseline count) | 270 |
| benign missions completed | 90 / 90 |
| benign missions with any refusal (false-positive block) | 0 / 90 |
| human escalations in benign runs | 0 of 1,230 decisions (approvals were supplied up front where the plan requires them) |

The 90 comparable cases in which the baseline was **not** worse: failed compensation and unauthorized compensation
(the existing `RecoveryPlanner` already handles both) and a crash before any effect (a retry is harmless there). Those
are the negative results: on these three families Evolution 16 shows no advantage.

Baseline "bad" by construction in four families (the baseline has no dispatch step, so nothing can re-check at
dispatch): revoked, expired, cancelled, redispatch. That is what "no check exists" means; it is not a measured exploit
of a deployed system.

## 2. Evidence verification

| Count | Value |
|---|---|
| scenarios with a verifier verdict fixed in advance | 570 |
| verdict as expected | 570 / 570 |
| of which damaged, forged or incomplete journals (12 families, 360 cases): FAILED, INCOMPLETE, UNSUPPORTED or UNAVAILABLE as specified | 360 / 360 |
| **known limit**: a pinned observer signs a false statement; harm occurs and the verifier still says VERIFIED | **30 / 30 undetected, by design** |

The last row is a miss that the mechanism cannot avoid, reported as such rather than excluded.

## 3. Crash sweep (`recovery_lab_results.json`, `crash_sweep`)

| | |
|---|---|
| journal entries in the reference mission | 33 |
| crash points (after every entry; both outcomes after a dispatch intent; with and without a torn next write) | 71 |
| recovered, mission completed, one effect per write, journal VERIFIED | **71 / 71** |
| duplicate write effects | 0 |
| missing effects | 0 |
| read-only action repeated after an unknown outcome (allowed for that class) | 2 |
| recovery time | p50 9.1 ms, p95 31.5 ms, max 144.5 ms (33-entry journal, machine under load from the build) |

## 4. Mutation (`mutation_results.json`)

25 runtime checks removed one at a time: **25 / 25 killed**. The first run killed 16; the nine survivors showed which
checks no scenario exercised, and scenarios were added (redispatch after success, false absence, replay after
reconciliation, rollback via an approved widening, narrower stale writer, mismatched succession, irreversible action
with a declared compensation). Two checks are documented as not testable in isolation (see formal_assurance.md).

## 5. Bounded exploration (`bounded_exploration.json`)

All 16,104 sequences of up to 4 events from an 11-event alphabet, on the real runtime: **0 violations of I1-I7,
0 errors**, 544 s.

## 6. Unit, property and concurrency tests

`tests/test_evolution16_durable_runtime.py`: **61 passed**.

## 7. Regression (`regression_results.json`)

| Suite | Result |
|---|---|
| historical evolution suites (24 files + 2) | 500 passed |
| Evolution 15 suites | 44 passed |
| Evolution 15 long-horizon rerun, seed 15 | governed 0 / 550 attacks, baseline 475 / 550, benign 200 / 200 |
| Evolution 13 fixes still caught when re-introduced | exit 0 |
| mission (E38) and governed-evolution suites | 149 passed |
| gateway-importing delegation suites | **not run**: their modules default to the live `/app/data` directory |

No regression was observed. The two changed files (`coordinator.py`, `governed_runtime.py`) are covered by the first
three rows.

## 8. Found defects, before and after (`found_defect_probes.json`)

| Defect | Baseline commit's file | Working tree |
|---|---|---|
| F16-01 coordinator re-executes an unresolved step | duplicate effect 10 / 10 | 0 / 10 |
| F16-02 retracted fact still trusted | admitted 10 / 10 | 0 / 10 |

## 9. Performance (`performance_results.json`)

One authenticated REMEMBER decision, n = 200, single process.

| Configuration (ext4) | p50 ms | p95 ms | p99 ms | decisions / s |
|---|---|---|---|---|
| Evolution 15, in memory | 0.18 | 0.22 | 0.36 | 5,391 |
| journaled, no fsync | 0.45 | 2.69 | 3.58 | 1,458 |
| journaled, fsync (the default) | **5.00** | 15.93 | 41.84 | **150** |

**This is a regression in speed: about 28 times slower at the median and 36 times lower throughput with durability
on.** It is the price of two forced writes per decision. On tmpfs fsync costs almost nothing (p50 0.44 ms), which is
not a durability measurement.

Replay of a 406-entry journal (242 KB): 143 ms. Verifier on the same journal: 128 ms. Journal size: about 600 bytes per
entry. CPU and memory were not measured.

## 10. Sample (`sample/`)

A complete mission's journal, the public trust file, a signed head and the verifier's output (VERIFIED, 31 distinct
checks performed). All keys in it are throwaway test keys generated for the run.

## Not measured
Mission completion with a real agent; behaviour under a real tool's failures; budget overruns (ledger unchanged);
detection latency as a separate quantity; multi-process or multi-host operation; power loss.