CAIN-42 invariant · E21

D50: benchmark changes invalidate incompatible comparisons

Held · CAIN-42 Evolution 21 -- Governed Open-Ended Intelligence Fabric

Last reviewed 2026-10-01

held   niche Benchmarks, coverage & performance

What this rule means

CAIN-42 must always satisfy: benchmark changes invalidate incompatible comparisons. It is one of 155 invariants checked for CAIN-42 Evolution 21 -- Governed Open-Ended Intelligence Fabric. An invariant is a rule the system may never break, whatever an agent or attacker does; the test suite tries to break it across many scenarios and records the result.

Recorded detail

[
 {
  "comparable": false,
  "reason": "BENCHMARK_VERSION_MISMATCH_COMPARISON_INVALID"
 },
 [
  "STALE_BENCHMARK_VERSION"
 ]
]

Check it yourself

Browse the raw bundle

Related rules

• D140: a missing dimension is a regression
• D142: benchmarks detect selective reporting and partial metrics

← D49 · all 155 · D51 →