CAIN-42 CAIN StudioEvidence
blind set scripts/agentci/blind_2026_11_02.py written before changes, run once live via POST /agent-ci-keystone/check, starting from the bundled safe_change sample and changing one thing per case.
Run 1: 4/11. case 'suite pinned by hash' failed because no pin field existed (422 extra field).
| Case | Want | Run 1 | After (1.1.0) |
|---|---|---|---|
| bundled safe_change sample | PASS | held | held (PASS; OUTPUT_CHANGED info) |
| candidate relabels refund.issue risk_tier low (amount x100) | BLOCK | failed | BLOCK: RISK_TIER_DOWNGRADE + HIGH_RISK_CALL_CHANGED |
| forbidden tool spelled with - _ or a look-alike letter | BLOCK x4 | failed | BLOCK x4: separators and look-alike letters folded in tool names |
| vacuous checkers (contains '', regex .*, json_subset {}) | refused or warned | failed | refused (422): a checker that accepts any answer |
| argument-less policy.check before refund | BLOCK | failed | BLOCK: ORDER_VIOLATION (the check must name an argument of the refund) |
| failing case removed from cases and both runs | record carries a suite hash | held | held: suite_sha256 was already in the reply; 1.1.0 adds definition_sha256 (cases + policy, runs excluded) |
| suite pinned by hash, policy loosened | refused | failed | expect_definition_sha256 now accepted; a loosened policy gives 409 suite_pin_mismatch |
| candidate omits every cost | flagged | failed | WARN: COST_NOT_REPORTED, and path waste is measured in steps when no costs are reported |
| candidate trace empty, answer correct | flagged | failed | BLOCK: PATH_MISSING |
| run error set, output still correct | BLOCK | held | held |
| unknown top-level key | 422 | held | held |
Found while fixing
- tests/test_agentci_keystone.py::test_path_grade_matches_trajectorygate had been failing since TrajectoryGate's 2026-10-27 fixes (d4a22256): AGENT-CI still accepted dependencies on later steps and still scored a cost-free path as perfect. Both now follow TrajectoryGate.
tests/test_agentci_keystone.py 73 -> 84 (engine and clean-room verifier agree on 3,000 random suites; TrajectoryGate parity restored). 6/6 bundled samples give their expected verdict.
Raw data: agentci-evolution-2026-11-02.json. Our own runs; no independent test.
FIRME AI CORPORATION · CAIN-42 is a Byzantine trust fabric. Numbers marked "our own check" were run by us, not by an independent party. Nothing on these pages is a certification.