CAIN-42 CAIN Studio

CAIN-42 evidence library

VAGI distribution-shift probe — paid model baseline (2026-11-18)

Evidence bundle: 0 claims, 0 of 0 invariants held.

Last reviewed 2026-11-19

SHA-256 manifest

Verify it in your browser

Your browser downloads the bundle's SHA256SUMS manifest and the file(s) behind this page, hashes them with SHA-256 locally (WebCrypto), and compares. A match shows the record you are reading is the published one; it does not by itself prove who published it (see the signed claims registry and the bundle verifier for that).

The bundle's own README

VAGI distribution-shift probe — paid model baseline (2026-11-18)#

The result that matters#

The hand-written library scores 5/100 on the 100-task distribution-shift probe. Three raw frontier models, given the probe's data and the intended policy card, score far higher:

ArmModelScoredAcceptedRate95% WilsonCost
haiku-shiftClaude Haiku 5.51007474.0%[0.6463, 0.816]$0.0149
sonnet-shiftClaude Sonnet 5.51008181.0%[0.7222, 0.8749]$0.2603
fable-shiftClaude Fable 5.2655483.1%[0.7218, 0.9028]$0.7511
fable-shift-replicationClaude Fable 5.2 (seed 9503)232087.0%[0.6787, 0.9546]$0.2859

The 5/100 is substantially a library-coverage gap, not a general-intelligence gap. The probe is adversarial and partly underspecified: the software hidden target (x % 3) is not inferable from the public tests, and all models scored 0 on tool_orchestration. That is a probe/prompt problem, not a general model failure. The result does not show VAGI; it shows the suite's shift probe is a coverage map, not an AGI test.

Cost and the cap#

Shift spend: $1.312174 of the $1.280558 remaining. Overshoot: $0.031616 because concurrent requests passed the budget check together. The Fable arm stopped at 65/100; the replication stopped at 23/100. Rates are on scored tasks only.

Per-domain#

DomainHaikuSonnetFable (scored)Library
business_decision10/1010/107/70/10
data_analysis10/1010/107/70/10
governance10/1010/106/60/10
memory_retrieval10/1010/106/60/10
planning7/106/106/60/10
recovery10/1010/106/60/10
science_calculation10/1010/107/70/10
security_triage2/1010/107/70/10
software_repair5/105/102/75/10
tool_orchestration0/100/100/60/10

Verify#

python3 -I verify_offline.py .

Recomputes counts and cost from raw rows, regenerates the shift tasks for the two seeds, re-scores every answer with the published shift checker, and checks that the prompt view contains no expect, hidden, fixes or bad field. It imports the suite source bundled here, not CAIN.

Limits#

Try CAIN-42 on your own agents

Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.

Create a free account →  ·  Try the sandbox  ·  See the whole ecosystem