CAIN-42 CAIN StudioAGENTIC AI 75 · #58
Repeated-run comparisons of agent frameworks
AI Multiple reported thousands of controlled runs comparing thirteen frameworks.
Last reviewed 2026-11-11
Agentic AI 75 / #58
2026 status: Independent benchmark report · Category: Evaluation
The 2026 development
AI Multiple reported thousands of controlled runs comparing thirteen frameworks.
The governance gap
Repeated runs are more informative than a demo, but only if the harness, model, prompts, tools and scoring are fixed and published. Otherwise the comparison measures the harness.
Where CAIN-42 fits: Moat 6 — Adversarial Trust Engine
Core question: Can CAIN break its own guarantees? Continuous attack, mutation and red-team testing of the controls themselves.
Publish the harness as a signed artifact and re-run it. CAIN's eval-run and agent-ci paths keep the suite, the model version and the result together.
How the wiring works
- Freeze the harness, tasks and scoring rubric.
- Pin the model and tool versions.
- Run enough repetitions to report variance.
- Publish the harness and the raw results.
What is real today
LIBRARY Production code and tests exist in this repository; there is no hosted route for this exact use.
The evaluation harness exists. CAIN has not run a thirteen-framework comparison.
| Component in this repository | What the module says it is |
|---|---|
keystone/evalrun.py | Signed evaluation runs: "these scores came from this code, this data, these model providers, unmodified". |
keystone/agentci.py | AGENT-CI KEYSTONE: a release gate for AI agents that grades the path, not only the answer. |
keystone/heldout.py | Held-out evaluation vault + contamination certificate, run in the CAIN agent jail, signed. |
Descriptions are the first line of each module's own docstring, read without importing it. A module path is a pointer into this repository, not a hosted endpoint. Its existence is checked by the test suite; its status for this use is the label above.
Check it yourself
- Run the live agent release gate
/agent-ci-keystone - Fire a live decision through the CAIN-42 pipeline
/fabric/try— the same enforcement path every product sits behind.
Commercial package
| Offer | Framework Evaluation Review |
|---|---|
| Buyer | Procurement teams and platform vendors |
| Price (proposal) | $5,000–$12,000 per study |
| Bundle | Code & Proof — Prove the fix, not just the diff. |
| Operator stream | Stream 10 of the 10-stream operator playbook |
Platform engineering, AI coding-tool vendors, security engineering is the bundle buyer. The entry offer for this bundle is Fabrication audit or long-run engineering attestation; the range is $1,500–$6,000 per audit; $2,500–$4,000/month retainer. A price on this page is a proposal, not a validated rate, and no customer is claimed.
Buy this as a product or service
CAIN-A75-058 — Framework Evaluation Review PILOT
A bounded pilot report: up to five scenarios, the relevant library engine or analysis, a written result and the stated limits. Engineering time is quoted per pilot.
Kind: Evaluation · Price (proposal): $5,000–$12,000 per study · Lead time: 10 business days.
Open the product page · all 75 products
What this page does not claim
What this page is not. CAIN-42 is not affiliated with, endorsed by, integrated into, or shipped with the organisation named in the 2026 item. The item is public context; the page describes how CAIN-42 would govern that class of capability. No page here claims a customer, a partnership, a certification, a patent, or a production integration. Prices are proposals. Built by a three-person team.