CAIN-42 CAIN StudioCAAE · Module 1 of 5 · 12 min
Agent CI: regression suites for behaviour
Agents regress silently: a prompt tweak, a model upgrade or a new tool changes behaviour. Agent CI runs fixed suites against a real agent endpoint on every change and diffs runs.
Make cases deterministic where possible: fixed inputs, pinned tool fixtures, recorded external responses. Score on outcomes and actions, not prose.
CAINBench runs regression suites and diffs any two runs.
Key points
- Run suites on every change.
- Pin inputs and tool fixtures.
- Score outcomes and actions.
- Diff runs.
In the CAIN-42 codebase
cainbench/— regression suites and run diffs
Practice check
Practice 1 · choose one
Agent CI should run…
Practice 2 · true or false