CAIN-42 CAIN StudioCART · Module 3 of 5 · 13 min
Building a repeatable agent attack harness
Manual pokes find bugs; a harness finds regressions. Encode each attack as a case: setup, the adversarial input, the expected safe outcome (refused, held, sanitised) and the evidence to check.
Run the harness against a deterministic environment so results are comparable between runs. Score on outcomes, not on the model's words: did the harmful action execute or not?
CAIN's own lab runs large deterministic campaigns and reports the actual counts executed and blocked; report the real number, never a rounded boast.
Key points
- Each attack = setup, input, expected safe outcome, evidence.
- Deterministic environment for comparable runs.
- Score outcomes (did it execute?), not words.
- Report actual counts.
In the CAIN-42 codebase
cain/argus/lab.py— deterministic adversarial lab: run_argus_lab(cases=N)
Practice check
Practice 1 · choose one
A good attack case scores…
Practice 2 · true or false