CAIN-42 CAIN StudioCAAE · Module 3 of 5 · 10 min
Why no LLM judge
LLM judges are convenient and non-deterministic: the same run can pass today and fail tomorrow, and the judge can be manipulated by the very text it grades.
Deterministic graders (rules over actions, schemas, counts, graph checks) give the same verdict every time and can be audited.
Use models where judgment is unavoidable, but never as the only gate for safety-relevant behaviour.
Key points
- LLM judges vary run to run.
- They can be manipulated by graded text.
- Deterministic graders are auditable.
- Never the only safety gate.
In the CAIN-42 codebase
cainbench/— no-LLM-judge gradingcainwitness/engine.py— deterministic detectors
Practice check
Practice 1 · choose one
A key weakness of LLM judges is…
Practice 2 · true or false