CAIN-42 CAIN StudioCAGA · Module 4 of 5 · 12 min
Designing human oversight that actually works
Human oversight fails in two ways: the human is never asked, or the human is asked so often they approve everything. Good design routes only consequential, uncertain actions to people.
CAIN holds actions with REQUIRE_APPROVAL when policy and risk pass but trust is unestablished, or when a threshold (amount, scope) is crossed. The agent can never approve itself, and the approval is recorded with the decision.
Measure oversight: approval rate, time to decision, and how often approvals are later overturned. A 100% approval rate is a warning sign.
Key points
- Ask humans only about consequential, uncertain actions.
- No self-approval; record every approval.
- Measure approval rate and reversals.
- 100% approval = rubber stamp.
In the CAIN-42 codebase
platform-gateway/cain_trust_kernel.py— REQUIRE_APPROVAL quorum halting
Practice check
Practice 1 · choose one
A reviewer approves 100% of held actions. This suggests…
Practice 2 · true or false