CAIN-42 CAIN StudioCBTF · Module 3 of 5 · 14 min
Operating a multi-region cluster
CAIN runs 4-replica clusters across regions (for example ATL, LAX, MIA, SJC). Operations focus on liveness (can it still commit?) and safety (do replicas agree on state?).
Watch for the boring killers: full disks, OOM, hourly backups that freeze a replica, unattended upgrades that restart services. Each has taken a CAIN replica down at some point.
Health must be reported honestly: a DEGRADED cluster with one unreachable replica still commits with 3, but it has no fault margin left.
Key points
- Liveness vs safety.
- Disks, OOM, backups and auto-upgrades cause most outages.
- DEGRADED = still committing, zero margin.
In the CAIN-42 codebase
deploy/multi-region/— cluster deploy records
Practice check
Practice 1 · true or false
“With one of four replicas down, the cluster can still commit.”
Practice 2 · choose one