CAIN-42 CAIN StudioCAIN-42 multi-region cluster: fault-injection proof
The live CAIN-42 PBFT cluster cain-mr-01 has 4 replicas on 3 hosts in 3 Vultr regions: Atlanta, Los Angeles ×2 and Miami, connected by a WireGuard mesh (n=4, f=1, quorum 3). During this run we stopped real replicas on their own hosts. Miami down: it kept committing. One Los Angeles replica down: it kept committing. The whole Los Angeles host down (2 of 4): it refused to commit. Atlanta down, including the primary: view change, and it kept committing. Then everything came back. This page loads every quorum certificate all four replicas produced, with the original Ed25519-signed votes, and your browser re-checks them. Nothing here asks you to trust the server. The live cluster page shows the same cluster running now.
Verify (about 5 seconds)
Check the fault schedule against the signatures
A stopped replica cannot sign. For each outage, this checks that none of the decisions taken during it carries a signature from the stopped replicas. It also checks that no request from the refused phase ever entered any decision chain, and that every decision was signed from at least 2 regions.
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, plus the fault schedule, without a browser (needs pip install cryptography, no CAIN code): for f in verify_pbft_qc_bundle.py verify_multi_region_bundle.py; do curl -so $f $f.txt; done; curl -so MULTI_REGION_BUNDLE.json PBFT_QC_BUNDLE.json; python3 verify_multi_region_bundle.py MULTI_REGION_BUNDLE.json