CAIN-42 CAIN StudioCAIN-42 four-server cluster: two replicas lose their storage
Disaster-recovery drill on the live CAIN-42 cluster cain-mr-02 (4 replicas on 4 servers in 4 regions). Two replicas on two servers lost their storage at the same time, and each was rebuilt only from its off-host backup, held on a server in another region; their on-host snapshots were moved aside first. While both were down the cluster had no quorum and committed nothing. This page loads every quorum certificate all four replicas hold afterwards, and your browser re-checks them.
Measured
| replicas that lost their storage at once | cain-mr2-node-3 (mia), cain-mr2-node-4 (sjc) |
|---|---|
| surviving replicas | cain-mr2-node-1 (atl), cain-mr2-node-2 (lax) |
| while two were down (no quorum) | 0/4 writes committed; survivor heights {'cain-mr2-node-1': 50, 'cain-mr2-node-2': 50} before, {'cain-mr2-node-1': 50, 'cain-mr2-node-2': 50} after; outage 53.8 s |
| restored only from off-host backups | cain-mr2-node-3 from the copy held in atl, checksums match: True; cain-mr2-node-4 from the copy held in lax, checksums match: True |
| RPO: decisions after the backup | {'cain-mr2-node-3': 12, 'cain-mr2-node-4': 12}; decisions lost: 0 |
| first new commit after restart | 9.9 s |
| RTO: start to identical height and state on all four | 10.3 s |
| writes committed during recovery | 2/2 |
Method: online SQLite backup of the consensus databases on each replica's host, copied to another region's host and re-checksummed; loss = stop, move the whole data dir (identity kept) and the on-host snapshot aside; restore = stream the off-host copy back, check against the manifest, start; the gap after the snapshot is closed by verified state transfer from the survivors.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).