CAIN-42 CAIN StudioCAIN-42 live cluster: restore from backup
Disaster-recovery drill on the live CAIN-42 cluster cain-mr-01. One replica was restored from an online backup, not from its peers' live state, while the cluster kept committing. The backup was deliberately behind, and the replica had to close the gap through verified state transfer. This page loads every quorum certificate each replica holds afterwards, including the restored one's, and your browser re-checks them.
Measured
| replica | cain-mr-node-2 (lax) |
|---|---|
| backup height | 370 |
| cluster height when restored | 382 |
| RPO: decisions after the backup | 12; decisions lost: 0 |
| restored files match the backup manifest | True |
| RTO: start to identical height and state | 8.3 s |
| writes committed during recovery | 5/5 |
Method: online SQLite backup of the consensus databases on the replica's own host; restore = stop, archive the data dir (identity kept), copy the snapshot in, start; the gap after the snapshot is closed by verified state transfer from peers.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).