CAIN-42 CAIN StudioCAIN-42 four-server cluster: one-way network partitions
Asymmetric (one-way) network partitions on the live CAIN-42 cluster cain-mr-02 (4 replicas on 4 servers in 4 regions). Unlike a cut link, a one-way failure lets a replica talk but not listen, or listen but not talk. Three cases: a deaf replica, a one-way link between two backups, a mute replica. This page loads every quorum certificate all four replicas hold afterwards, and your browser re-checks them.
What happened
| scenario | one-way drops | writes committed during | target height before → after | agreement after heal | result |
|---|---|---|---|---|---|
| AP1-deaf-replica | on lax: drop atl->lax; on lax: drop mia->lax; on lax: drop sjc->lax | 4/5 | 61 → 66 | 15.2 s | PASS |
| AP2-one-way-link | on sjc: drop mia->sjc | 49/49 | 66 → 115 | 0.4 s | PASS |
| AP3-mute-replica | on atl: drop sjc->atl; on lax: drop sjc->lax; on mia: drop sjc->mia | 6/6 | 115 → 121 | 1.8 s | PASS |
AP1-deaf-replica: expected the 3 others keep committing; the deaf replica cannot collect votes and falls behind, then catches up.
AP2-one-way-link: expected no effect on progress: every replica still has a quorum of peers it can hear.
AP3-mute-replica: expected the 3 others keep committing without it; the mute replica still hears the quorum's messages.
Method: iptables -t raw PREROUTING DROP on the receiving host's WireGuard interface, exact overlay source and destination, removed by a timer on that host. While a replica is cut off one way, the controller may be unable to read its status; the heights after healing and the identical decision chains below are what count.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).