CAIN-42 CAIN StudioCAIN-42 live cluster: operational drill
Operational drills on the live CAIN-42 cluster cain-mr-01: 4 PBFT replicas on 3 hosts in 3 regions (Atlanta, Los Angeles ×2, Miami) over WireGuard. The drills were a load test, a rolling restart of all four replicas under continuous writes, and a total disk loss of one replica. This page loads every quorum certificate each replica holds afterwards, including the rebuilt one, and your browser re-checks them.
Operations measured
| Load (concurrent clients) | committed | per second | p50 ms | p95 ms | p99 ms |
|---|---|---|---|---|---|
| 1 | 10/10 | 2.04 | 509.8 | 642.8 | 642.8 |
| 4 | 24/24 | 4.85 | 761.9 | 1209.1 | 1220.6 |
| 8 | 40/40 | 3.4 | 1643.6 | 5812.8 | 7050.5 |
| 16 | 64/64 | 3.29 | 3837.9 | 10210.1 | 12335.4 |
Rolling restart under continuous writes: 83/83 writes committed, 0 client-visible failures, longest gap between commits 2.67 s. Catch-up per replica: cain-mr-node-1 (atl) 9.2 s, cain-mr-node-2 (lax) 12.3 s, cain-mr-node-3 (lax) 10.9 s, cain-mr-node-4 (mia) 9.1 s.
Total disk loss: cain-mr-node-4 (mia) restarted with only its identity key and rebuilt from its peers to the cluster's height and state in 13.6 s; 20/20 client writes committed meanwhile. Its full quorum-signed history is in this bundle and is checked below like every other replica's.
Method: clients run on the atl host and reach replicas over WireGuard; each write is POST /api/v1/cluster/pbft/request, timed to the committed ALLOW, retried with the same request_id on another replica on failure; atl is a 2 vCPU / 3.4 GB VM that also runs the public gateway and older clusters.
Verify (about 5 seconds)
What is checked
- The membership configuration hash is recomputed from the 4 member ids and Ed25519 public keys. Every node and every certificate must carry it.
- For every sequence on every node, the COMMIT_QC and the PREPARE_QC. Each vote must be an Ed25519 signature by a distinct member over SHA-256 of the canonical signed message. It must have the right type (a COMMIT vote never counts as a PREPARE vote) and match this cluster, epoch, view, sequence and digest. Each certificate needs at least 3 distinct signers. The leader's proposal must be signed by the primary of that view, and its digest must bind the proposed operation.
- The certificate hash and signature-bundle hash are recomputed from the content.
- Evolution 3 fast path: a
FAST_COMMIT_QC(a decision taken without the COMMIT round) is accepted only if the published membership declares the fast path and all four members signed it. Three of four is never enough for a fast commit. - The decision chain is folded from genesis:
decision_hash(seq) = H(cluster, epoch, seq, digest, parent). It must be contiguous and identical on all four nodes, and the nodes must end with the same application-state hash. - View-change quorum certificates, and one consensus-to-enforcement AuthorizationCertificate per node.
- Negative controls: a certificate is tampered with in seven ways, and every tampered copy must be rejected.
The same checks, without a browser: curl -so verify_pbft_qc_bundle.py verify_pbft_qc_bundle.py.txt && python3 verify_pbft_qc_bundle.py PBFT_QC_BUNDLE.json (needs pip install cryptography, no CAIN code).