CAIN-42 CAIN Studio

CAIN-42 evidence library

Cluster restore — 2026-11-16

Evidence bundle: 0 claims, 0 of 0 invariants held.

Last reviewed 2026-11-19

The bundle's own README

Cluster restore — 2026-11-16#

Two PBFT clusters were degraded. Both are back to 4 of 4 healthy replicas with quorum-valid certificates. This bundle is the evidence of what was wrong, what was changed, and what is still not proven.

(docs/audits/PBFT-SAFETY-INCIDENT-2026-11-15.md). It stays stopped.

Nothing here is a third-party audit or a certification. The 48-hour soak is at 0.01 of 48 h, not passed. Evidence class: self-observed, same-admin-domain.


1. What was found#

1.1 cain-mr-03 had lost a replica (root cause: the guardian revived a retired node)#

The Moat #7 Sentinel (cain-sentinel-guardian.service) still listed the retired, quarantined container cain-mr2-node-1 as a required container. At 2026-11-16T10:54:29Z it found that container stopped and auto-started it.

cain-mr-02 and cain-mr-03 were built on the same overlay addresses and ports (10.88.0.1:29811 … 10.88.0.4:29814). The revived cain-mr2-node-1 took 10.88.0.1:29811 from the live replica, and at 11:04:17Z cain-mr3-node-1 exited with code 3:

ERROR: [Errno 98] error while attempting to bind on address ('10.88.0.1', 29811): address already in use

The guardian had degraded the cluster it was guarding. The live cluster ran on 3 of 4 replicas (quorum was still met, so it did not halt) while the 48-hour soak kept logging CHECK ERROR (state comparison skipped).

The retired container was revived once more by the old guardian at 11:35:26Z, seconds after it was stopped and about a minute before the guardian was restarted with the fix. It could not bind and died at 11:48:06Z. It has not been started since; the retired-container guard is exercised directly in post-fix/retired-guard-proof.txt.

1.2 cain-mr-01 was stalled and committing without quorum#

A separate, more serious fault. Replica heights were frozen at different values:

Replicacommit_indexhealth
node-1143271HEALTHY
node-2143230HEALTHY
node-3143230HEALTHY
node-4143035SAFE_MODE

node-4 carried wal_write_failure: "sequence 143035: OperationalError: disk I/O error" and had been in SAFE_MODE since that write, with no restart since 2026-11-03.

A write sent to node-1 returned:

{"status": "COMMITTED", "decision": "ALLOW",
 "quorum_certificate_status": "INVALID: INSUFFICIENT_QUORUM",
 "quorum_certificate_accepted_signers": 1, "sequence": 143273}

So the node reported COMMITTED while only one replica had signed, below the quorum of 3. The local commit is real (the height advanced); the outcome was not quorum-certified. The response made that visible only in an adjacent string field — a caller reading status alone would treat an uncertified outcome as final. That is the fail-open reporting defect recorded in section 3.2.

After the repair the same write returns "quorum_certificate_status": "VALID" with 4 accepted signers (mr01/post-fix-write-probes.txt).


2. What was done#

#ChangeFile
1Stopped the invalid soak runsystemctl stop cain-mr03-soak-48h-2026-11-15b
2Stopped the revived retired container (container and data preserved)docker stop cain-mr2-node-1
3Required container cain-mr2-node-1 → cain-mr3-node-1deploy/scripts/cain_sentinel_guardian.py
4The guardian now refuses to start any retired cain-mr2-node-* containersame
5Cluster monitor cain-mr-02 → cain-mr-03same
6Gateway knows cain-mr-03platform-gateway/routers/live_cluster.py
7Gateway refuses to count a replica whose reported node_id/cluster_id does not match the requested clustersame
8outcome_certified / outcome_final added to the commit response (staged; not live until the node image is rebuilt)platform-gateway/cain_pbft_engine_33.py
9Restarted cain-mr3-node-1; it caught up from its peersdocker start cain-mr3-node-1
10Restarted the SAFE_MODE replica cain-mr-01/node-4; it caught up 143035 → 143275docker restart cain-mr-node-4
11Started a new, clean 48-hour soak with a new namecain-mr03-soak-48h-2026-11-16c

Change 7 matters for honesty: before it, GET /api/v1/live-cluster/health?cluster=cain-mr-02 reported 3 of 4 replicas reachable at height 11468 for a cluster that no longer exists — it was counting cain-mr-03's replicas, which answer on the same overlay addresses. After the change the retired cluster reports HALTED, 0/4 and the live cluster reports HEALTHY, 4/4.


3. Verification after the repair#

3.1 Live cluster cain-mr-03#

All four replicas: view 59→, identical commit_index, identical application_state_hash, health_state: HEALTHY. A write returns VALID with 4 accepted signers.

3.2 Second cluster cain-mr-01#

All four replicas at commit 143275 and advancing; three consecutive writes return VALID with 4 accepted signers. Host free disk at repair time: atl 241 GB, lax 9.2 GB, mia 26 GB, sjc 62 GB.

3.3 Soak#

New run soak-multiregion-48h-2026-11-16-mr03c started 2026-11-16T11:48:09Z. Its preflight passed (all four containers running, all HEALTHY at one height and state, 12-sequence decision window readable from at least two replicas with no disagreement). checkpoint-0000.json is VALID, 48 certificates verified; verdict RUNNING (0.009 of 48.0 h).

The previous run soak-multiregion-48h-2026-11-15-mr03b is marked INVALID in its own bundle (INVALID-RUN.md): it is 18 h of a degraded window, not a 48-hour run, and must not be cited.


4. Follow-ups (fixed the same day)#

Two open items from this restore were fixed on 2026-11-16 and are recorded in followups/:

1. The soak verdict is no longer divergences == 0 alone. It now fails on a sustained streak of unreadable checks with no injected fault, on a cluster that is not healthy at one height when the run ends, and on a host below the new --min-free-gb-run floor. verdict() is a pure function; tests/test_soak_verdict.py is 8/8. 2. The lax host was reclaimed from 9.23 GB to 14.94 GB by removing four clean, reproducible verification clones. The quarantined cain-mr-02 data (20.5 GB) was not touched.

The run that enforces the new verdict is soak-multiregion-48h-2026-11-16-mr03d (started 12:05:15Z, checkpoint-0000 VALID). It is 0.01 of 48 h.


5. What this does not prove#

interrupted. Do not publish a 48h or 72h claim from this bundle.

error was not root-caused.** Miami has 26 GB free now; the error is consistent with the host having been full earlier (the same host filled up once before), but that is not established here.

image (cain-cluster-node:mr-214970f) still has the old response shape until it is rebuilt and redeployed. The invalid-quorum state is documented, not prevented.

not independent witnesses of each other.

used as a trust anchor.


6. Reproduce this check#

python3 verify_restore_2026_11_16.py            # verifies MANIFEST.json and the recorded facts
curl -s 'http://127.0.0.1:8088/api/v1/live-cluster/health?cluster=cain-mr-03'
curl -s 'http://127.0.0.1:8088/api/v1/live-cluster/health?cluster=cain-mr-01'
systemctl is-active cain-mr03-soak-48h-2026-11-16c.service
python3 /root/cain-studio-platform/scripts/cain42_bft/verify_soak.py \
  /root/cain-studio-platform/platform-gateway/frontend/proof/bundle/soak-multiregion-48h-2026-11-16-mr03c

7. Files#

FileContents
RESTORE.jsonmachine-readable summary: findings, changes, before/after states
pre-fix/docker state, exit log, sentinel auto-heal log, soak tail, per-replica status before
mr01/cain-mr-01 before-fix status, the invalid-quorum response, after-fix write probes
followups/soak-verdict fix and tests, disk reclaim before/after, the new run's checkpoint-0000
post-fix/per-replica status, gateway health for both clusters, sentinel state, soak verification
RECOVERY-COMMANDS.txtthe exact commands run, in order
verify_restore_2026_11_16.pystandalone verifier, imports no CAIN code
MANIFEST.jsonSHA-256 of every file in this bundle

Try CAIN-42 on your own agents

Create a free account and every new account starts with a 30-day trial of the full platform. Or try the sandbox first, with no account at all.

Create a free account →  ·  Try the sandbox  ·  See the whole ecosystem