# cain-mr-03 — rolling deploy of the view-184 fix, then a fresh 48 h soak

Date: 2026-11-18

## What is being deployed

- Hotfix commit: `2478e9cbf32e69d48f750270b9bdab47dd7ecfdc` (`hotfix/mr03-view184`)
- Parent (currently deployed): `e8df8998f`
- Diff: **one file**, `platform-gateway/cain_pbft_engine_33.py` (+30/−6)
- Image: `cain-cluster-node:mr-2478e9c`
  - Derived from the running `cain-cluster-node:mr-e8df8998f` image plus the one patched file
    (`docker create` + `docker cp` + `docker commit`), so no unrelated branch code ships.
  - Verified: `/app/main.py` hash identical to the running image; engine hash equals the hotfix
    tree; `python3 -c "import cain_pbft_engine_33"` succeeds inside the image.
- Why not build HEAD: HEAD is 45 commits / 1,660 files ahead of the deployed image.
- Watchdogs (`cain-mr-autoheal.timer`, `cain-sentinel-guardian.service`) are paused during the
  roll and resumed afterwards.

## Rolling deploy command

```bash
cd /root/cain-studio-platform
python3 -u scripts/cain42_bft/deploy_multi_region.py \
  2478e9cbf32e69d48f750270b9bdab47dd7ecfdc --topology mr03 --rolling
```

The script reconstructs each container from its own config (same data dir, env, command,
healthcheck), keeps the old one as `<name>-was-e8df8998f-<stamp>`, and requires each replica to
reach its peers' height and view before touching the next. Primary goes last.

## Post-deploy checks (all must hold before the soak)

1. `GET /api/v1/live-cluster/health?cluster=cain-mr-03` → `HEALTHY`, 4/4.
2. All four replicas: one `view`, one `commit_index`, one `application_state_hash`.
3. Every container reports `cain42.commit=2478e9c…` and is `running/healthy`.
4. `docker inspect` shows the patched image on all four hosts.
5. Watchdogs active again; autoheal plan is `ok` for every node.
6. Lax `/` free ≥ 12 GB (preflight).

## Soak

Unit draft: `/root/cain-mr03-soak-48h-2026-11-18.service.draft`

- name: `soak-multiregion-48h-2026-11-18`
- hours: 48, interval 6 s, fault every 1200 s, publish every 3600 s
- record: `deploy/multi-region/cain_mr03_deploy_record.json`
- min free: 12 GB start / 5 GB during
- expected end: ~2026-11-20T04:40Z

Preflight is fail-closed: all four containers running, HEALTHY at one height/state, and a
12-sequence decision window readable from at least two replicas with no disagreement.

## Rollback

The deploy script keeps every old container as `cain-mr3-node-N-was-e8df8998f-<stamp>` with the
old image. To roll one node back: stop the new `cain-mr3-node-N`, rename the `-was-` container back,
and start it.

## Honesty

Self-observed, same-admin-domain. No third-party audit. The 48 h soak is not passed until a final
signed checkpoint reaches 48 h and `verify_soak.py` returns PASS.
