# Timeline — cain-mr-03 view-184 orphan (2026-11-18 UTC)

Sources: `journalctl`, `/var/log/cain-cluster-health.log`, `logs/deploy.log`,
`soak/first-fault.log`, the gateways' `/api/v1/live-cluster/*` endpoints.

| Time (UTC) | Event |
|---|---|
| 00:06:27 | autoheal restarts `cain-mr3-node-1` (`status=running health=unhealthy`) |
| 00:06:52 | container starts on `mr-e8df8998f` |
| 00:06:56 | sentinel starts the same container again (double-start race) |
| ~00:45 | monitor reports DEGRADED: `VIEW_SPREAD: views [183, 184]` |
| 03:17 | evidence captured; node-1 at view 184 / 142173, peers at view 183 / 142181 |
| 03:23:13 | node-1 restarted with watchdogs paused |
| 03:23:46 | node-1 recovers to **view 183**, catches up |
| 03:41:31 | 4/4 HEALTHY, view 183, height 144846, identical state |
| 04:31 | rolling deploy of `mr-2478e9c` starts (watchdogs paused) |
| 04:33–04:35 | image shipped to lax, mia, sjc |
| 04:36:14 | node-1 upgraded and caught up (139 s) |
| 04:38 | node-3 upgraded (43 s) |
| 04:39 | node-4 upgraded (32 s) |
| 04:40 | primary node-2 upgraded (42 s); record updated at 04:40:33 |
| 04:41:19 | 4/4 HEALTHY on `mr-2478e9c`, view 190, height 147937 |
| 04:43:23 | soak clock starts (`started_at` in heartbeat) |
| 04:44:01 | soak log: start on cain-mr-03, 48 h |
| 04:45:00 | checkpoint-0000 published, height 148069, **VALID** (48 certificates) |
| 05:00:21 | transient DEGRADED (view change in progress); next sample HEALTHY |
| 05:03:37 | first injected fault: node-1 killed |
| 05:04:34 | node-1 restarted; 5 expected check errors, 0 unexpected |
| 05:06:49 | 4/4 HEALTHY, view 203, height 149337, identical state |
| 05:09:28–05:10:27 | four consecutive `ReadTimeout` check errors during a view change; `max_unexpected_check_streak=4` (limit 5) |
| 05:13:07 | stable at view 207, all four at the same height |
| 05:16:33 | `cain_mr_restore_validate.py --publish` reniced to 19 / `ionice` class 3 (not stopped) |

The monitor's `/var/log/cain-cluster-health.log` shows `SAFE` at 05:00:08 (the 15-minute
sampler) and the gateway returned HEALTHY at 05:01:07; the 05:00:21 DEGRADED was a view-change
snapshot.
