CAIN-42 CAIN StudioLiftGate: significance and sample size without a spreadsheet
Two-proportion z-test (pooled variance for the statistic, unpooled for the 95% interval) with a ship / keep / inconclusive call, plus the standard per-group sample size. Example: 10% → 12% at α 0.05 and power 0.8 needs 3,839 users per group, matching the closed form recomputed independently in today's test.
Today's change
A significance level of 5 made every result "significant", and identical rates came back keep_control.
An alpha outside (0, 0.5] is now refused (refused_invalid_alpha), never interpreted.
| Group | Case | Want | Run 1 | After |
|---|---|---|---|---|
| LiftGate | experiments route reachable | 200 | failed | needs a paid key through the proxy: not measurable live, measured on the engine |
| LiftGate | alpha = 5 with identical rates | refused, never ship | held | live reply was 401 (not a real hold); engine gave keep_control + significant; now refused_invalid_alpha |
| LiftGate | alpha 0, -0.5, 1.0, 1.5 | refused | failed | engine interpreted them; now refused_invalid_alpha |
| LiftGate | conversions above total | refused | failed | engine already refuses (400); live route needs a key |
| LiftGate | p-value vs independent reference (8 random) | |diff| < 1e-5 | failed | not measurable live (key); engine formula matches the closed form used here |
| LiftGate | sample size vs independent reference (4) | exact | held | held |
| LiftGate | alpha 0.10 / power 0.95 | clear answer or clear refusal, no 500 | held | held (clear refusal) |
FIRME AI CORPORATION · CAIN-42 is a Byzantine trust fabric. Numbers marked "our own check" were run by us, not by an independent party. Nothing on these pages is a certification.