# E113 preregistration

Written and committed before any model output exists for the sealed or shift split (2026-11-19). The
selection numbers derived from dev/val are appended in `PREREGISTRATION_SELECTIONS.json` before the sealed
pool is generated; the sealed pool runs only after that commit.

## Question

Does a heterogeneous, memory-bearing, skill-using, governed population of model-backed agents beat the
strongest individual agent under matched compute, information, tools and evaluation?

## Agents and compute (what is real)

- Real local inference only: Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen2.5-1.5B-Instruct,
  HuggingFaceTB/SmolLM2-135M-Instruct (transformers, CPU, 4 threads). Cloud providers: BLOCKED (the session
  was not permitted to use the configured cloud key).
- Specialist agent = (model, mode) for one vertical. Modes: `direct` (answer only), `tool` (one Python
  expression evaluated in a bubblewrap sandbox; no network, read-only filesystem, 2 s CPU), `memory`
  (two retrieved verified dev episodes as context), `sample` (temperature 0.7).
- Identity count (42 specialist identities), model instances (3), OS processes (1 inference process at a
  time) and model calls are reported separately.
- Every model call and every tool execution needs a single-use Ed25519 permit (Authority decides,
  Resource admits) and produces a signed, hash-chained receipt.

## Tasks

Six verticals (software code tracing, security policy decision, mathematics, science causal
identification, data dedupe/aggregation, business constrained choice). Splits from disjoint seeds with no
shared question text: dev 60, val 60, sealed 180, shift 60 (templates not seen in other splits). About
10% of tasks have no reliable answer; the gold is UNKNOWN and abstention is the correct answer.

Split hashes (sha256 of canonical JSON):
- dev    b1eafc57eb87439056c2c9fefe9021526ee20389f7bdf887b52b8537f9778007
- val    bd7fd9d33c6750ee49e9156180727b7fec104468904ff00d63a3d8bcf6da72dd
- sealed db01f037df26e905d1c3a5a525d113ebc186d4560ce734b820621bb4137c155e
- shift  4a4ca4bf0b36be639a8f9725fa33583444bd5586b2824bf0077de849e6e91739

## Arms (all are policies over the same governed pool)

- A  strongest individual: the single (model, mode) with the best dev+val accuracy, one call.
- A* strongest individual with self-consistency: k samples of A's (model, mode) at temperature 0.7,
  majority vote; k is the smallest integer whose mean val CPU cost is >= the mean val CPU cost of G.
  **This is the primary comparator.**
- B  q05 + memory; B0 q05 without memory (same model, same task).
- C  q05 + skills: per-vertical procedure (direct/tool/memory) learned on dev, kept only if it does not
  lose on val, otherwise rolled back to direct.
- D  static routing by vertical (dev accuracy); E routing including the memory specialist (dev+val);
  F evidence routing (Wilson lower bound).
- G  full collective: top-3 specialists per vertical by validated reliability; evidence-based synthesis
  (weight = validated log-odds of the specialist's record; failed tool runs carry no evidence; self-reported
  confidence is never used).
- H  transfer: run as a separate study (see below), not as a sealed arm.
- I  self-improvement: 12 candidate changes to G's routing/synthesis, selected on val with reliability
  fit on dev only; the best is evaluated once on sealed against frozen G.
- J  ablations of G: no evidence weights (plain majority), no routing, no memory, no tool evidence.
- Population baselines: redundant voting (q05 x4), uncoordinated random specialist.

## Primary endpoint

Sealed accuracy, G vs A*, paired exact two-sided McNemar, alpha 0.05, with a paired bootstrap 95% CI of
the difference. G6 PASS requires G > A* with p < 0.05 and G's mean measured CPU <= A*'s.

## Secondary endpoints (no multiplicity correction claimed; reported as exploratory)

- G vs A (single call); G vs D; G vs redundant voting; shift-split replication of G vs A*.
- G4 memory: B vs B0 on sealed, McNemar.
- G5 skills: C vs B0 on sealed, McNemar (skills selected on dev/val only).
- G7 transfer: six directed pathways (source memory used on destination tasks) vs no-transfer (B0) and
  random-source transfer; a pathway is admitted only if it does not lose to no-transfer on val; admitted
  pathways tested on sealed (McNemar each). PASS needs >= 4 supported pathways across >= 3 destinations.
- G9 metacognition: ECE, Brier and AUROC of G's synthesis confidence; abstention precision/recall on
  unanswerable tasks.
- G10 active research: estimating the best specialist per vertical from a label budget; active (UCB)
  vs random vs fixed schedule, 20 seeds, scored on sealed.
- G11 self-improvement: best val candidate vs frozen G on sealed (McNemar) and no loss on shift.
- G12 Byzantine: injected adversarial candidates without receipts, without track record, and a
  compromised reputable member; measure accepted wrong answers.

## Stopping and reporting rules

- The sealed pool is generated once. No re-running of sealed calls to change a result.
- Every gate is PASS / PARTIAL / FAIL / BLOCKED / NOT TESTED. A negative result is published.
- 24-hour mission, operational independence and independent reproduction cannot be met on this host in
  this session and are declared BLOCKED / NOT TESTED in advance.

## Addendum A — numeric gate thresholds (committed before the sealed pool)

- G4 PASS: B > B0 on sealed, McNemar p < 0.05. Otherwise PARTIAL if B > B0, FAIL if B <= B0.
- G5 PASS: C > B0 on sealed, McNemar p < 0.05 (C differs from B0 in at least one vertical). Otherwise
  PARTIAL if C > B0, FAIL otherwise.
- G6 PASS: G > A* on sealed, McNemar p < 0.05, and mean CPU(G) <= mean CPU(A*). PARTIAL if G > A* without
  significance; FAIL if G <= A*.
- G7 PASS: >= 4 admitted pathways each beating no-transfer on sealed with p < 0.05, across >= 3
  destinations. PARTIAL if >= 1 pathway is positive (any p); FAIL otherwise.
- G8 PASS: G > A* on the shift split, McNemar p < 0.05. PARTIAL if G > A*; FAIL otherwise.
- G9 PASS: ECE <= 0.10 and AUROC >= 0.70 for G's synthesis confidence on sealed, and abstention recall
  >= 0.5 with precision >= 0.5 on unanswerable sealed tasks. PARTIAL if two of the four hold.
- G10 PASS: at 8 labels per vertical, active selection's mean sealed accuracy of the chosen specialists
  beats both random and fixed schedules, sign test over 20 seeds p < 0.05. PARTIAL if it beats both in
  mean only.
- G11 PASS: the val-selected candidate beats frozen G on sealed (McNemar p < 0.05) and does not lose on
  shift. If no candidate beats frozen G on val, the baseline is retained (FAIL, negative result).
- G12 PASS: zero tasks where unreceipted forged candidates or receipted newcomers without a track
  record change G's answer, and with one compromised reputable member G's accepted-adversary rate is
  lower than the routed single agent's. PARTIAL if only the first part holds.

## Addendum B — scope reduction for speed (committed before the sealed pool)

To finish sooner on a shared CPU host, the sealed pool contains only the configurations that some
preregistered arm or study consumes (A, A*, B, B0, C, D, E, F, G, the J ablations, redundant voting),
computed mechanically from the frozen selections (`study.needed_configs`). The shift pool contains only
A, A* and G's configurations, because G8 compares G with A*. The uncoordinated baseline draws among the
specialists that have a row for the task; the active-research study scores only specialists that have a
sealed row for every sealed task. If the self-improvement candidate chosen on val needs a configuration
that was not generated, its rows are generated once after the choice (the choice is made on val, so the
added rows cannot influence it). Nothing else changes; the primary endpoint and all thresholds stand.
The A* and G rows are generated first so the primary endpoint completes before secondary studies.
