How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
ARCHIVED · mock→physics direction · superseded 2026-07-29 · do not cite or extend Current Paper 002: WM expansion · archive index
Pre-reg: paper002_prereg_v0.2.md
Method: paper002_method_spec_v0.1.md
Date: 2026-07-28
Test in order · stop interpretation chain on failure:
H1 (ρ rank) → H2 (tier IR) → H3 (LLM vs rule)
│ │
└─ FAIL → no confirmatory claim · optional exploratory report
Multiplicity: no Bonferroni across H1–H3 (hierarchical gate). Secondary/exploratory: uncorrected · labeled Tier B.
Unit: one of 40 specs (pooled across dreamers).
| Variable | Source |
|---|---|
| Mock score | Binary mock_informative ∈ {0, 1} |
| Isaac score | Binary isaac_informative ∈ {0, 1} (majority over 8 seeds) |
Statistic: Spearman ρ (tie-aware rank correlation · implementation in compute_study2_h3_mock_isaac.py).
Pass criteria (confirmatory):
reason: ok · n ≥ 36 (after exclusions)Sanity (negative control · Tier B):
Per-dreamer (secondary): report Gaussian ρ · diffusion ρ · not registry pass/fail.
Strata: selection_tier ∈ {top, bottom} · pooled across dreamers (n=20 each).
| Metric | Definition |
|---|---|
IR_top |
#isaac_informative / n_top |
IR_bottom |
#isaac_informative / n_bottom |
tier_gap |
IR_top − IR_bottom |
Pass criteria (both required):
IR_bottom ≤ 0.5tier_gap ≥ 0.4IR_top > 0.7 with n=20 · p < 0.05 (H₀: p_top ≤ 0.7)Secondary: per-dreamer tier IR · Fisher exact top vs bottom 2×2.
Comparison: same metrics on LLM mock export + LLM Isaac ablation.
Pass criteria (both required for LLM claim):
reason: okDescriptive (not pass/fail):
tier_gap_llm vs tier_gap_ruleIR_bottom_llm vs IR_bottom_ruleKill (no LLM body claim): H3a FAIL · or H3b FAIL with LLM ρ < 0.3
| ID | Analysis |
|---|---|
| S1 | Export seed sensitivity: ρ for seeds 42–46 · report median · IQR |
| S2 | Gaussian vs diffusion ρ · tier_gap · param_diversity (mock) |
| S3 | Mock yield (informative_rate) by agent · descriptive |
| S4 | Hybrid dreamer — post-hoc only · not in confirmatory run |
| Table | Content |
|---|---|
| T1 | Design factors · n · pre-reg version |
| T2 | Per-spec mock vs Isaac flags (40 rows · rule) |
| T3 | H1 ρ · bootstrap CI · permutation p |
| T4 | H2 tier IR · binomial CI |
| T5 | Rule vs LLM summary (if Leg 6 run) |
| T6 | Pilot vs confirmatory comparison (Study 002 v0.4 · n=20) |
| Step | Script |
|---|---|
| Export | scripts/export_study2_isaac_specs.py |
| ρ + per-spec | scripts/compute_study2_h3_mock_isaac.py |
| Bootstrap CI | extend compute script or notebook · seed=20260728 |
All promoted JSON must include: prereg_version, git_sha, run_id, n_excluded.