How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
ARCHIVED · mock→physics direction · superseded 2026-07-29 · do not cite or extend Current Paper 002: WM expansion · archive index
Superseded by paper002_prereg_v0.3.md · not executed
Experiment: EXP-SURG-003 · confirmatory generative curriculum
Method: paper002_method_spec_v0.1.md
Analysis: paper002_analysis_plan_v0.1.md
RQ: paper002_rq_v0.2.md
Config:sandbox_v0.3.yaml
Pilot: Study 002 Phase 1–2 · Tier B ·../stage2/
Date frozen: 2026-07-28
| Parameter | Study 002 Phase 2 (pilot) | Paper 002 confirmatory |
|---|---|---|
| Mock episodes / dreamer | 48 | 128 |
| Mock seeds | 3 (42–44) | 5 (42–46) |
| Export | top-5 + bottom-5 | top-10 + bottom-10 |
| Specs / dreamer | 10 | 20 |
| Total Isaac specs / agent | 20 | 40 |
| Isaac seeds / spec | 5 | 8 (0–7) |
| Isaac informative @ spec | single-seed impl. | majority ≥ 5/8 seeds |
| Agents | rule only | rule · llm-json |
| Isaac runner | study1d | study1d · gain_scale_flag v0.1 |
| Primary export seed | 43 | 43 (sensitivity: 42, 44, 45, 46) |
Test H1 → H2 → H3 in order. Leg 6 (LLM Isaac) runs only if H1 and H2 PASS @ rule.
Pooled Spearman ρ(mock_informative, isaac_informative) ≥ 0.5 on the 40-spec top+bottom pack from
mock_confirmatory_v0.1/records_seed43.json.
Pass (all):
reason: ok · n ≥ 36 after exclusionsPilot anchor: Study 002 H3′ ρ=0.899 @ n=20 · confirmatory threshold lowered for replication conservatism.
On Isaac @ rule:
IR_bottom≤ 0.5 ANDtier_gap≥ 0.4 (pooled tiers · n=20 each).
Pass (all):
IR_top > 0.7 @ n=20 · p < 0.05Pilot anchor: H4′ top 1.0 · bottom 0.3 @ n=20.
LLM curriculum (prompt v0.1 · llm_curriculum_protocol_v0.1.md) satisfies both:
Kill (no LLM claim): H3a FAIL · or LLM ρ < 0.3
| ID | Analysis | Tier |
|---|---|---|
| S1 | Export seed sensitivity · ρ for seeds 42–46 · median · IQR | B |
| S2 | Gaussian vs diffusion · ρ · tier_gap · mock diversity | B |
| S3 | Permutation null for H1 · shuffle mock labels · 1000 draws | B |
| S4 | LLM mock yield vs rule · descriptive | B |
| Agent | Mode | Outputs |
|---|---|---|
| Rule | rule |
Taxonomy cycle goals |
| LLM | json-file |
Goal JSON per seed · dreamer draws params |
LLM model ID: recorded once in artifacts/llm_run_log_v0.1.json before mock Leg 3 · no mid-study swap.
| Metric | Definition |
|---|---|
| mock_informative | CONTINUE fail ∧ REPLAN ok · mock CF · deterministic per spec |
| isaac_informative | Majority ≥ 5/8 seeds: CONTINUE fail ∧ REPLAN_d0 ok |
| informative_rate (IR) | #informative / #specs in stratum |
| tier_gap | IR_top − IR_bottom |
| ρ | Spearman on per-spec binary flags |
| Case | Action |
|---|---|
| Isaac spec run error | Exclude spec · log in aggregate |
| < 4 valid seed pairs | Exclude spec |
| Dedupe collision at export | First in rank wins (frozen algorithm) |
Report n_analyzed · n_excluded in all summaries.
| Leg | Work | Gate |
|---|---|---|
| 0 | Commit v0.2 pre-reg + method + analysis | — |
| 1 | LLM curricula · 5 seeds · schema valid | — |
| 2 | Mock rule · 5 seeds · mock_confirmatory_v0.1 |
CPU |
| 3 | Mock LLM · 5 seeds · mock_confirmatory_llm_v0.1 |
CPU |
| 4 | Export · seed 43 · 40 specs / agent | CPU |
| 5 | Isaac rule ablation | GPU · zero_agent smoke |
| 6 | Isaac LLM ablation | GPU · H1∧H2 pass |
| 7 | Analysis per paper002_analysis_plan_v0.1.md | CPU |
| Outcome | Action |
|---|---|
| H1∧H2∧H3 pass | Tier A paper |
| H1∧H2 pass · H3 fail | Tier B · no LLM claim |
| H1 pass · H2 fail | Stop · tier filter unreliable |
| H1 fail | Stop · skip Leg 6 · negative doc |
One Isaac ablation per agent per frozen design. No retry on same cell without v0.3 pre-reg.
| Version | Date | Note |
|---|---|---|
| v0.1 | 2026-07-28 | Desk draft · superseded |
| v0.2 | 2026-07-28 | Method spec · hierarchical tests · Isaac majority · H3 non-inferiority |