How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
ARCHIVED · mock→physics direction · superseded 2026-07-29
Use instead: paper002_manuscript_wm_expansion_v0.1.md
Archive context: archive/mock_to_physics/README.md
Draft body before Isaac confirmatory run · fill T1–T6 after execution
Companion (historical): paper002_description_v0.1.md · paper002_prereg_v0.3.md
Physical-AI systems are developed under a limited collection of disturbances selected by researchers. Such disturbances may show that a system can fail, but not whether the failure is useful for comparing alternative responses or constructing subsequent experiments.
Informative mismatch: produces a reliable, task-relevant difference between two prespecified responses from the same initial state — not merely task difficulty.
Paper 001 established the same-state counterfactual fork @ S:
S ──mismatch──► CONTINUE vs REPLAN@0
Paper 002 addresses the selection problem: given many candidate mismatches, can a low-cost approximation identify which are likely informative in physics?
Three questions often conflated:
Primary contribution: questions 2 and 3. Generation is an experimental factor, not sufficient evidence of curriculum quality.
Under a frozen occlusion contract, does continuous mock-based ranking predict same-state counterfactual informativeness in Isaac simulation on the frozen export set — and does an LLM goal planner meet prespecified adequacy criteria relative to a rule planner?
| # | Claim | Hypothesis |
|---|---|---|
| 1 | Proxy validity | H1 — rank transfer |
| 2 | Selection utility | H2 — top-tier enrichment |
| 3 | Planner adequacy | H3 — LLM floor + non-inferiority |
Does not test whether either planner improves a learned policy.
Spearman ρ between continuous aggregated mock score (\bar M_i) and continuous Isaac score (I_i) on the 40-candidate export set (20 rule + 20 LLM):
Primary enrichment decision (H2 pass requires all):
Supporting inference: one-sided exact binomial (H_0: p_{top} \leq 0.50). Report exact p. Not an additional pass requirement. For n=10, 8/10 → p = 0.0547 (borderline); 9/10 → p = 0.0107.
Evaluated post-hoc · hierarchical after H1 and H2 on rule set.
Both required for LLM claim.
Mixed factorial:
Export scope: within-planner — top 10 + bottom 10 pooled across dreamers per planner → 40 Isaac specs total.
Mock aggregation: median (\bar M_i) over mock seeds 42–46 (not seed 43 alone). Seed 43 = sensitivity only.
[ M_i = \underbrace{\mathbb{1}[REPLAN\ ok] - \mathbb{1}[CONTINUE\ ok]}_{\text{success gap}}
where (\tau) = mock tolerance (0.02 m). Implemented in mock_reach.cf_margin().
Binary informative label = secondary: ( \hat Y^{mock}_i = \mathbb{1}[M_i \geq \theta_M]) with (\theta_M = 0.5) frozen.
Spearman uses continuous (\bar M_i), not binary.
[ \bar M_i = \mathrm{median}{s \in {42,\ldots,46}} M{is} ]
Candidates matched by canonical spec key (round(shift_m,4), onset_step, round(occlusion_gain,3)).
| Mock field | Isaac field | Transfer |
|---|---|---|
occlusion_gain |
visibility_fraction |
clip(1−g, 0.05, 1) |
onset_step |
perturbation start | identical |
shift_m |
target shift (Y+) | direct |
tier |
metadata | no sim effect |
Runner: study1d · proxy: gain_scale_flag_v0.1 · level 1.
See paper002_analysis_plan_v0.2.md.
n=40 export set — suitable for detecting ρ ≳ 0.5, not fine-grained interactions. Non-inferiority for LLM is bounded confirmatory — failure to establish non-inferiority ≠ proven inferiority.
Without middle-ranked candidates, H1 is extreme-export-set rank transfer — manuscript must not claim global proxy validity over full candidate space.
| Planner | Raw | Valid | Unique | Exported | Isaac-valid | Confirmatory |
|---|---|---|---|---|---|---|
| Rule | 20 | |||||
| LLM | 20 | |||||
| Total | 40 |
| Population | n | Spearman ρ | Bootstrap 95% CI | Perm p | Pass |
|---|---|---|---|---|---|
| Rule export set | |||||
| LLM export set | |||||
| Pooled (descriptive) |
| Planner | Group | n | Informative | IR | Exact 95% CI |
|---|---|---|---|---|---|
| Rule | Top | 10 | |||
| Rule | Bottom | 10 | |||
| LLM | Top | 10 | |||
| LLM | Bottom | 10 |
| Hyp | Criterion | Estimate | Pass |
|---|---|---|---|
| H1 | ρ≥0.50 ∧ CI_lo>0.25 | ||
| H2a–c | IR_top · IR_bottom · gap | ||
| H2 binomial (support) | exact p one-sided | — | |
| H3a | ρ_LLM≥0.50 | ||
| H3b | CI_lo(Δρ)>−0.15 |
| Pattern | Statement |
|---|---|
| H1+H2 pass | Proxy-guided selection supported on export set |
| H1+H2+H3 pass | + LLM non-inferior proposer |
| H1 pass · H2 fail | Association without usable tier filter |
| H1 fail | No validated rank transfer under tested contract |
| H3a pass · H3b fail | LLM floor ok · non-inferiority not established |
Rule generated [N] contract-valid candidates; LLM [N] parsed / [N] valid. After five-seed median aggregation and within-planner export, 20 candidates per planner were frozen (checksum: [hash]) before Isaac.
Rule planner: ρ = [X], 95% CI [L, U]. H1 [pass/fail].
Rule top tier: [k]/10 informative (IR=[X]). Bottom: [k]/10. H2 [pass/fail].
ρ_LLM = [X], Δρ = [X−Y], 95% CI for Δρ [L, U]. H3 [pass/fail].