How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
Experiment: EXP-SURG-002 · Dream curriculum sandbox
Date frozen: 2026-07-24 · Phase 1 closed: 2026-07-24
Prerequisite: Phase 1 + selection ablationselection_ablation_v0.1· H3 v0.2 ρ=0.145 FAIL
Code:exp_surg_002_dream_curriculum· Config:sandbox_v0.2.yaml
Run protocol:study2_phase2_run_protocol_v0.1.md
| Finding | Implication |
|---|---|
| Primary yield FAIL (both dreamers 5/5 top-k) | Do not re-run top-k yield comparison |
| Selection ablation Tier B PASS (top 1.0 · bottom 0.8) | Mock tier has signal; per-spec rank weak |
| H3 v0.2 ρ=0.145 FAIL | Mock rank is not a strong continuous predictor |
| Diffusion bottom: mock 0/5 informative · Isaac 5/5 | Root cause: Phase 1 Isaac arm used shift+onset only; mock used occlusion_gain |
| Gaussian per-dreamer ρ=0.67 · diffusion ρ=null (zero variance) | Alignment may recover H3 for shift-sensitive dreamer first |
Phase 2 question (pivot):
After mock–Isaac perturbation alignment, does mock informative rank predict Isaac informative rank on a top+bottom spec pack (Spearman ρ ≥ 0.5)?
Not asking again: diffusion beats Gaussian on yield (Phase 1 closed · coverage vs diversity).
| In scope | Out of scope |
|---|---|
| Mock–Isaac occlusion alignment (study1d proxy) | Classifier-guided diffusion (stretch · Phase 2b backlog) |
| Selection ablation rerun on new mock records | Phase 1 top-k confirmatory rerun |
| Harder perturbation cell (wider shift range) | ManiSkill / new sim |
| Tier + ρ pass criteria (Tier B/C) | Predicate invention / IVNTR |
CPU mock mock_smoke_v0.4 before GPU |
Extra mock yield episodes at v0.1 config |
Paper 1 (001 Phase C) remains separate.
On a 20-spec top+bottom pack exported from fresh mock records, mock per-spec informative rank correlates with Isaac informative rank with Spearman ρ ≥ 0.5 (pooled or both dreamers individually).
Informative @ spec (unchanged): CONTINUE unsuccessful ∧ REPLAN successful (same-state CF @ S).
| ID | Hypothesis | Pass |
|---|---|---|
| H4′ | Tier separation on Isaac: bottom.informative_rate ≤ 0.6 and < top |
Tier B direction (stricter floor than Phase 1) |
| H5′ | Alignment lift: ρ(v0.4 aligned) > ρ(v0.2 unaligned) by ≥ 0.25 | Descriptive · not registry if H3′ fails |
| H6 (exploratory) | Gaussian vs diffusion tier gap (top−bottom IR) differs | Coverage vs diversity · secondary narrative |
| Parameter | Phase 1 (v0.1) | Phase 2 (v0.2) |
|---|---|---|
| Isaac runner | orbit_reach_study1a · shift + onset |
orbit_reach_study1d · shift + onset + visibility_fraction |
| Occlusion map | ignored on Isaac | visibility_fraction = max(0.05, 1.0 − occlusion_gain) |
| Occlusion level | — | L1 · gain_scale_flag_v0.1 (Paper 001 contract) |
| Dream space shift | [0.02, 0.08] | [0.015, 0.10] (harder negatives) |
| Mock records | mock_smoke_v0.2 (reuse) |
mock_smoke_v0.4 (new · 3 seeds · same episodes) |
| Isaac export | top_bottom · 20 specs | same strategy · new records |
| Seeds / spec | 0–4 (n=5) | unchanged |
| Branches | CONTINUE · REPLAN_d0 | unchanged |
| Platform | RunPod 4090 · VESSL A100 | VESSL preferred (workspace kept · Jupyter) |
Dream space (v0.2): shift_m ∈ [0.015, 0.10] · onset_step ∈ [10, 40] · occlusion_gain ∈ [0, 0.85]
| Leg | Label | Where | Gate |
|---|---|---|---|
| 0 | Code align | PR: study2 Isaac loop → study1d + occlusion map | unit smoke |
| 1 | Mock v0.4 | CPU · --compare × seeds 42,43,44 |
records committed |
| 2 | Export | top-5 + bottom-5 / dreamer → 20 specs | isaac_specs_v0.4.json |
| 3 | Isaac ablation | VESSL · zero_agent smoke → full 20 spec | selection_ablation_v0.2/ |
| 4 | H3′ | CPU · compute_study2_h3_mock_isaac.py |
h3_mock_isaac_v0.4/ |
Do not run Leg 3 before Leg 0–2 complete.
Phase 2 GO (publishable H3′ story):
reason: ok), andPartial (Tier B only):
FAIL (stop Phase 2 GPU spiral):
RUN_ID: 20260724T053134Z · VESSL A100 · study1d + occlusion align · mock mock_smoke_v0.4
| Leg | Result |
|---|---|
| H3′ | PASS — Spearman ρ = 0.899 (reason: ok, n=20) |
| H4′ | PASS — top IR 1.0 · bottom 0.3 |
| H5′ | PASS — lift vs v0.2 = 0.899 − 0.145 = 0.754 |
| by_dreamer | Gaussian 0.8 · diffusion 0.5 (ceiling broken on diffusion) |
Narrative: Mock–Isaac occlusion alignment was the missing edge. Phase 1 per-spec rank failure (ρ=0.15) reversed after study1d transfer; tier filter and rank correlation supported at Tier B.
Promoted: selection_ablation_v0.2/ · h3_mock_isaac_v0.4/
| Metric | Definition |
|---|---|
| informative_rate | #(CONTINUE fail ∧ REPLAN ok) / #specs |
| mock–isaac agree | Spearman ρ on per-spec informative (0/1) |
| param_diversity | mean std(shift, onset, occlusion) across specs |
| alignment_lift | ρ(v0.4) − ρ(v0.2) = 0.145 baseline |
| Phase | Est. |
|---|---|
| Mock v0.4 (CPU) | < 15 min |
| Bootstrap (cold VESSL) | 15–25 min once |
| Isaac 20 specs × 5 seeds × 2 branches | ~2–3 h |
| H3′ recompute | < 5 min CPU |
| Total | ~0.5 workspace-day |
| Version | Date | Note |
|---|---|---|
| v0.1 | 2026-07-24 | Design freeze · mock–Isaac alignment + ablation rerun |
| v0.1-exec | 2026-07-24 | Executed · H3′ ρ=0.899 PASS · H4′ bottom=0.3 PASS |
study2_phase1_design_v0.1.mdselection_ablation_run_protocol_v0.1.mdvessl_isaac_setup_v0.1.mdstudy1d_occlusion_proxy_v0.1.md