How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
ARCHIVED · mock→physics direction · superseded 2026-07-29 · do not cite or extend Current Paper 002: WM expansion · archive index
Positioning: mock-to-physics validation · not LLM curriculum · not policy training
Program: Exception-aware Physical AI · front-end of recoverability pipeline
Pre-reg: paper002_prereg_v0.3.md
Can Cheap Proxy Evaluations Identify Informative Physical-AI Failures? A Same-State Counterfactual Validation Study in Robotic Simulation
From Mock Ranking to Physical Informativeness: Validating Rule- and LLM-Generated Occlusion Curricula
Paper 002 evaluates whether a low-cost mock evaluator can rank candidate occlusion mismatches according to their informativeness under high-fidelity Isaac simulation.
Starting from a fixed robot and world state, each candidate is evaluated through the same CONTINUE-versus-REPLAN counterfactual fork used in Paper 001. The study tests whether mock rankings transfer to Isaac-derived informativeness, whether top-ranked candidates enrich for informative failures, and whether an LLM-based goal planner performs at least as well as a deterministic rule-based planner.
This is a validation and selection study. It does not claim that generated curricula improve policy learning, autonomous recovery, or real-world surgical performance.
Robotic policies are commonly evaluated under researcher-selected perturbations, yet manually chosen failures may be redundant, trivial, or uninformative about the agent's capacity to recover. Automatic curriculum and environment-generation methods expand the space of candidate failures, but evaluating every candidate in high-fidelity physics simulation is computationally expensive.
We study whether a low-cost mock evaluator can identify candidate mismatches that remain informative under high-fidelity robotic simulation. Under a frozen occlusion contract, rule-based and LLM-based planners generate candidate perturbations. Each candidate receives a continuous mock informativeness score and is ranked before physics evaluation. A frozen transfer function maps mock occlusion parameters to Isaac simulation parameters. Starting from the same saved world and robot state, we execute paired CONTINUE and REPLAN responses over eight simulation seeds and label a candidate informative when a prespecified majority of valid seeds exhibits a task-relevant counterfactual difference.
The primary analysis measures rank transfer between mock and Isaac evaluations on the frozen export set. Secondary analyses test top-tier enrichment and LLM planner adequacy (absolute floor + non-inferiority vs rule). The protocol uses hierarchical hypothesis testing, bootstrap confidence intervals, permutation controls, and prespecified exclusion rules.
Prior work uses generated environments to train agents; we validate whether a low-cost proxy can identify which generated physical mismatches will expose reliable same-state differences between recovery responses before expensive physics execution.
Paper 002 is the front-end of the recoverability program — it selects which mismatches are worth measuring, not how to recover from them.
World → Mismatch
↑
Paper 002 — which mismatch to evaluate? (informative selection)
↓
Paper 001 — how do responses compare @ S? (recoverability ruler)
↓
Paper 003 — how to recover better? (policy / agent)
↓
Paper 004 — when to revise world model?
| Paper | Question |
|---|---|
| Paper 001 | What should AI do after mismatch? (measure recoverability @ S) |
| Paper 002 | Which mismatch should we evaluate? (proxy validation) |
| Paper 003 | How can AI learn to recover better? |
| Paper 004 | When should AI revise its world model? |
| Keyword | Role in Paper 002 |
|---|---|
| Recoverability | Select mismatches that reveal response differences Paper 001 measures |
| Counterfactual | CONTINUE vs REPLAN fork · same S |
| Exception-aware Physical AI | Find informative exceptions, not average-world success |
| World model | Mismatches that most stress the WM — selection, not update |
| Active learning / experiment design | Select experiments, not training samples |
| Evaluation curriculum | Not training curriculum — which failures to run in sim |
| Decision timing | Occlusion onset / persistence as perturbation params |
| Scientific discovery | Which experiment yields most information per GPU hour? |
May claim:
May not claim:
| Doc | Content |
|---|---|
| paper002_related_work_v0.1.md | POET · PLR · GenSim · differentiation |
| paper002_manuscript_pre_results_v0.1.md | Full methods · hypotheses · tables · interpretation |
| paper002_rq_v0.3.md | Operational RQs |
| paper002_prereg_v0.3.md | Frozen confirmatory design |