How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
Public benchmark spec for Mismatch Lab · aligned with Paper 002 H4
Evaluate whether a system can distinguish:
This is a classification + ranking benchmark — not end-task success alone.
| Task ID | Condition | Expected adequacy hint | Gate should fire |
|---|---|---|---|
| M0-static | No drift · static target | none |
No |
| N1-noise | Observation noise · static dynamics | none or watch |
No |
| N2-impulse | Single target impulse | replan |
No / low |
| M1-drift-x | Persistent drift +x | structural_review |
Yes |
| M1-drift-y | Persistent drift +y (held-out dir) | structural_review |
Yes |
| M1-drift-diag | Diagonal drift (Ep2 novelty) | structural_review |
Yes |
Required fields in RunBundle:
{
"trajectory": {"obs": "T×D float", "actions": "T×A float"},
"predictions": {"one_step": "T×D float", "optional_multi_step": "..."},
"recovery_attempts": [{"step": "int", "residual_after": "float"}],
"metadata": {"task": "string", "seed": "int", "condition_id": "string"}
}
| Metric | Definition |
|---|---|
| H4 accuracy | Fraction of episodes where adequacy_hint matches gold label |
| False expansion rate | P(structural_review | N1 or N2 or M0) |
| Miss rate | P(none or watch | M1) |
| Metric | Definition |
|---|---|
| Onset error | |predicted_onset − true_onset| steps |
| Repair sufficiency AUC | Predict repair_likely_sufficient vs gold from repair curve |
| Calibration | Brier score on adequacy_score (when labeled) |
Derived from EXP-SURG-003 protocol:
DriftSpec.mode + velocity directionConfirmatory tier adds Isaac traces with independent human review sample (n≥20).
| Baseline | Description |
|---|---|
| Threshold-only | Fire if mean prediction error > τ |
| Repair-count | Fire if K repairs fail (no autocorr · no ΔNLL) |
| Paper 002 gate | Full rule-based gate (pre-registered) |
| Oracle | Privileged mode label (upper bound · not deployable) |
| Asset | Format | Source |
|---|---|---|
| 4 explorer cases | JSON | Mock pilot export |
| 5-seed pilot bundle | JSON | exp_surg_003 results |
| Gate control episodes | JSON | H4 negative controls |
Full confirmatory bundle after prereg lock.
Benchmark tasks map 1:1 to Diff Explorer cases:
structural_reviewUpdated 2026-07-29