How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
Target: 4–6 weeks · public lab + evaluator prototype
North star: one working Diff case end-to-end before homepage polish
Mismatch Lab
Robot Diff — compare two robot runs; auto-highlight differences.
Difference → Mismatch → Adequacy → Revision (research)
Users never see "failure lab." They see Understand · Compare · Explain · Discover.
A visitor can open Case 3 (Repair plateau), scrub the timeline, and answer:
Without reading Paper 002.
┌─────────────────────────────────────────────────────────┐
│ Public Lab (Mismatch Lab) │
│ · Diff Explorer (web) │
│ · 4 preloaded cases │
│ · Benchmark spec (public) │
│ · Pilot waitlist │
└───────────────────────┬─────────────────────────────────┘
│
┌───────────────────────▼─────────────────────────────────┐
│ Evaluation SDK (Python · v0.1 prototype) │
│ mismatch.diff() · mismatch.analyze() │
└───────────────────────┬─────────────────────────────────┘
│
┌───────────────────────▼─────────────────────────────────┐
│ Research OS (research-os repo) │
│ EXP-SURG-003 · Paper 002 · prereg · mock pilot results │
└─────────────────────────────────────────────────────────┘
All cases export from EXP-SURG-003 mock env (2D reach + drift) unless noted.
| Field | Value |
|---|---|
| ID | policy_ablation |
| User question | "What changed after my policy / hyperparameter change?" |
| Setup | Same drift spec · two behavior seeds or MPC vs scripted |
| Expected diff | Behavior shift mid-episode · prediction error similar |
| Adequacy hint | none — difference is policy, not model class |
| CTA | Compare intervention timing |
| Field | Value |
|---|---|
| ID | drift_onset |
| User question | "When did the world stop matching predictions?" |
| Setup | Static pre-onset · drift velocity at step 8 |
| Expected diff | Prediction–observation gap opens at onset · autocorr rises |
| Adequacy hint | watch — sustained divergence starting t=8 |
| CTA | Pin interesting moment |
| Field | Value |
|---|---|
| ID | repair_plateau |
| User question | "Why didn't retuning fix it?" |
| Setup | Ep1 drift · K=3 L1 repairs · residuals plateau above τ |
| Expected diff | Repair residual curve flat · gate reasons light up |
| Adequacy hint | structural_review — repair_likely_sufficient=false |
| CTA | Show expansion vs repair PE on Ep2 |
| Field | Value |
|---|---|
| ID | drift_vs_noise |
| User question | "Is this sensor noise or real dynamics change?" |
| Setup | Side-by-side: NOISE spec vs DRIFT spec |
| Expected diff | Noise: high autocorr, low sustained residual · Drift: all gate signals |
| Adequacy hint | Drift only → structural_review |
| CTA | Open benchmark task |
★ Case 3 is the hero demo for homepage embed and product overview.
See interactive wireframe: diff_explorer_v0.1.html (static · GitHub Pages)
┌──────────────────────────────────────────────────────────────────┐
│ Mismatch Lab [Case ▼] [Run A ▼] vs [Run B ▼] │
├──────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────┐ ┌──────────────────────────┐ │
│ │ 2D reach view │ │ Diff summary │ │
│ │ · EE trace A (solid) │ │ Max divergence: 0.041 │ │
│ │ · EE trace B (dashed) │ │ Onset step: 8 │ │
│ │ · Target true / obs │ │ Shifts found: 3 │ │
│ │ · Prediction overlay (opt) │ │ Classification: drift │ │
│ └─────────────────────────────┘ │ [Adequacy hint card] │ │
│ └──────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────────┐│
│ │ Timeline slider ●──────────■──────────■──────────────○ t ││
│ │ ▲ interesting moments (pins) ││
│ └──────────────────────────────────────────────────────────────┘│
│ ┌──────────────────────────────────────────────────────────────┐│
│ │ Charts (stacked) ││
│ │ · Distance to target ││
│ │ · Prediction error (1-step) ││
│ │ · Repair residual (Case 3 only) ││
│ │ · Difference magnitude |A−B| ││
│ └──────────────────────────────────────────────────────────────┘│
├──────────────────────────────────────────────────────────────────┤
│ Actions: [Continue] [Repair] [Replan] [Expand] [Escalate] │
│ (counterfactual preview · not live control in v0.1) │
└──────────────────────────────────────────────────────────────────┘
| Interaction | Behavior |
|---|---|
| Timeline scrub | Sync all charts + 2D marker |
| Pin click | Jump to step · show Explain panel |
| Run A/B swap | Recompute diff |
| Adequacy card expand | Evidence bullets + link to Paper 002 |
| Action buttons | Show expected outcome tooltip (simulated, not executed) |
Full JSON Schema: api_schema_v0.1.json
mismatch.diffreport = mismatch.diff(
run_a=RunBundle(trajectory, predictions, metadata),
run_b=RunBundle(trajectory, predictions, metadata),
align="time", # time | event
options={"include_adequacy_hint": True},
)
mismatch.analyze (single-run adequacy)report = mismatch.analyze(
trajectory=rollout,
predictions=model_predictions,
recovery_attempts=attempts,
context=task_metadata,
)
| Field | Type | Description |
|---|---|---|
diff_id |
string | UUID |
classification |
enum | policy_shift · transient_noise · persistent_drift · impulse · unknown |
max_divergence |
float | Peak prediction–obs or A–B diff |
onset_step |
int | First step above threshold |
interesting_moments |
list | {step, label, severity} |
adequacy_hint |
enum | none · watch · repair · replan · structural_review · escalate |
adequacy_score |
float | 0–1 (internal calibration · pilot only) |
repair_likely_sufficient |
bool | |
recommended_next_test |
string | e.g. add_drift_mode_counterfactual |
evidence |
list[str] | Human-readable bullets |
confidence |
enum | low · moderate · high |
Path (proposed): scripts/mismatch_lab/evaluator.py
| Module | Responsibility |
|---|---|
io.py |
Load RunBundle from JSON / NPZ (mock pilot export) |
align.py |
Time-align two trajectories |
diff.py |
Difference metrics · interesting moments |
adequacy.py |
Wrap wm_expansion.gate + repair curve |
report.py |
Emit JSON matching api_schema |
Not in v0.1: upload server · auth · billing · live robot control
CLI:
python scripts/mismatch_lab/evaluator.py diff \
--run-a cases/repair_plateau/run_a.json \
--run-b cases/repair_plateau/run_b.json \
--out report.json
| Week | Deliverable |
|---|---|
| 1 | Export 4 cases from mock pilot · JSON schema · evaluator.py stub |
| 2 | diff() + analyze() on Case 3 · unit tests vs pilot summary |
| 3 | Static Diff Explorer (HTML/React or canvas) · Case 3 interactive |
| 4 | Cases 1–2–4 · benchmark spec page · homepage copy live on GitHub Pages |
| 5 | Pilot waitlist · API docs · Research OS cross-links |
| 6 | Buffer · product overview polish · first external demo |
api_schema_v0.1.json| Surface | URL (proposed) | Content |
|---|---|---|
| Research OS | github.com/.../research-os |
Papers · prereg · protocols · promoted results |
| Mismatch Lab | /mismatch_lab on Pages |
Diff · benchmark · pilot · SDK sketch |
Do not replace Research OS with a product pitch. Commercial hosting and customer operations stay out of research-os. Add Mismatch Lab as a sibling entry in docs/index.md.
| Metric | Target |
|---|---|
| Case 3 time-to-insight | < 2 min without docs |
| Pilot signups | 20 qualified (8 weeks post-launch) |
| API prototype adopters | 3 teams run evaluator on own JSON |
Do not publish mock pilot % improvements on homepage until confirmatory CI defined.
Updated 2026-07-29