After the Spill

How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.

View the Project on GitHub higuseonhye/research-os

Mismatch Lab v0.1 — Product Spec

Target: 4–6 weeks · public lab + evaluator prototype
North star: one working Diff case end-to-end before homepage polish


1. Product definition

Public name

Mismatch Lab

First feature

Robot Diff — compare two robot runs; auto-highlight differences.

Internal engine

Difference → Mismatch → Adequacy → Revision (research)

Users never see "failure lab." They see Understand · Compare · Explain · Discover.

Success criterion (v0.1)

A visitor can open Case 3 (Repair plateau), scrub the timeline, and answer:

  1. Why did repair fail?
  2. What evidence suggests structural mismatch?
  3. Did expansion regress static behavior?

Without reading Paper 002.


2. Architecture

┌─────────────────────────────────────────────────────────┐
│  Public Lab (Mismatch Lab)                               │
│  · Diff Explorer (web)                                   │
│  · 4 preloaded cases                                     │
│  · Benchmark spec (public)                               │
│  · Pilot waitlist                                        │
└───────────────────────┬─────────────────────────────────┘
                        │
┌───────────────────────▼─────────────────────────────────┐
│  Evaluation SDK (Python · v0.1 prototype)                │
│  mismatch.diff() · mismatch.analyze()                    │
└───────────────────────┬─────────────────────────────────┘
                        │
┌───────────────────────▼─────────────────────────────────┐
│  Research OS (research-os repo)                            │
│  EXP-SURG-003 · Paper 002 · prereg · mock pilot results  │
└─────────────────────────────────────────────────────────┘

3. Four visual cases (preloaded)

All cases export from EXP-SURG-003 mock env (2D reach + drift) unless noted.

Case 1 — Policy A vs B (no adequacy)

Field Value
ID policy_ablation
User question "What changed after my policy / hyperparameter change?"
Setup Same drift spec · two behavior seeds or MPC vs scripted
Expected diff Behavior shift mid-episode · prediction error similar
Adequacy hint none — difference is policy, not model class
CTA Compare intervention timing

Case 2 — Sim baseline vs drift onset (mismatch grows)

Field Value
ID drift_onset
User question "When did the world stop matching predictions?"
Setup Static pre-onset · drift velocity at step 8
Expected diff Prediction–observation gap opens at onset · autocorr rises
Adequacy hint watch — sustained divergence starting t=8
CTA Pin interesting moment

Case 3 — Repair plateau (adequacy escalates) ★ hero case

Field Value
ID repair_plateau
User question "Why didn't retuning fix it?"
Setup Ep1 drift · K=3 L1 repairs · residuals plateau above τ
Expected diff Repair residual curve flat · gate reasons light up
Adequacy hint structural_review — repair_likely_sufficient=false
CTA Show expansion vs repair PE on Ep2

Case 4 — Drift vs noise (H4 education)

Field Value
ID drift_vs_noise
User question "Is this sensor noise or real dynamics change?"
Setup Side-by-side: NOISE spec vs DRIFT spec
Expected diff Noise: high autocorr, low sustained residual · Drift: all gate signals
Adequacy hint Drift only → structural_review
CTA Open benchmark task

★ Case 3 is the hero demo for homepage embed and product overview.


4. Diff UI wireframe

See interactive wireframe: diff_explorer_v0.1.html (static · GitHub Pages)

Layout (desktop)

┌──────────────────────────────────────────────────────────────────┐
│ Mismatch Lab          [Case ▼]  [Run A ▼]  vs  [Run B ▼]        │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────────────────┐  ┌──────────────────────────┐ │
│  │ 2D reach view               │  │ Diff summary             │ │
│  │ · EE trace A (solid)        │  │ Max divergence: 0.041    │ │
│  │ · EE trace B (dashed)       │  │ Onset step: 8            │ │
│  │ · Target true / obs         │  │ Shifts found: 3          │ │
│  │ · Prediction overlay (opt)  │  │ Classification: drift    │ │
│  └─────────────────────────────┘  │ [Adequacy hint card]     │ │
│                                    └──────────────────────────┘ │
│  ┌──────────────────────────────────────────────────────────────┐│
│  │ Timeline slider  ●──────────■──────────■──────────────○  t   ││
│  │ ▲ interesting moments (pins)                                   ││
│  └──────────────────────────────────────────────────────────────┘│
│  ┌──────────────────────────────────────────────────────────────┐│
│  │ Charts (stacked)                                             ││
│  │ · Distance to target                                       ││
│  │ · Prediction error (1-step)                                  ││
│  │ · Repair residual (Case 3 only)                              ││
│  │ · Difference magnitude |A−B|                                ││
│  └──────────────────────────────────────────────────────────────┘│
├──────────────────────────────────────────────────────────────────┤
│ Actions: [Continue] [Repair] [Replan] [Expand] [Escalate]        │
│          (counterfactual preview · not live control in v0.1)     │
└──────────────────────────────────────────────────────────────────┘

Mobile

Interaction spec

Interaction Behavior
Timeline scrub Sync all charts + 2D marker
Pin click Jump to step · show Explain panel
Run A/B swap Recompute diff
Adequacy card expand Evidence bullets + link to Paper 002
Action buttons Show expected outcome tooltip (simulated, not executed)

5. API schema (summary)

Full JSON Schema: api_schema_v0.1.json

mismatch.diff

report = mismatch.diff(
    run_a=RunBundle(trajectory, predictions, metadata),
    run_b=RunBundle(trajectory, predictions, metadata),
    align="time",  # time | event
    options={"include_adequacy_hint": True},
)

mismatch.analyze (single-run adequacy)

report = mismatch.analyze(
    trajectory=rollout,
    predictions=model_predictions,
    recovery_attempts=attempts,
    context=task_metadata,
)

Report fields (user-facing)

Field Type Description
diff_id string UUID
classification enum policy_shift · transient_noise · persistent_drift · impulse · unknown
max_divergence float Peak prediction–obs or A–B diff
onset_step int First step above threshold
interesting_moments list {step, label, severity}
adequacy_hint enum none · watch · repair · replan · structural_review · escalate
adequacy_score float 0–1 (internal calibration · pilot only)
repair_likely_sufficient bool
recommended_next_test string e.g. add_drift_mode_counterfactual
evidence list[str] Human-readable bullets
confidence enum low · moderate · high

6. Python evaluator prototype (v0.1 scope)

Path (proposed): scripts/mismatch_lab/evaluator.py

Module Responsibility
io.py Load RunBundle from JSON / NPZ (mock pilot export)
align.py Time-align two trajectories
diff.py Difference metrics · interesting moments
adequacy.py Wrap wm_expansion.gate + repair curve
report.py Emit JSON matching api_schema

Not in v0.1: upload server · auth · billing · live robot control

CLI:

python scripts/mismatch_lab/evaluator.py diff \
  --run-a cases/repair_plateau/run_a.json \
  --run-b cases/repair_plateau/run_b.json \
  --out report.json

7. Build plan (4–6 weeks)

Week Deliverable
1 Export 4 cases from mock pilot · JSON schema · evaluator.py stub
2 diff() + analyze() on Case 3 · unit tests vs pilot summary
3 Static Diff Explorer (HTML/React or canvas) · Case 3 interactive
4 Cases 1–2–4 · benchmark spec page · homepage copy live on GitHub Pages
5 Pilot waitlist · API docs · Research OS cross-links
6 Buffer · product overview polish · first external demo

Definition of done


8. Public surfaces (this repository)

Surface URL (proposed) Content
Research OS github.com/.../research-os Papers · prereg · protocols · promoted results
Mismatch Lab /mismatch_lab on Pages Diff · benchmark · pilot · SDK sketch

Do not replace Research OS with a product pitch. Commercial hosting and customer operations stay out of research-os. Add Mismatch Lab as a sibling entry in docs/index.md.


9. Metrics (v0.1 — internal only)

Metric Target
Case 3 time-to-insight < 2 min without docs
Pilot signups 20 qualified (8 weeks post-launch)
API prototype adopters 3 teams run evaluator on own JSON

Do not publish mock pilot % improvements on homepage until confirmatory CI defined.


Updated 2026-07-29