# Mismatch Lab v0.1 — Product Spec

> **Target:** 4–6 weeks · public lab + evaluator prototype  
> **North star:** one working Diff case end-to-end before homepage polish

---

## 1. Product definition

### Public name

**Mismatch Lab**

### First feature

**Robot Diff** — compare two robot runs; auto-highlight differences.

### Internal engine

```text
Difference → Mismatch → Adequacy → Revision (research)
```

Users never see "failure lab." They see **Understand · Compare · Explain · Discover**.

### Success criterion (v0.1)

A visitor can open **Case 3 (Repair plateau)**, scrub the timeline, and answer:

1. Why did repair fail?
2. What evidence suggests structural mismatch?
3. Did expansion regress static behavior?

Without reading Paper 002.

---

## 2. Architecture

```text
┌─────────────────────────────────────────────────────────┐
│  Public Lab (Mismatch Lab)                               │
│  · Diff Explorer (web)                                   │
│  · 4 preloaded cases                                     │
│  · Benchmark spec (public)                               │
│  · Pilot waitlist                                        │
└───────────────────────┬─────────────────────────────────┘
                        │
┌───────────────────────▼─────────────────────────────────┐
│  Evaluation SDK (Python · v0.1 prototype)                │
│  mismatch.diff() · mismatch.analyze()                    │
└───────────────────────┬─────────────────────────────────┘
                        │
┌───────────────────────▼─────────────────────────────────┐
│  Research OS (research-os repo)                            │
│  EXP-SURG-003 · Paper 002 · prereg · mock pilot results  │
└─────────────────────────────────────────────────────────┘
```

---

## 3. Four visual cases (preloaded)

All cases export from EXP-SURG-003 mock env (2D reach + drift) unless noted.

### Case 1 — Policy A vs B (no adequacy)

| Field | Value |
| --- | --- |
| **ID** | `policy_ablation` |
| **User question** | "What changed after my policy / hyperparameter change?" |
| **Setup** | Same drift spec · two behavior seeds or MPC vs scripted |
| **Expected diff** | Behavior shift mid-episode · prediction error similar |
| **Adequacy hint** | `none` — difference is policy, not model class |
| **CTA** | Compare intervention timing |

### Case 2 — Sim baseline vs drift onset (mismatch grows)

| Field | Value |
| --- | --- |
| **ID** | `drift_onset` |
| **User question** | "When did the world stop matching predictions?" |
| **Setup** | Static pre-onset · drift velocity at step 8 |
| **Expected diff** | Prediction–observation gap opens at onset · autocorr rises |
| **Adequacy hint** | `watch` — sustained divergence starting t=8 |
| **CTA** | Pin interesting moment |

### Case 3 — Repair plateau (adequacy escalates) ★ hero case

| Field | Value |
| --- | --- |
| **ID** | `repair_plateau` |
| **User question** | "Why didn't retuning fix it?" |
| **Setup** | Ep1 drift · K=3 L1 repairs · residuals plateau above τ |
| **Expected diff** | Repair residual curve flat · gate reasons light up |
| **Adequacy hint** | `structural_review` — repair_likely_sufficient=false |
| **CTA** | Show expansion vs repair PE on Ep2 |

### Case 4 — Drift vs noise (H4 education)

| Field | Value |
| --- | --- |
| **ID** | `drift_vs_noise` |
| **User question** | "Is this sensor noise or real dynamics change?" |
| **Setup** | Side-by-side: NOISE spec vs DRIFT spec |
| **Expected diff** | Noise: high autocorr, low sustained residual · Drift: all gate signals |
| **Adequacy hint** | Drift only → `structural_review` |
| **CTA** | Open benchmark task

★ Case 3 is the hero demo for homepage embed and product overview.

---

## 4. Diff UI wireframe

See interactive wireframe: [diff_explorer_v0.1.html](diff_explorer_v0.1.html) (static · GitHub Pages)

### Layout (desktop)

```text
┌──────────────────────────────────────────────────────────────────┐
│ Mismatch Lab          [Case ▼]  [Run A ▼]  vs  [Run B ▼]        │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────────────────────┐  ┌──────────────────────────┐ │
│  │ 2D reach view               │  │ Diff summary             │ │
│  │ · EE trace A (solid)        │  │ Max divergence: 0.041    │ │
│  │ · EE trace B (dashed)       │  │ Onset step: 8            │ │
│  │ · Target true / obs         │  │ Shifts found: 3          │ │
│  │ · Prediction overlay (opt)  │  │ Classification: drift    │ │
│  └─────────────────────────────┘  │ [Adequacy hint card]     │ │
│                                    └──────────────────────────┘ │
│  ┌──────────────────────────────────────────────────────────────┐│
│  │ Timeline slider  ●──────────■──────────■──────────────○  t   ││
│  │ ▲ interesting moments (pins)                                   ││
│  └──────────────────────────────────────────────────────────────┘│
│  ┌──────────────────────────────────────────────────────────────┐│
│  │ Charts (stacked)                                             ││
│  │ · Distance to target                                       ││
│  │ · Prediction error (1-step)                                  ││
│  │ · Repair residual (Case 3 only)                              ││
│  │ · Difference magnitude |A−B|                                ││
│  └──────────────────────────────────────────────────────────────┘│
├──────────────────────────────────────────────────────────────────┤
│ Actions: [Continue] [Repair] [Replan] [Expand] [Escalate]        │
│          (counterfactual preview · not live control in v0.1)     │
└──────────────────────────────────────────────────────────────────┘
```

### Mobile

- Stack: summary → timeline → chart → 2D view
- Case picker as horizontal scroll chips

### Interaction spec

| Interaction | Behavior |
| --- | --- |
| Timeline scrub | Sync all charts + 2D marker |
| Pin click | Jump to step · show Explain panel |
| Run A/B swap | Recompute diff |
| Adequacy card expand | Evidence bullets + link to Paper 002 |
| Action buttons | Show **expected outcome** tooltip (simulated, not executed) |

---

## 5. API schema (summary)

Full JSON Schema: [api_schema_v0.1.json](api_schema_v0.1.json)

### `mismatch.diff`

```python
report = mismatch.diff(
    run_a=RunBundle(trajectory, predictions, metadata),
    run_b=RunBundle(trajectory, predictions, metadata),
    align="time",  # time | event
    options={"include_adequacy_hint": True},
)
```

### `mismatch.analyze` (single-run adequacy)

```python
report = mismatch.analyze(
    trajectory=rollout,
    predictions=model_predictions,
    recovery_attempts=attempts,
    context=task_metadata,
)
```

### Report fields (user-facing)

| Field | Type | Description |
| --- | --- | --- |
| `diff_id` | string | UUID |
| `classification` | enum | `policy_shift` · `transient_noise` · `persistent_drift` · `impulse` · `unknown` |
| `max_divergence` | float | Peak prediction–obs or A–B diff |
| `onset_step` | int | First step above threshold |
| `interesting_moments` | list | `{step, label, severity}` |
| `adequacy_hint` | enum | `none` · `watch` · `repair` · `replan` · `structural_review` · `escalate` |
| `adequacy_score` | float | 0–1 (internal calibration · pilot only) |
| `repair_likely_sufficient` | bool | |
| `recommended_next_test` | string | e.g. `add_drift_mode_counterfactual` |
| `evidence` | list[str] | Human-readable bullets |
| `confidence` | enum | `low` · `moderate` · `high` |

---

## 6. Python evaluator prototype (v0.1 scope)

**Path (proposed):** `scripts/mismatch_lab/evaluator.py`

| Module | Responsibility |
| --- | --- |
| `io.py` | Load RunBundle from JSON / NPZ (mock pilot export) |
| `align.py` | Time-align two trajectories |
| `diff.py` | Difference metrics · interesting moments |
| `adequacy.py` | Wrap `wm_expansion.gate` + repair curve |
| `report.py` | Emit JSON matching api_schema |

**Not in v0.1:** upload server · auth · billing · live robot control

**CLI:**

```bash
python scripts/mismatch_lab/evaluator.py diff \
  --run-a cases/repair_plateau/run_a.json \
  --run-b cases/repair_plateau/run_b.json \
  --out report.json
```

---

## 7. Build plan (4–6 weeks)

| Week | Deliverable |
| --- | --- |
| **1** | Export 4 cases from mock pilot · JSON schema · `evaluator.py` stub |
| **2** | `diff()` + `analyze()` on Case 3 · unit tests vs pilot summary |
| **3** | Static Diff Explorer (HTML/React or canvas) · Case 3 interactive |
| **4** | Cases 1–2–4 · benchmark spec page · homepage copy live on GitHub Pages |
| **5** | Pilot waitlist · API docs · Research OS cross-links |
| **6** | Buffer · product overview polish · first external demo |

### Definition of done

- [ ] Case 3 Diff explorer works offline from exported JSON
- [ ] API report validates against `api_schema_v0.1.json`
- [ ] Benchmark spec published
- [ ] Homepage hero + 4 case cards (static OK)
- [ ] Paper 002 linked as research evidence · not product claim

---

## 8. Public surfaces (this repository)

| Surface | URL (proposed) | Content |
| --- | --- | --- |
| Research OS | `github.com/.../research-os` | Papers · prereg · protocols · promoted results |
| Mismatch Lab | `/mismatch_lab` on Pages | Diff · benchmark · pilot · SDK sketch |

Do **not** replace Research OS with a product pitch. Commercial hosting and customer operations stay **out of research-os**. Add Mismatch Lab as a sibling entry in [docs/index.md](../index.md).

---

## 9. Metrics (v0.1 — internal only)

| Metric | Target |
| --- | --- |
| Case 3 time-to-insight | < 2 min without docs |
| Pilot signups | 20 qualified (8 weeks post-launch) |
| API prototype adopters | 3 teams run evaluator on own JSON |

Do **not** publish mock pilot % improvements on homepage until confirmatory CI defined.

---

*Updated 2026-07-29*
