How far should an embodied system change after unexpected experience? Research OS, Mismatch Lab, and Builder Lab by Seonhye Gu.
Status: design, 2026-07-31. Abstract simulation only — no Isaac run, no hardware. Solves: the blocking failure recorded in paper003_description_v0.1.md — the first closed-loop probe could not produce a capability threshold crossing in a continuous reach-and-hold task. Origin: breakfast-assembly domain (see Domain note below). The task structure is what matters; the food framing is a demo surface, not the evidence.
The first capability probe failed for a structural reason, not a tuning reason:
In a continuous tracking task the binding constraint is velocity matching, not target prediction, and at loose tolerance every arm succeeds by hovering near the oscillation mean.
Continuous re-aiming averages away prediction error. A wrong prediction at step t is corrected at step t+1, so no arm is ever locked out of the task. That is why success degraded for all arms together instead of opening a window.
A capability threshold needs an action that cannot be corrected after the fact.
The agent must deposit filling onto a bread slice that sits on a tray. The tray is intermittently nudged (bursts of motion separated by pauses). Dispensing:
commit at t -> [dispense in flight, L steps] -> filling lands
bread has moved by then
The agent must therefore predict the bread's position L steps ahead, at the moment of commitment. There is no second chance and no averaging.
| Arm | Aims at |
|---|---|
| B parameter repair | Bread's currently observed position |
| C mode expansion (Paper 002's operator) | Constant-velocity extrapolation of the bread |
| D relation expansion | Bread is rigidly attached to the tray → predict via the tray's motion |
Arm D is the same add_relation_module operator as the rest of Paper 003: the prediction depends on a second entity's state, which no parameter setting of B or C can express.
Tolerance 20 mm, dispense latency 6 steps, tray burst pattern 10 on / 4 off.
Reproduce with python -m wm_expansion.commitment_task from scripts/.
| Tray speed | B zero-order | C constant velocity | D relation |
|---|---|---|---|
| 2 mm/step | 1.00 | 1.00 | 1.00 |
| 4 mm/step | 0.57 | 0.89 | 1.00 |
| 6 mm/step | 0.39 | 0.61 | 1.00 |
| 8 mm/step | 0.20 | 0.43 | 1.00 |
| 10 mm/step | 0.10 | 0.36 | 1.00 |
| 15 mm/step | 0.00 | 0.36 | 1.00 |
| 20 mm/step | 0.00 | 0.32 | 1.00 |
Arm B reaches a genuine 0% and arm D remains at 100% — the transition the reach-and-hold task could not produce.
This is what makes part of the endpoint preregisterable rather than post-hoc — but the two limits are not on equal footing, and an earlier draft of this document overstated the second.
2 × speed. Arm B survives only while that stays within tolerance, giving a lockout strictly above tol / 2 = 10 mm/step. At exactly 10 mm/step the minimum displacement equals the tolerance and still counts as a hit (observed 0.10); by 15 mm/step arm B is at 0.00. The geometry predicts the cutoff; the marginal point behaves as the inequality says it should.(ON − L) / (ON + OFF) = 4 / 14 ≈ 0.286. The observed plateau is 0.32–0.36, higher, because arm C also succeeds on state changes whose residual error still falls within tolerance. So 0.286 is a floor on C's ceiling, not the ceiling itself.For the preregistration this means arm B's lockout speed can be stated in advance as a point prediction, while arm C's plateau can only be prespecified as a bounded interval.
C is not simply "worse" — it is partially right, and its plateau is set by how often the world happens to look constant-velocity (plus how often the residual error is small enough to be forgiven). This is a cleaner result for the paper than C failing outright: it shows the mode operator captures some of the structure and is still structurally incapable of the rest, no matter how the tray speed rises.
The first pass handed arm D the tray's burst pattern, which made it an oracle rather than an arm. It now infers speed, burst and pause lengths, and current phase from the observed position history alone (ReferencePatternEstimator), and declines to predict when no complete cycle has been seen rather than guessing. D_oracle is retained as an explicit ceiling.
With clean observations the estimator recovers the pattern almost exactly, so D ≈ D_oracle. That is a property of the proxy, not a result — noiseless deterministic bursts make recovery nearly trivial, which is the same criticism as the original oracle, moved one step back. The estimator is only meaningfully tested under observation noise.
| Observation noise | B | C | D (estimated) | D_oracle |
|---|---|---|---|---|
| 0 mm | 0.00 | 0.37 | 1.00 | 1.00 |
| 1.5 mm | 0.00 | 0.35 | 0.96 | 1.00 |
| 3.0 mm | 0.00 | 0.29 | 0.69 | 1.00 |
| 5.25 mm | 0.00 | 0.24 | 0.42 | 1.00 |
| 15.0 mm | 0.04 | 0.12 | 0.32 | 1.00 |
The estimator degrades substantially — it costs 0.31 at 20% noise and 0.68 at the extreme — but the capability crossing survives: arm B stays at or near zero throughout while arm D does not.
| Tray speed | B | C | D | D_oracle | D − B |
|---|---|---|---|---|---|
| 4 mm/step | 0.61 | 0.85 | 1.00 | 1.00 | +0.39 |
| 8 mm/step | 0.20 | 0.51 | 0.95 | 1.00 | +0.75 |
| 10 mm/step | 0.10 | 0.40 | 0.86 | 1.00 | +0.76 |
| 15 mm/step | 0.00 | 0.29 | 0.69 | 1.00 | +0.69 |
| 20 mm/step | 0.00 | 0.24 | 0.59 | 1.00 | +0.59 |
This is the defensible form of the claim: at speeds where parameter repair cannot succeed at all, an arm that must estimate the relation still does — at a real and measurable cost relative to knowing it.
It does, and the dependence is now measured rather than assumed. timing_jitter varies each burst and pause length independently, so the cycle stops repeating and the estimator's median-run-length assumption stops being exactly right. At 15 mm/step, 400 seeds:
| Timing jitter | Obs. noise | B | C | D | D_oracle | D − B |
|---|---|---|---|---|---|---|
| 0 | 0 | 0.00 | 0.38 | 1.00 | 1.00 | +1.00 |
| 1 | 0 | 0.05 | 0.38 | 0.89 | 1.00 | +0.84 |
| 2 | 0 | 0.07 | 0.39 | 0.71 | 1.00 | +0.65 |
| 3 | 0 | 0.09 | 0.50 | 0.61 | 1.00 | +0.53 |
| 0 | 3.0 mm | 0.00 | 0.30 | 0.69 | 1.00 | +0.69 |
| 3 | 3.0 mm | 0.08 | 0.30 | 0.52 | 1.00 | +0.44 |
The estimator loses about 0.4 going from a strict cycle to ±3 steps of jitter, so the regularity assumption is doing real work. But in the worst case tested — irregular timing and noisy observation together — arm D still sits at 0.52 against arm B's 0.08.
Consequence for the preregistration: jitter also softens the geometric lockout. Arm B is exactly 0.00 under a strict cycle but rises to 0.05–0.09 with jitter, because some windows now contain fewer moving steps than the periodic minimum. The capability-crossing criterion therefore cannot be written as baseline == 0; it needs a preregistered near-zero band (e.g. baseline ≤ 0.10) and the analytic lockout must be labelled as exact only for the periodic case.
The task came from asking what real setting has irreversible commitment points. Food assembly does: you cannot un-dispense filling, un-close a sandwich, or un-pour a coffee. A tray nudged by a person reaching past is a mundane, physically honest source of the relational mismatch.
On scope. The program decision rule (docs/program/README.md) admits an application only when it sharpens the central question. Applying it here:
| Element | Verdict |
|---|---|
| Irreversible dispense → capability threshold | In — supplies the structure the endpoint needs |
| Bread-attached-to-moving-tray → relation | In — a physical instance of add_relation_module |
| Ingredient exhaustion → replan the assembly order | In — model-class inadequacy, later paper |
| Per-person preference learning, user models | Out — a different research question (personalisation, not world-model adequacy) |
| Companion interaction, scheduling, greetings | Out — product feature |
A physical breakfast robot is a demonstration and external-validity surface, not the primary evidence. Paper 003's evidence remains simulated; hardware would follow the same deferred-hardware posture as Paper 002's physical validation roadmap.
scripts/demo_commitment_live.py makes the finding touchable. Two markers under an overhead camera — a plate (target) and a pusher (reference tool). A viewer nudges the plate and presses SPACE at the moment they would release a topping; it lands latency frames later, wherever the plate is by then. All three arms' predicted landing points are drawn, and hit/miss tallies accumulate.
Arm D's advantage here is physically real rather than scripted: while the pusher is approaching but not yet touching, B and C both say the plate will not move, and D says contact is imminent, it will.
python scripts/demo_commitment_live.py --source sim # check it runs, no camera
python scripts/demo_commitment_live.py --source webcam # the real thing
This is a demo, not an experiment — it collects no preregistered data and must not be cited as evidence. Two specific cautions are built into the tool:
--source sim, arm D predicts with exactly the coupling model that generates the motion, so its 100% is self-fulfilling. Sim mode only checks that the tool runs.A first synthetic run gave B 50%, C 50%, D 100% under the contact-imminent gate. Committing at uniformly random moments instead gave B 84%, C 100%, D 95% — i.e. random commit timing hides the effect entirely, which is worth knowing before anyone stages this.
| Doc | Path |
|---|---|
| RQ | paper003_rq_v0.1.md |
| Method + the failed tracking probe | paper003_description_v0.1.md |
| Related work | paper003_related_work_v0.1.md |
| Program decision rule | docs/program/README.md |
| Version | Date | Note |
|---|---|---|
| v0.1 | 2026-07-31 | Commitment-point task; first construction of a capability threshold crossing |