Haga / Lab / P2-E2 generative cohort (frozen thresholds)
Generative world-model cohort
The frozen-threshold physics-consistency checker scored over three world models' held-out generations, against real control clips. Sensitivity and specificity are reported together; a specificity number is published only when the real-control cohort is large enough to have a usable denominator.
How much of what the checker sees is a real physical violation, and how much is the checker's own sensitivity?
Proposition
What the run claims
A checker that fires on generated video is only useful if it stays quiet on real video. This run scores one frozen threshold set over held-out generations from three world models, against real controls.
Apparatus
What was measured
Held-out generations from CogVideoX, Cosmos and Wan, plus a real-footage control cohort. Seeds and perspectives are fixed by protocol before anything is generated.
Method
How it was measured
The superseded binary FLAG/CLEAN checker (no ABSTAIN state) scores every cohort under frozen thresholds. Flag rates are reported per model, with intervals. Real-control specificity is 0.000 (15/15 flagged). Re-scoring under the SAP v2 three-state gate is outstanding.
Observations
What was observed
The headline figures below are the run's own. The video shows the same run as the pipeline produced it.
- cogvideox flagged
- 0.889
- 8 of 9 held-out clips flagged
- cosmos flagged
- 1.000
- 9 of 9 held-out clips flagged
- wan flagged
- 0.889
- 8 of 9 held-out clips flagged
- Real-control specificity
- 0.000
- 15 real controls, 15 flagged
| Observation | Value | 95% interval | Note |
|---|---|---|---|
| cogvideox flag rate | 0.889 | [0.565, 0.9801] | 9 clips |
| cosmos flag rate | 1.000 | [0.7008, 1] | 9 clips |
| wan flag rate | 0.889 | [0.565, 0.9801] | 9 clips |
| Real-control specificity | 0.000 | — | 15 real controls, 15 flagged |
Limitations
What this run does not show
How much of what the checker sees is a real physical violation, and how much is the checker's own sensitivity?
- Real-control specificity is measured over n=15 real clips, which is the smallest cohort the frozen protocol accepts.
- The scoring device is part of the result: CUDA and Apple MPS disagreed on one borderline Cosmos clip. The Kaggle snapshot is canonical.
- Held-out protocol v1 was frozen 2026-07-19 with seeds 2-4, n=9 per model.
- Position-only checks cannot see a gripper: a grasped, stationary object is indistinguishable from a generative freeze.