Pilot
Run one evaluation, end to end, and get the report.
A pilot is one artifact, one frozen criterion and one report. Nothing about it is a demo: the criterion is agreed before anything runs, the run is on public or private footage you choose, and the report says what was found, including the failures.
What it is
What a pilot is
You give us one artifact — a robot policy, a set of world-model clips, or both. We agree the criterion and the thresholds with you before anything runs, and we freeze them. Then the pipeline runs the same way it ran on the published results, and the report comes back with a verdict, the per-clip evidence, the calibration state, the limitations and the hashes.
The point of a pilot is not the green or red in the report. It is that you end up with a criterion you agreed to, a number you can check, and a limitation list you did not have to ask for.
The timeline
How long it takes
A pilot runs two weeks. That covers agreeing the criterion, the run itself, and the report.
The run is compute-bound rather than engineering-bound, and it is the same pipeline the published results came from, so there is no bespoke work to schedule around.
What you get
What Haga hands back
The report, in the shape the published runs are published in. It carries:
- the verdict, and the numbers behind it, each with its interval
- the per-clip evidence — what was tracked, what was measured, what was compared against which threshold
- the overlay videos: the footage with the measurement drawn on the pixels that were judged
- the calibration state — what the thresholds were fitted on, and whether the calibration and the test set were disjoint
- the limitations, including the ones that make the result weaker than it looks
- the hashes: the digest of every evidence file the report rests on
What you supply
What we need from you
The harness needs the artifact and the thing to judge it against. In practice that means:
- the artifact — the policy, the model, or the clips to score
- its family and the task it is for, so the right geometry and the right detectors apply
- the success criteria, or a starting point for them: what counts as a violation, in your terms
- the footage, and whether it may be published — the published runs use public datasets, and a private run is reported the same way but with the evidence withheld
- the timeline you need the answer by
The cost
What a pilot costs
The standard scopes are priced on the ladder below (indicative, design-partner pricing). A pilot outside those scopes is scoped on a call.
| Scope | What it covers | Price |
|---|---|---|
| Policy stress · Standard | 4 tasks × 4 tiers × 50 episodes, branded report | $15,000–25,000 |
| Policy stress · Focused | 1–2 tasks × 3 tiers × 50 episodes | $5,000–10,000 |
| Policy stress · Continuous | Monthly re-run on policy checkpoints | $3,000–5,000/mo |
| World-model physics · Standard | Up to 50 clips, tracked, full physics-check battery | $10,000–20,000 |
| World-model physics · Held-out | Frozen pre-registered protocol, n=9 per scenario | $15,000–30,000 |
| World-model physics · Continuous | Integrated into customer release pipeline (CI/CD gate) | $2,000–4,000/mo |
| Fixed-scope audit | 6 clips, 5 business days, branded report | $1,900 Foundation · $3,500 Standard · +$1,500 rush |
Out of scope
What a pilot is not
A pilot is not a consulting engagement, and it is not a rewrite of your system. It is one frozen criterion, one run and one report.
- it does not fix the artifact — it tells you what the artifact does, and where it fails
- it does not certify anything: a verdict against an agreed criterion is not a certification, and the report says so
- it does not cover real-robot execution — the policy stress runs in simulation, and the world-model gate runs on video
- it does not promise a passing result. Two of the three published runs failed, and they are published with the same prominence as the rest
Confidentiality
Confidentiality and data handling
Haga's own research runs are published with failures. Customer evaluations are confidential unless the customer opts in in writing.
For private runs, the report is delivered to the customer only. Evidence (overlays, per-clip tables, calibration state) is withheld from publication; delivered to the customer in the private report. Hashes of the evidence files are delivered so the customer can verify integrity later.
Haga retains anonymised aggregate statistics (flag rates, verdict distributions) for internal benchmarking. Raw footage and per-clip evidence are not reused without written permission.
Retention: private reports and evidence hashes are retained for 12 months after delivery unless the customer requests deletion. Anonymised aggregates are retained indefinitely.
Evaluation
Submit a policy or a world model.
Tell us what should be scored. We acknowledge every submission, agree the criterion and the thresholds before anything runs, and report what we find — including the failures.
Or write directly: mushood@hagalabs.com