| Accuracy | 86.4% (95% CI 84.0–88.7%), over 2,389 questions |
|---|---|
| Decision-zone calibration gap (60%–90% confidence) | 7.4 pts (95% CI 4.0 to 11.1 pts) |
| Signed gap (accuracy minus confidence) | +7.4 pts (95% CI 3.6 to 11.1 pts): underconfident |
| Answers in the decision zone | 1,104 (46.2% of all answers) |
| At threshold | vendor's documented threshold, 70% (TypeSafe worked example: hold irreversible actions below 0.7): accuracy 94.6% (95% CI 92.7–96.3%), coverage 74.7% (95% CI 72.1–77.3%) — it was sure enough to act on 1,784 of 2,389 |
On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.
| Stated confidence | Answers | Mean confidence | Actually right |
|---|---|---|---|
| 60%–70% | 356 | 64.6% | 70.5% |
| 70%–80% | 349 | 74.8% | 84.8% |
| 80%–90% | 399 | 84.7% | 91.2% |
| Expected calibration error (secondary) | 0.052 (95% CI 0.034–0.072). ECE 95% CI [0.034, 0.072] spans 0.05: inconclusive |
|---|---|
| Brier score (lower is better) | 0.096 |
| No answer given | 0.0% |
| Same probability on every repeat | 29.1% of 859 repeated questions |
| Cost per 1,000 answers | 2.6 cents |
| Model | jev-latest |
| Measured under | decision-models study (closed) |
On the 2,383 questions both answered, Jev was 8.1 pts less accurate than pplx-decider v1.1 27B (paired difference -8.1 pts, 95% CI -10.1 to -6.2 pts).
Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.
Start with the question you actually arrived with — there are five: