| Accuracy | 88.0% (95% CI 85.6–90.2%), over 2,343 questions |
|---|---|
| Decision-zone calibration gap (60%–90% confidence) | 1.6 pts (95% CI 1.3 to 8.9 pts) |
| Signed gap (accuracy minus confidence) | +0.2 pts (95% CI -6.4 to 6.3 pts): no clear direction |
| Answers in the decision zone | 507 (21.6% of all answers) |
| At threshold | default 70% — none documented: accuracy 91.3% (95% CI 89.1–93.2%), coverage 89.6% (95% CI 87.5–91.7%) — it was sure enough to act on 2,100 of 2,343 |
On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.
| Stated confidence | Answers | Mean confidence | Actually right |
|---|---|---|---|
| 60%–70% | 105 | 64.5% | 65.7% |
| 70%–80% | 156 | 74.8% | 76.9% |
| 80%–90% | 246 | 85.5% | 84.1% |
| Expected calibration error (secondary) | 0.034 (95% CI 0.026–0.063). ECE 95% CI [0.026, 0.063] spans 0.05: inconclusive |
|---|---|
| Brier score (lower is better) | 0.098 |
| No answer given | 0.1% |
| Same probability on every repeat | 100.0% of 858 repeated questions |
| Cost per 1,000 answers | 4.8 cents |
| Model | gpt-6-luna |
| Measured under | decision-models study (closed) |
On the 2,337 questions both answered, OpenAI Decisions was 7.2 pts less accurate than pplx-decider v1.1 27B (paired difference -7.2 pts, 95% CI -9.4 to -5.0 pts).
Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.
Start with the question you actually arrived with — there are five: