Rank 2 of 8 by accuracy

OpenAI Decisions

OpenAIMaker's APIMeasured 2026-10-11Protocol 1.0

Headline

Accuracy88.0% (95% CI 85.6–90.2%), over 2,343 questions
Decision-zone calibration gap (60%–90% confidence)1.6 pts (95% CI 1.3 to 8.9 pts)
Signed gap (accuracy minus confidence)+0.2 pts (95% CI -6.4 to 6.3 pts): no clear direction
Answers in the decision zone507 (21.6% of all answers)
At thresholddefault 70% — none documented: accuracy 91.3% (95% CI 89.1–93.2%), coverage 89.6% (95% CI 87.5–91.7%) — it was sure enough to act on 2,100 of 2,343

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%10564.5%65.7%
70%–80%15674.8%76.9%
80%–90%24685.5%84.1%

Other measures

Expected calibration error (secondary)0.034 (95% CI 0.026–0.063). ECE 95% CI [0.026, 0.063] spans 0.05: inconclusive
Brier score (lower is better)0.098
No answer given0.1%
Same probability on every repeat100.0% of 858 repeated questions
Cost per 1,000 answers4.8 cents
Modelgpt-6-luna
Measured underdecision-models study (closed)

Compared with the leader

On the 2,337 questions both answered, OpenAI Decisions was 7.2 pts less accurate than pplx-decider v1.1 27B (paired difference -7.2 pts, 95% CI -9.4 to -5.0 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.