Rank 5 of 8 by accuracy

Gemini Decisions

GoogleMaker's APIMeasured 2026-11-19Protocol 1.0

Headline

Accuracy82.5% (95% CI 79.9–85.2%), over 2,397 questions
Decision-zone calibration gap (60%–90% confidence)10.7 pts (95% CI 5.7 to 16.7 pts)
Signed gap (accuracy minus confidence)-10.7 pts (95% CI -16.7 to -5.0 pts): overconfident
Answers in the decision zone795 (33.2% of all answers)
At thresholddefault 70% — none documented: accuracy 90.3% (95% CI 87.9–92.5%), coverage 82.5% (95% CI 79.7–85.1%) — it was sure enough to act on 1,977 of 2,397

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%24665.1%45.1%
70%–80%23174.7%66.2%
80%–90%31885.5%80.2%

Other measures

Expected calibration error (secondary)0.044 (95% CI 0.028–0.068). ECE 95% CI [0.028, 0.068] spans 0.05: inconclusive
Brier score (lower is better)0.111
No answer given0.0%
Same probability on every repeat100.0% of 859 repeated questions
Cost per 1,000 answers4.8 cents
Modelgemini-decision-1
Measured underdecision-models study (closed)

Compared with the leader

On the 2,391 questions both answered, Gemini Decisions was 12.0 pts less accurate than pplx-decider v1.1 27B (paired difference -12.0 pts, 95% CI -14.4 to -9.7 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.