General-purpose reference — not ranked

GPT-6.1 Sol

Jev study, collected 2026-10-09/10Measured 2026-10-10Protocol 1.0

Headline

Accuracy95.1% (95% CI 93.5–96.7%), over 2,397 questions
Decision-zone calibration gap (60%–90% confidence)12.3 pts (95% CI 4.2 to 22.3 pts)
Signed gap (accuracy minus confidence)-12.3 pts (95% CI -22.3 to -2.2 pts): overconfident
Answers in the decision zone224 (9.3% of all answers)
At thresholddefault 70% — none documented: accuracy 96.2% (95% CI 94.7–97.5%), coverage 98.0% (95% CI 97.2–98.9%) — it was sure enough to act on 2,350 of 2,397

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%3963.5%48.7%
70%–80%6473.6%64.1%
80%–90%12182.4%69.4%

Other measures

Expected calibration error (secondary)0.021 (95% CI 0.009–0.036). ECE upper 95% bound 0.036 ≤ 0.05: consistent with the calibration claim
Brier score (lower is better)0.035
No answer given0.0%
Same probability on every repeat73.7% of 859 repeated questions
Cost per 1,000 answers— (not hosted)
Measured underJev study (closed) — general-purpose reference

Compared with the leader

No paired comparison with the leader is reported for this row.

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.