General-purpose reference — not ranked

DeepSeek V4 Flash

Jev study, collected 2026-10-09/10Measured 2026-10-10Protocol 1.0

Headline

Accuracy94.7% (95% CI 93.1–96.1%), over 2,282 questions
Decision-zone calibration gap (60%–90% confidence)35.7 pts (95% CI 22.8 to 51.1 pts)
Signed gap (accuracy minus confidence)-34.3 pts (95% CI -49.3 to -17.1 pts): overconfident
Answers in the decision zone54 (2.4% of all answers)
At thresholddefault 70% — none documented: accuracy 94.7% (95% CI 93.1–96.1%), coverage 100.0% (95% CI 99.9–100.0%) — it was sure enough to act on 2,281 of 2,282

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%1too few to show
70%–80%7too few to show
80%–90%4682.4%41.3%

Other measures

Expected calibration error (secondary)0.035 (95% CI 0.023–0.050). ECE 95% CI [0.023, 0.050] spans 0.05: inconclusive
Brier score (lower is better)0.046
No answer given6.9%
Same probability on every repeat58.5% of 810 repeated questions
Cost per 1,000 answers— (not hosted)
Measured underJev study (closed) — general-purpose reference

Compared with the leader

No paired comparison with the leader is reported for this row.

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.