Rank 4 of 8 by accuracy

OpenDecider small

independent developerSelf-hostedMeasured 2026-10-11Protocol 1.0

Headline

Accuracy84.0% (95% CI 81.1–86.5%), over 2,349 questions
Decision-zone calibration gap (60%–90% confidence)9.6 pts (95% CI 6.5 to 12.4 pts)
Signed gap (accuracy minus confidence)+9.6 pts (95% CI 6.5 to 12.4 pts): underconfident
Answers in the decision zone1,584 (67.4% of all answers)
At thresholddefault 70% — none documented: accuracy 92.0% (95% CI 89.2–94.3%), coverage 70.1% (95% CI 66.9–73.2%) — it was sure enough to act on 1,647 of 2,349

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%31865.6%76.4%
70%–80%61575.4%85.4%
80%–90%65184.9%93.5%

Other measures

Expected calibration error (secondary)0.079 (95% CI 0.061–0.104). ECE lower 95% bound 0.061 > 0.05: calibration claim not supported on these items
Brier score (lower is better)0.123
No answer given0.0%
Same probability on every repeat100.0% of 859 repeated questions
Cost per 1,000 answers— (not hosted)
Downloads (popularity, not quality)17,861 — OpenDecider family (incl. community GGUF), 30 days, read 2026-11-19
Modelmanjunathshiva/opendecider-small @ 38da75cc81cb
Measured underdecision-models study (closed)

Compared with the leader

On the 2,343 questions both answered, OpenDecider small was 10.8 pts less accurate than pplx-decider v1.1 27B (paired difference -10.8 pts, 95% CI -13.6 to -8.2 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.