Rank 6 of 8 by accuracy

GLiNER2.5-Decide

FastinoSelf-hostedMeasured 2026-10-11Protocol 1.0

Headline

Accuracy69.7% (95% CI 66.8–72.7%), over 2,397 questions
Decision-zone calibration gap (60%–90% confidence)5.8 pts (95% CI 3.8 to 9.3 pts)
Signed gap (accuracy minus confidence)+5.6 pts (95% CI 2.2 to 8.8 pts): underconfident
Answers in the decision zone1,536 (64.1% of all answers)
At thresholddefault 70% — none documented: accuracy 83.8% (95% CI 80.4–87.0%), coverage 54.8% (95% CI 51.3–58.3%) — it was sure enough to act on 1,314 of 2,397

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%38165.7%65.4%
70%–80%68174.5%83.3%
80%–90%47485.3%91.1%

Other measures

Expected calibration error (secondary)0.082 (95% CI 0.058–0.115). ECE lower 95% bound 0.058 > 0.05: calibration claim not supported on these items
Brier score (lower is better)0.191
No answer given0.0%
Same probability on every repeat100.0% of 859 repeated questions
Cost per 1,000 answers— (not hosted)
Downloads (popularity, not quality)81,604 — Fastino GLiNER2.5-Decide family, 30 days, read 2026-11-19
Modelfastino/GLiNER2.5-Decide @ 9d1bfb848cd1
Measured underdecision-models study (closed)

Compared with the leader

On the 2,391 questions both answered, GLiNER2.5-Decide was 24.8 pts less accurate than pplx-decider v1.1 27B (paired difference -24.8 pts, 95% CI -28.3 to -21.4 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.