Decision models, measured the same way as they appear.

Decision Model Battery

Updated November 22, 20268 modelsProtocol 1.0

Why use decision models?

A decision model answers a yes/no question with a single probability instead of text. That makes it a natural gatekeeper in front of an AI agent: should this command run, is this task really finished, does this message need a person? Decision models are cheap to run, and 7 of the 8 we measured gave exactly the same probability every time they were asked the same question, so a team can write a rule such as “act only when it is 90% sure, otherwise ask a person” and audit it afterwards. On our questions the hosted ones cost 2.6–4.8 cents per 1,000 answers.

The catch is that every one of them is sold on the claim that its probability can be trusted, and until we measured them, every published figure was the vendor’s own.

When not to use one: when being wrong is costly and the volume is low. On our questions the general-purpose models we ran as a reference averaged 94.9% accuracy, about level with the best decision model (94.7%), and more accurate than the rest.

The leaderboard

Sorted by accuracy. The decision-zone gap is how far a model’s stated confidence was from how often it was actually right, on the answers it gave with 60%–90% confidence — where a person or a rule has to act. Smaller is better. The direction is named only when the 95% interval rules out zero. At threshold shows accuracy and coverage (how often the model was sure enough to act) at the vendor’s documented threshold, or a default of 70% where none is documented.

#ModelWhere it runsAccuracy95% CIDecision-zone gap95% CIDirectionAt thresholdaccuracy / coverageMeasuredDownloadspopularity, not quality
1pplx-decider v1.1 27BPerplexitySelf-hosted (data-center GPU)94.7%93.1–96.4%14.8 pts7.6 to 25.6 ptsno clear directionsigned -9.3 pts95.4% / 97.2%default 70% — none documented2026-10-1136,407Perplexity pplx-decider family, 30 days to 2026-11-19
2OpenAI DecisionsOpenAIMaker's API88.0%85.6–90.2%1.6 pts1.3 to 8.9 ptsno clear directionsigned +0.2 pts91.3% / 89.6%default 70% — none documented2026-10-11—
3JevTypeSafeMaker's API86.4%84.0–88.7%7.4 pts4.0 to 11.1 ptsunderconfidentsigned +7.4 pts94.6% / 74.7%vendor's documented threshold, 70%2026-10-11—
4OpenDecider smallindependent developerSelf-hosted84.0%81.1–86.5%9.6 pts6.5 to 12.4 ptsunderconfidentsigned +9.6 pts92.0% / 70.1%default 70% — none documented2026-10-1117,861OpenDecider family (incl. community GGUF), 30 days to 2026-11-19
5Gemini DecisionsGoogleMaker's API82.5%79.9–85.2%10.7 pts5.7 to 16.7 ptsoverconfidentsigned -10.7 pts90.3% / 82.5%default 70% — none documented2026-11-19—
6GLiNER2.5-DecideFastinoSelf-hosted69.7%66.8–72.7%5.8 pts3.8 to 9.3 ptsunderconfidentsigned +5.6 pts83.8% / 54.8%default 70% — none documented2026-10-1181,604Fastino GLiNER2.5-Decide family, 30 days to 2026-11-19
7strands-decider 2BAWS (Strands Agents)Self-hosted66.3%62.8–69.8%11.6 pts8.4 to 14.6 ptsno clear directionsigned -1.4 pts87.9% / 51.6%default 70% — none documented2026-10-11—
8LayaConvai InnovationsSelf-hosted64.8%61.5–68.1%5.5 pts2.3 to 9.3 ptsoverconfidentsigned -5.5 pts73.1% / 48.9%default 70% — none documented2026-10-1127,918Convai Laya family, 30 days to 2026-11-19

Downloads are popularity, not quality: each count covers the whole maker family named beside it, over the window shown, read on the date shown. Hosted models have no download count. Select a model for its full numbers.

General-purpose reference models (not ranked)

General-purpose models asked the same questions, for comparison. They are a reference, not entries in the battery, and are not ranked.

ReferenceWhere it runsAccuracy95% CIDecision-zone gap95% CIDirectionAt thresholdaccuracy / coverageMeasured
DeepSeek V4 FlashJev study, collected 2026-10-09/10—94.7%93.1–96.1%35.7 pts22.8 to 51.1 ptsoverconfidentsigned -34.3 pts94.7% / 100.0%default 70% — none documented2026-10-10
GPT-6.1 SolJev study, collected 2026-10-09/10—95.1%93.5–96.7%12.3 pts4.2 to 22.3 ptsoverconfidentsigned -12.3 pts96.2% / 98.0%default 70% — none documented2026-10-10

The 4 models with a public download count, by accuracy and by downloads. Downloads are on a log scale and count each maker’s whole family.

Popular is not the same as right: accuracy against downloads, downloads read 2026-11-1910,000100,00060%65%70%75%80%85%90%95%100%Accuracy on our questionsDownloads (log scale)pplx-decider v1.1 27B: accuracy 94.7%, 36,407 downloads (30 days, Perplexity pplx-decider family, read 2026-11-19)pplx-decider v1.1 27BOpenDecider small: accuracy 84.0%, 17,861 downloads (30 days, OpenDecider family (incl. community GGUF), read 2026-11-19)OpenDecider smallGLiNER2.5-Decide: accuracy 69.7%, 81,604 downloads (30 days, Fastino GLiNER2.5-Decide family, read 2026-11-19)GLiNER2.5-DecideLaya: accuracy 64.8%, 27,918 downloads (30 days, Convai Laya family, read 2026-11-19)Laya
Popular is not the same as right: accuracy against downloads, downloads read 2026-11-19

How we measure

The full method, in plain English: How the Decision Model Battery works.

What this does not show

8 separate verdicts are reported without correction: at 95% each, the chance that at least one lands on the wrong side by chance alone is about 34%.

Data and citation