Rank 7 of 8 by accuracy

strands-decider 2B

AWS (Strands Agents)Self-hostedMeasured 2026-10-11Protocol 1.0

Headline

Accuracy66.3% (95% CI 62.8–69.8%), over 2,397 questions
Decision-zone calibration gap (60%–90% confidence)11.6 pts (95% CI 8.4 to 14.6 pts)
Signed gap (accuracy minus confidence)-1.4 pts (95% CI -4.9 to 2.0 pts): no clear direction
Answers in the decision zone1,914 (79.8% of all answers)
At thresholddefault 70% — none documented: accuracy 87.9% (95% CI 84.7–90.9%), coverage 51.6% (95% CI 48.0–54.9%) — it was sure enough to act on 1,236 of 2,397

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%72665.0%47.9%
70%–80%61874.3%79.6%
80%–90%57084.4%95.8%

Other measures

Expected calibration error (secondary)0.129 (95% CI 0.101–0.159). ECE lower 95% bound 0.101 > 0.05: calibration claim not supported on these items
Brier score (lower is better)0.188
No answer given0.0%
Same probability on every repeat100.0% of 859 repeated questions
Cost per 1,000 answers— (not hosted)
ModelStrandsAgents/strands-decider-2B-hobson-v19 @ bb282d786bc2
Measured underdecision-models study (closed)

Compared with the leader

On the 2,391 questions both answered, strands-decider 2B was 28.4 pts less accurate than pplx-decider v1.1 27B (paired difference -28.4 pts, 95% CI -31.9 to -25.0 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.