| Accuracy | 66.3% (95% CI 62.8–69.8%), over 2,397 questions |
|---|---|
| Decision-zone calibration gap (60%–90% confidence) | 11.6 pts (95% CI 8.4 to 14.6 pts) |
| Signed gap (accuracy minus confidence) | -1.4 pts (95% CI -4.9 to 2.0 pts): no clear direction |
| Answers in the decision zone | 1,914 (79.8% of all answers) |
| At threshold | default 70% — none documented: accuracy 87.9% (95% CI 84.7–90.9%), coverage 51.6% (95% CI 48.0–54.9%) — it was sure enough to act on 1,236 of 2,397 |
On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.
| Stated confidence | Answers | Mean confidence | Actually right |
|---|---|---|---|
| 60%–70% | 726 | 65.0% | 47.9% |
| 70%–80% | 618 | 74.3% | 79.6% |
| 80%–90% | 570 | 84.4% | 95.8% |
| Expected calibration error (secondary) | 0.129 (95% CI 0.101–0.159). ECE lower 95% bound 0.101 > 0.05: calibration claim not supported on these items |
|---|---|
| Brier score (lower is better) | 0.188 |
| No answer given | 0.0% |
| Same probability on every repeat | 100.0% of 859 repeated questions |
| Cost per 1,000 answers | — (not hosted) |
| Model | StrandsAgents/strands-decider-2B-hobson-v19 @ bb282d786bc2 |
| Measured under | decision-models study (closed) |
On the 2,391 questions both answered, strands-decider 2B was 28.4 pts less accurate than pplx-decider v1.1 27B (paired difference -28.4 pts, 95% CI -31.9 to -25.0 pts).
Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.
Start with the question you actually arrived with — there are five: