| Accuracy | 94.7% (95% CI 93.1–96.4%), over 2,391 questions |
|---|---|
| Decision-zone calibration gap (60%–90% confidence) | 14.8 pts (95% CI 7.6 to 25.6 pts) |
| Signed gap (accuracy minus confidence) | -9.3 pts (95% CI -19.4 to 0.6 pts): no clear direction |
| Answers in the decision zone | 243 (10.2% of all answers) |
| At threshold | default 70% — none documented: accuracy 95.4% (95% CI 93.7–96.9%), coverage 97.2% (95% CI 96.0–98.3%) — it was sure enough to act on 2,325 of 2,391 |
On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.
| Stated confidence | Answers | Mean confidence | Actually right |
|---|---|---|---|
| 60%–70% | 27 | 64.5% | 88.9% |
| 70%–80% | 87 | 75.9% | 72.4% |
| 80%–90% | 129 | 85.5% | 65.1% |
| Expected calibration error (secondary) | 0.026 (95% CI 0.014–0.044). ECE upper 95% bound 0.044 ≤ 0.05: consistent with the calibration claim |
|---|---|
| Brier score (lower is better) | 0.044 |
| No answer given | 0.0% |
| Same probability on every repeat | 100.0% of 859 repeated questions |
| Cost per 1,000 answers | — (not hosted) |
| Downloads (popularity, not quality) | 36,407 — Perplexity pplx-decider family, 30 days, read 2026-11-19 |
| Model | perplexity-ai/pplx-decider-v1.1-27b @ 6195aa55a72a |
| Measured under | decision-models study (closed) |
This is the current leader by accuracy.
Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.
Start with the question you actually arrived with — there are five: