| Accuracy | 64.8% (95% CI 61.5–68.1%), over 2,394 questions |
|---|---|
| Decision-zone calibration gap (60%–90% confidence) | 5.5 pts (95% CI 2.3 to 9.3 pts) |
| Signed gap (accuracy minus confidence) | -5.5 pts (95% CI -9.3 to -1.7 pts): overconfident |
| Answers in the decision zone | 1,854 (77.4% of all answers) |
| At threshold | default 70% — none documented: accuracy 73.1% (95% CI 68.6–77.6%), coverage 48.9% (95% CI 45.5–52.2%) — it was sure enough to act on 1,170 of 2,394 |
On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.
| Stated confidence | Answers | Mean confidence | Actually right |
|---|---|---|---|
| 60%–70% | 720 | 65.5% | 57.9% |
| 70%–80% | 717 | 74.5% | 71.5% |
| 80%–90% | 417 | 83.8% | 77.7% |
| Expected calibration error (secondary) | 0.051 (95% CI 0.029–0.086). ECE 95% CI [0.029, 0.086] spans 0.05: inconclusive |
|---|---|
| Brier score (lower is better) | 0.224 |
| No answer given | 0.0% |
| Same probability on every repeat | 100.0% of 859 repeated questions |
| Cost per 1,000 answers | — (not hosted) |
| Downloads (popularity, not quality) | 27,918 — Convai Laya family, 30 days, read 2026-11-19 |
| Model | convaiinnovations/laya @ 7b928d828b7b |
| Measured under | decision-models study (closed) |
On the 2,388 questions both answered, Laya was 29.9 pts less accurate than pplx-decider v1.1 27B (paired difference -29.9 pts, 95% CI -33.6 to -26.2 pts).
Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.
Start with the question you actually arrived with — there are five: