Rank 8 of 8 by accuracy

Laya

Convai InnovationsSelf-hostedMeasured 2026-10-11Protocol 1.0

Headline

Accuracy64.8% (95% CI 61.5–68.1%), over 2,394 questions
Decision-zone calibration gap (60%–90% confidence)5.5 pts (95% CI 2.3 to 9.3 pts)
Signed gap (accuracy minus confidence)-5.5 pts (95% CI -9.3 to -1.7 pts): overconfident
Answers in the decision zone1,854 (77.4% of all answers)
At thresholddefault 70% — none documented: accuracy 73.1% (95% CI 68.6–77.6%), coverage 48.9% (95% CI 45.5–52.2%) — it was sure enough to act on 1,170 of 2,394

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%72065.5%57.9%
70%–80%71774.5%71.5%
80%–90%41783.8%77.7%

Other measures

Expected calibration error (secondary)0.051 (95% CI 0.029–0.086). ECE 95% CI [0.029, 0.086] spans 0.05: inconclusive
Brier score (lower is better)0.224
No answer given0.0%
Same probability on every repeat100.0% of 859 repeated questions
Cost per 1,000 answers— (not hosted)
Downloads (popularity, not quality)27,918 — Convai Laya family, 30 days, read 2026-11-19
Modelconvaiinnovations/laya @ 7b928d828b7b
Measured underdecision-models study (closed)

Compared with the leader

On the 2,388 questions both answered, Laya was 29.9 pts less accurate than pplx-decider v1.1 27B (paired difference -29.9 pts, 95% CI -33.6 to -26.2 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.