Rank 3 of 8 by accuracy

Jev

TypeSafeMaker's APIMeasured 2026-10-11Protocol 1.0

Headline

Accuracy86.4% (95% CI 84.0–88.7%), over 2,389 questions
Decision-zone calibration gap (60%–90% confidence)7.4 pts (95% CI 4.0 to 11.1 pts)
Signed gap (accuracy minus confidence)+7.4 pts (95% CI 3.6 to 11.1 pts): underconfident
Answers in the decision zone1,104 (46.2% of all answers)
At thresholdvendor's documented threshold, 70% (TypeSafe worked example: hold irreversible actions below 0.7): accuracy 94.6% (95% CI 92.7–96.3%), coverage 74.7% (95% CI 72.1–77.3%) — it was sure enough to act on 1,784 of 2,389

Decision zone, bin by bin

On the answers given with each level of confidence, how often the model was right. A well-calibrated model is right about as often as it says.

Stated confidenceAnswersMean confidenceActually right
60%–70%35664.6%70.5%
70%–80%34974.8%84.8%
80%–90%39984.7%91.2%

Other measures

Expected calibration error (secondary)0.052 (95% CI 0.034–0.072). ECE 95% CI [0.034, 0.072] spans 0.05: inconclusive
Brier score (lower is better)0.096
No answer given0.0%
Same probability on every repeat29.1% of 859 repeated questions
Cost per 1,000 answers2.6 cents
Modeljev-latest
Measured underdecision-models study (closed)

Compared with the leader

On the 2,383 questions both answered, Jev was 8.1 pts less accurate than pplx-decider v1.1 27B (paired difference -8.1 pts, 95% CI -10.1 to -6.2 pts).

Data generated November 22, 2026. We publish the question, never the trap: no question, answer key or raw response is shown. How we measure.