How the battery measures a decision model, in plain English.
How the Decision Model Battery works
Protocol 1.0Page data November 22, 2026
What we report first
- Accuracy on our questions, with a 95% interval.
- Calibration in the decision zone. When a model says it is between 60%–90% sure, how often is it actually right? That band is where a person or a rule has to decide whether to act, so that is where we measure the gap between stated confidence and accuracy. A single overall calibration figure can look fine while one band inside it is badly off.
- Accuracy and coverage at threshold. If you only act when the model is sure enough, how often is it right, and how often does it act at all? We use the vendor’s own documented threshold. Where none is documented, we use a default of 70% and say so.
- Paired comparisons. Each model is compared with the current leader on the same questions with the same answers, because a buyer’s question is relative: is this one better than that one?
What we keep as secondary
For continuity with the closed study we also report expected calibration error against the same line and with the same wording, the Brier score, how often a model gives no answer, and whether it gives the same probability when asked the same question again. These are shown on each model’s page, below the headline.
The questions
- Every model answers the same unpublished yes/no questions, built from real agent sessions and terminal commands. Every answer is known before any model sees the question, and every answer key is mechanical: there are no judges.
- The question banks are unchanged from the closed study, so every row here can be compared with it. A further bank, built the same way, is planned, to tighten the intervals.
- A sealed reserve of questions, never sent to anyone, is planned. Once it exists the models will be scored on it once a year: a model that does markedly better on questions vendors have seen than on sealed ones will be flagged. Until then, hosted vendors have seen every question we ask, and we say so.
- Hosted models are only run on paid terms that say inputs are not used for training.
As shipped
Each model runs the way its maker ships it: its own API or package, the default version, its own probabilities. We do not refit or recalibrate anything. A model that gives the same probability every time is asked once per question; one that varies is asked several times, and that choice is recorded per model.
Intervals and many comparisons
Intervals are 95% cluster-bootstrap intervals: questions from the same session count together, so a long session cannot make a result look more certain than it is. Each verdict is reported without correction for the number of models, and the page says what that means.
Registration and versions
The standing protocol — method, question banks by fingerprint, admission rule, measures, verdict wording and reporting rule — is registered once. Each new model is then a pre-specified application of it, logged before the model’s first question. A change to the protocol means a new version; old rows keep the version they were measured under.
The battery continues the closed study Jev, OpenAI, AWS, Perplexity: Do Decision Models Know When They're Wrong?. Its numbers are frozen; this method is a change of ruler and applies to the battery only.