← Decision Model Battery
How the battery measures a decision model, in plain English.

How the Decision Model Battery works

Protocol 1.0Page data November 22, 2026

What we report first

What we keep as secondary

For continuity with the closed study we also report expected calibration error against the same line and with the same wording, the Brier score, how often a model gives no answer, and whether it gives the same probability when asked the same question again. These are shown on each model’s page, below the headline.

The questions

As shipped

Each model runs the way its maker ships it: its own API or package, the default version, its own probabilities. We do not refit or recalibrate anything. A model that gives the same probability every time is asked once per question; one that varies is asked several times, and that choice is recorded per model.

Intervals and many comparisons

Intervals are 95% cluster-bootstrap intervals: questions from the same session count together, so a long session cannot make a result look more certain than it is. Each verdict is reported without correction for the number of models, and the page says what that means.

Registration and versions

The standing protocol — method, question banks by fingerprint, admission rule, measures, verdict wording and reporting rule — is registered once. Each new model is then a pre-specified application of it, logged before the model’s first question. A change to the protocol means a new version; old rows keep the version they were measured under.

Protocol 1.0Registered. Current.

The battery continues the closed study Jev, OpenAI, AWS, Perplexity: Do Decision Models Know When They're Wrong?. Its numbers are frozen; this method is a change of ruler and applies to the battery only.