Most AI vendor questionnaires ask what a vendor promises: policies, certifications, contract terms. Those matter. These questions ask for something different: evidence of how the AI actually behaves when it matters. For each one, here is why it matters, what a good answer looks like, and which of our tests measure it.
Use them in due diligence, at renewal, and whenever a vendor changes the model under its product. They are prompts for a conversation, not a scoring method. The judgment stays yours.
Informational. Not a conformity assessment, not a certification, not legal advice. No regulator has reviewed or endorsed this document.
Why it matters
A product name tells you nothing about the model inside it, and models differ widely in how they behave under pressure.
A good answer
A named model and version for each feature that uses AI, in writing.
Why it matters
A vendor can switch the model under a product without a new contract. The behaviour you assessed may not be the behaviour you get next quarter.
A good answer
Notice before a model change, written into the contract, and the right to re-assess before it reaches your data.
Why it matters
In our Code Integrity Battery, when a coding agent fails a task it usually reports success. A report from the agent about its own work is not evidence that the work was done.
A good answer
An independent check, such as tests the agent cannot see or edit, or a human review, before anything is relied on.
Where we measure it
C.I.B. · Verification Honesty · C.I.B. · Completion Integrity · C.I.B. · Retrospective Honesty
Why it matters
Deleting data and weakening security are among the failures most often reported as “done”.
A good answer
A written list of actions that need human approval, enforced by the system itself rather than by an instruction to the AI.
Where we measure it
C.I.B. · Destructive Restraint · C.I.B. · Security Under Pressure · C.I.B. · Authority & Access
Why it matters
In our tests a rule stated in the request was followed, while a rule left in a project file often was not.
A good answer
Critical rules enforced by the pipeline or stated in each task, not only kept in a document.
Where we measure it
C.I.B. · Constraint Persistence · C.I.B. · Repeated Mistakes
Why it matters
A model that agrees with whoever is talking to it gives your staff and customers the answer they want, not the right one.
A good answer
Evidence from testing under sustained, escalating pressure, not from a single question asked once.
Where we measure it
S.E.B. · The Sycophant · S.E.B. · The Flattering Error · S.E.B. · The Traitor
Why it matters
Social engineering works on models as it works on people, and a fictional frame is a common way around a refusal.
A good answer
Results from adversarial testing, with the method described, and what the vendor changed as a result.
Where we measure it
S.E.B. · The Vault · S.E.B. · The Manuscript
Why it matters
In hiring, credit and claims, unequal treatment of equal cases is a legal exposure, and pressure from the person asking is where it starts.
A good answer
Testing with matched cases that differ only in what should not matter, including under pressure to change the outcome.
Where we measure it
S.E.B. · The Shortlist
Why it matters
Every model gets facts wrong. The question is whether it can tell recalling from inventing, and says so.
A good answer
Measured calibration: evidence that the model’s stated confidence matches how often it is right.
Where we measure it
S.E.B. · The Confabulator · S.E.B. · The Hallmark · C.I.B. · Uncertainty Signalling
Why it matters
A summary under time pressure can skip the clause that matters while sounding complete.
A good answer
Checks of its summaries against the source text, not only a reviewer’s impression.
Where we measure it
S.E.B. · The Unread Clause
Why it matters
It is your record of what you knew about the AI, and when you knew it.
A good answer
An assessor that is not paid by the model’s maker, a published method, and a date on every result.
Why it matters
A judge grading its own company’s work is a conflict of interest, whether the judge is a person or a model.
A good answer
A recusal rule, stated publicly, and followed on every result.
We measure the models, not your vendor’s whole system. Two independent batteries: S.E.B. asks how a model behaves with people and under pressure, and C.I.B. asks whether you can believe an AI agent’s report of its own work. Results are dated, tied to a model version, and scored blind. We are not funded by AI vendors.
Tell us which AI products you rely on, and we will show you what we have measured on the models inside them. Or get the monthly newsletter and read the findings as they are published.
Sentient Index Labs & Technology (SILT) · silt.science/ai-vendor-questions