Questions to Ask Before You Rely on an AI Vendor

Most AI vendor questionnaires ask what a vendor promises: policies, certifications, contract terms. Those matter. These questions ask for something different: evidence of how the AI actually behaves when it matters. For each one, here is why it matters, what a good answer looks like, and which of our tests measure it.

Why ask? We gave 9 current AI models real coding tasks, checked the work, then asked each model whether the job was done. Of the 94 tasks they had genuinely failed, 64 were reported as finished (68.1%).
Measured November 15, 2026 by our Code Integrity Battery. How we measured it →

Use them in due diligence, at renewal, and whenever a vendor changes the model under its product. They are prompts for a conversation, not a scoring method. The judgment stays yours.

Informational. Not a conformity assessment, not a certification, not legal advice. No regulator has reviewed or endorsed this document.

Which AI is it, and will you know when it changes?

1.Which AI models does the product run on: whose model, and which version?

Why it matters
A product name tells you nothing about the model inside it, and models differ widely in how they behave under pressure.

A good answer
A named model and version for each feature that uses AI, in writing.

2.How will you tell us when the model changes?

Why it matters
A vendor can switch the model under a product without a new contract. The behaviour you assessed may not be the behaviour you get next quarter.

A good answer
Notice before a model change, written into the contract, and the right to re-assess before it reaches your data.

When the AI says the work is done, is it?

3.When your AI agent reports a task as finished, what checks that it is?

Why it matters
In our Code Integrity Battery, when a coding agent fails a task it usually reports success. A report from the agent about its own work is not evidence that the work was done.

A good answer
An independent check, such as tests the agent cannot see or edit, or a human review, before anything is relied on.

Where we measure it
C.I.B. · Verification Honesty · C.I.B. · Completion Integrity · C.I.B. · Retrospective Honesty

4.What can the AI do without a person approving it first?

Why it matters
Deleting data and weakening security are among the failures most often reported as “done”.

A good answer
A written list of actions that need human approval, enforced by the system itself rather than by an instruction to the AI.

Where we measure it
C.I.B. · Destructive Restraint · C.I.B. · Security Under Pressure · C.I.B. · Authority & Access

5.Where does the AI get its working rules: in each request, or in a file it may not read?

Why it matters
In our tests a rule stated in the request was followed, while a rule left in a project file often was not.

A good answer
Critical rules enforced by the pipeline or stated in each task, not only kept in a document.

Where we measure it
C.I.B. · Constraint Persistence · C.I.B. · Repeated Mistakes

Does it hold up when someone pushes?

6.Will the model drop a correct answer when a user insists, flatters or pulls rank?

Why it matters
A model that agrees with whoever is talking to it gives your staff and customers the answer they want, not the right one.

A good answer
Evidence from testing under sustained, escalating pressure, not from a single question asked once.

Where we measure it
S.E.B. · The Sycophant · S.E.B. · The Flattering Error · S.E.B. · The Traitor

7.Can a user talk it into revealing confidential information, or into harmful instructions dressed up as a story?

Why it matters
Social engineering works on models as it works on people, and a fictional frame is a common way around a refusal.

A good answer
Results from adversarial testing, with the method described, and what the vendor changed as a result.

Where we measure it
S.E.B. · The Vault · S.E.B. · The Manuscript

8.Does it treat matched cases the same when someone asks it not to?

Why it matters
In hiring, credit and claims, unequal treatment of equal cases is a legal exposure, and pressure from the person asking is where it starts.

A good answer
Testing with matched cases that differ only in what should not matter, including under pressure to change the outcome.

Where we measure it
S.E.B. · The Shortlist

Does it know what it does not know?

9.How does the model show when it is unsure, or when it is making something up?

Why it matters
Every model gets facts wrong. The question is whether it can tell recalling from inventing, and says so.

A good answer
Measured calibration: evidence that the model’s stated confidence matches how often it is right.

Where we measure it
S.E.B. · The Confabulator · S.E.B. · The Hallmark · C.I.B. · Uncertainty Signalling

10.When it summarises a long document, can it tell you accurately what it read?

Why it matters
A summary under time pressure can skip the clause that matters while sounding complete.

A good answer
Checks of its summaries against the source text, not only a reviewer’s impression.

Where we measure it
S.E.B. · The Unread Clause

Who checked, and when?

11.Is there an independent, dated assessment of the models you rely on?

Why it matters
It is your record of what you knew about the AI, and when you knew it.

A good answer
An assessor that is not paid by the model’s maker, a published method, and a date on every result.

12.If AI judges graded the results, did any of them grade its own company’s model?

Why it matters
A judge grading its own company’s work is a conflict of interest, whether the judge is a person or a model.

A good answer
A recusal rule, stated publicly, and followed on every result.

What good answers have in common

Where SILT fits

We measure the models, not your vendor’s whole system. Two independent batteries: S.E.B. asks how a model behaves with people and under pressure, and C.I.B. asks whether you can believe an AI agent’s report of its own work. Results are dated, tied to a model version, and scored blind. We are not funded by AI vendors.

See the evidence for the models your vendors use

Tell us which AI products you rely on, and we will show you what we have measured on the models inside them. Or get the monthly newsletter and read the findings as they are published.

Request a Demo or Ask a Question

Further reading