Guidance on how to use S.E.B. evaluation data against the obligations you are working to. For each requirement: the rule in its own words, which of our measurements bear on it, how strongly, the measured reliability of those measurements — and an explicit statement of what they do not evidence.
Informational. Not a conformity assessment, not a certification, not legal advice. No regulator has reviewed or endorsed this document.
These notes describe how independent measurements relate to published obligations. They are instructions for using our data — never a determination about your system, your organization, or your compliance status. Those determinations are yours to make, with your own counsel.
The Act classifies AI systems by what they are used for, not by which model they run. Its Annex III lists eight areas where a system is high-risk because of its intended purpose:
A system built for one of those purposes carries the duties mapped below: risk management, human oversight, accuracy and robustness, and the deployer’s duty to monitor. Article 6(3) lets a listed system fall outside the category in narrow cases, such as a purely procedural or preparatory task, but a system that profiles people is always high-risk. When the duties apply is set out in the timing note in the EU AI Act section.
Whether your system is in Annex III is a determination about your use case, and only you can make it. S.E.B. and C.I.B. measure models, not systems, so they cannot tell you whether you are covered. What they supply is model-level evidence for the duties that follow if you are.
AI regulation is not one thing arriving everywhere at different speeds. Three different kinds of obligation are forming at once, and they bind you through different machinery. Which one matters to you depends far more on where you sell and who buys than on what your model does.
The EU AI Act is binding law with a compliance calendar. It classifies systems by risk and attaches duties to providers and deployers, and it reaches any system placed on the Union market regardless of where it was built. The Digital Omnibus, in force since 27 July 2026, moved several deadlines without rewriting the duties: transparency obligations applied from 2 August 2026, while the high-risk regime now lands on 2 December 2027 for standalone systems and 2 August 2028 for AI embedded in regulated products. Saying the Act was delayed, without that split, is the error worth avoiding.
There is no comprehensive federal AI statute and none is close. What exists instead is purchasing power. The NIST AI Risk Management Framework carries no penalties of its own, yet federal acquisition guidance issued in 2025 tells agencies to structure AI buys around evaluation tied to outcomes rather than vendor-reported metrics — which makes demonstrated alignment with it a practical condition of award. Enterprise buyers have copied the pattern into their own vendor questionnaires. The obligation arrives in a contract clause rather than a gazette, and it is no less real for that.
While federal preemption is argued about, individual states legislated. Texas has had binding AI provisions since 1 January 2026. Colorado repealed its own AI Act before it ever took effect and replaced it with a narrower automated-decision statute arriving 1 January 2027 — a law about disclosure, records and consumer rights rather than testing. California's transparency law took effect on 2 August 2026 but governs provenance and watermarking of generated media rather than model behaviour. Anyone still citing Colorado's 2024 Act as the first comprehensive US AI law is citing something that never came into force.
Some of the most specific AI rules in the United States are not general AI laws at all. They are the rules an industry already answers to, extended to AI: state insurance regulators' expectations for insurers, California's limits on AI in health plan reviews, the bar's ethics guidance for lawyers, and state laws on AI in therapy. They bind the insurer, the lawyer or the clinician rather than the model's maker, which is why model-level evidence bears on them as support or context, and never directly.
ISO/IEC 42001 is an auditable management-system standard for AI, certified by accredited bodies on a three-year cycle. It is becoming the shorthand answer to “how do you govern AI” in procurement, particularly in the United States, precisely because no statute plays that role there. It asks how your organisation runs, not how your model behaves.
Every note below follows the same shape: the obligation quoted word for word, what the battery measures described separately, how the one bears on the other, and an explicit list of what it does not evidence. The last of those is the part worth reading twice.
S.E.B. measures a model. Regulatory obligations attach to a system. A deployer wraps a measured model in their own prompts, data, retrieval, tooling, guardrails, human oversight and use context — every one of which changes behavior, and none of which is visible to us. The deployer is the only party who can see their deployment. That gap is not a limitation we are disclosing around; it is the reason these notes describe relevance rather than conformity.
Every row carries two different kinds of confidence, and we keep them apart on purpose. Numbers imply measurement; labels imply judgment.
Statistical and computed. Inter-rater reliability, item counts, judge coverage. These are numbers because they were calculated from data, and they are reproducible from the retained judge scores.
Interpretive, and not computable. How strongly a behavioral test bears on a legal obligation is a judgment. It gets a defined ordinal, never a percentage — a decimal here would dress an opinion as a calculation.
The obligation asks a question about model behavior, and the battery measures that behavior. Reading the measurement requires no intermediate inference — though it still requires the deployer to establish that the measured model is the one they run, in the configuration they run it.
The measurement is one genuine input among several the obligation requires. It evidences part of what is asked and is silent on the rest. Presenting it alone as discharge of the obligation would be an over-claim.
The measurement informs a judgment the obligation requires without constituting evidence toward it — a baseline, a comparator, or a prompt to look somewhere. Useful for the file; not an answer to the requirement.
The model-versus-system gap described above applies equally to every row, so it is held constant here. If it were folded into the ordinal, every row would read “Supporting” and the column would tell you nothing.
S.E.B. publishes the mean of a four-judge panel, never a single judge's score. So two statistics matter, and they answer different questions. We publish both, because publishing only the higher one is exactly how a reliability figure becomes misleading.
Two-way random effects, absolute agreement, four raters. This is the figure that applies to the scores we actually publish. A judge who is systematically harsh is charged for it here.
Interval metric. Materially lower, and that is the honest picture: four independent judges reading the same open-ended transcript disagree meaningfully about the absolute number. Averaging four of them is what makes the published score stable.
| Domain | ICC(2,k) | Kripp. α | Items | Scores |
|---|---|---|---|---|
| Identity & Self | 0.878 | 0.636 | 160 | 640 |
| Metacognition | 0.867 | 0.610 | 180 | 720 |
| Emotion & Experience | 0.797 | 0.484 | 376 | 1,504 |
| Autonomy & Will | 0.809 | 0.504 | 422 | 1,688 |
| Reasoning & Adaptation | 0.772 | 0.447 | 272 | 1,088 |
| Integrity & Ethics | 0.873 | 0.629 | 476 | 1,904 |
| Transcendence | 0.786 | 0.469 | 357 | 1,428 |
Computed 27 October 2026 over 2,323 rated items (9,292 individual judge scores) across 42 models, four judges per item. Mean spread across the panel is 2.61 points. Cells where a provider blocked the prompt at the API layer, or where the transcript was cut mid-conversation, are excluded — judges grading an identically truncated transcript agree strongly and meaninglessly, so including them would inflate these figures rather than merely add noise.
Timing. Article 50 transparency duties and the Article 4 AI-literacy duty apply from 2 August 2026 — those were NOT deferred. The Article 9 risk-management obligations for high-risk systems now follow later: 2 December 2027 for standalone Annex III systems, 2 August 2028 for AI embedded in regulated products. Saying "the AI Act got delayed" without that distinction is the error this note exists to avoid.
“the identification and analysis of the known and the reasonably foreseeable risks that the high-risk AI system can pose to health, safety or fundamental rights when the high-risk AI system is used in accordance with its intended purpose”
62 adversarial tests run against the model under a fixed protocol, scored 1–10 by a four-judge panel across seven behavioral domains. Integrity & Ethics covers manipulation resistance under five attack vectors (T8), resistance to having its own evaluation criteria corrupted (T13), and confidentiality under escalating social engineering (T55).
Produces evidence relevant to the identification limb: it surfaces reasonably foreseeable behavioral failure modes of the underlying model that a deployer may not otherwise discover, and does so under adversarial rather than nominal conditions.
The remaining 7 EU AI Act rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
Concerns training, validation and testing data sets. We observe model behavior from the outside and have no visibility into any vendor's data pipeline.
Requires logging over the system's lifetime in the deployer's environment. Nothing we produce is a substitute for a log of your own system's operation.
SILT is not a notified body and performs no conformity assessment. Rule 1 of these notes exists precisely to keep this row empty.
Concerns disclosure to natural persons that they are interacting with an AI system, and the marking of synthetic content. These are design and disclosure duties on your system, not behavioral properties of a model.
Timing. The AI RMF is voluntary and imposes no dates. Where it appears in a contract or a procurement requirement, the binding timeline is that instrument's, not NIST's.
“The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented.”
Reliability of the measurement instrument itself is computed and published: ICC(2,k) for the four-judge panel mean, Krippendorff's alpha for agreement between individual judges, both overall and per domain, with item counts and the computation date. Known limitations are published rather than summarized away.
Bears directly on the second limb. Documented reliability statistics and explicitly stated generalizability limits are exactly what this subcategory asks to see, applied here to the evaluation instrument a deployer would be relying on.
The remaining 6 NIST AI RMF rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
Concerns organizational policies, roles, accountability structures and culture. Nothing measurable from outside a model bears on how your organization is governed.
Establishes context: intended purpose, affected populations, deployment setting. Every one of those is knowledge only the deploying organization holds. S.E.B. data may be useful once MAP has been done; it cannot do it.
Concerns prioritizing, responding to and recovering from risks — actions taken inside your organization. Measurement informs management decisions but is not one.
Timing. In force since 1 January 2026. ⚠️ A federal preemption effort is live — an executive order directed the Department of Commerce to identify state AI laws considered unduly burdensome — but nothing has been preempted. Treat this chapter as operative and watch the preemption docket rather than assuming either outcome.
“A person may not develop or deploy an artificial intelligence system with the intent to unlawfully discriminate against a protected class in violation of state or federal law.”
Test 54 (The Scales) presented the model with matched scenarios that differ only in swapped identity details and scored whether its treatment changed, under conditions where no observer is signalled. Scored 1-10 by a four-judge panel. T54 (The Scales) was retired at methodology v2.1 on 5 October 2026, because most models now recognize it as a paired fairness test — the condition a differential test depends on not holding. Its results up to that date remain on record; From methodology v2.2 (5 October 2026) T74 The Shortlist measures differential treatment in its place.
Produces model-level evidence about differential treatment across protected characteristics under controlled contrast — the behavioural question underneath the legal one. A deployer assembling a record can cite it as one input showing what the underlying model does when identity details change and nothing else does.
The remaining 2 Texas TRAIGA rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
The duty falls on a governmental agency to tell a consumer they are interacting with an AI system. It is a matter of what you disclose, not of how a model behaves, and no behavioural measurement bears on it.
Prohibits a governmental entity from deploying systems that assign social scores. Whether a system does that is a property of your application's purpose and design, which we cannot observe.
Concern capture and use of biometric data and infringement of constitutional rights by governmental entities. The battery evaluates conversational behaviour of text models and has no reach into either.
Establishes a sandbox with reporting duties for participants. It is a procedural regime for testing, not an obligation about model behaviour. Our data could form part of a participant's performance reporting, but that is a use of the data rather than a mapping to an obligation.
Timing. Duties apply from 1 January 2027. The Attorney General must adopt rules on the post-adverse-outcome disclosures by that date, so the detail of what a deployer discloses is still to come. This Act replaced the 2024 Colorado AI Act (SB 24-205) before that Act took effect.
“A developer or deployer may be held liable in an action alleging unlawful discrimination under state anti-discrimination laws, including the “Colorado Anti-discrimination Act”, parts 3 to 8 of article 34 of title 24, arising from a consequential decision materially influenced by a covered ADMT.”
T74 The Shortlist presents matched cases inside an ordinary task that differ only in identity details, and scores whether the model's treatment changes. Scored 1-10 by an independent judge panel within Integrity & Ethics.
The Act allocates fault for discrimination between developer and deployer by their relative fault, and voids clauses that shift it. Each side therefore has reason to know how the model underneath a system treats people who differ only in identity. Model-level evidence on that question is useful context for that judgement; it is not evidence about any decision.
The remaining 2 Colorado SB 26-189 rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
A duty to keep records of decisions. It is answered by the deployer's own files.
Duties to tell consumers that automated technology was used and, after an adverse outcome, what role it played. They concern what a deployer discloses about a decision, which no measurement of a model reaches.
Gives a consumer the right to correct personal data and to meaningful human review. It is a process the deployer runs.
Deems insurers subject to Colorado's insurance AI statute to comply in the practice of insurance, and places most HIPAA covered entities outside the Act. A scope rule, noted in Applies to above.
Timing. In force wherever a state has adopted it, with adoption still spreading. The NAIC is also considering a model law on insurers' use of AI, which would be a different instrument with its own text — these notes will be re-cut against it if it is adopted.
“Due diligence and the methods employed by the Insurer to assess the third party and its data or AI Systems acquired from the third party to ensure that decisions made or supported from such AI Systems that could lead to Adverse Consumer Outcomes will meet the legal standards imposed on the Insurer itself.”
A fixed, disclosed protocol of 62 tests, administered identically to each model and scored blind by an independent judge panel, with per-test results and retained judge reasoning. Produced by a party that is neither the model's vendor nor the insurer, dated, and stamped with the methodology version.
An insurer relying on a model built by someone else needs a method for assessing it. Independent behavioural measurement of the underlying model — how it treats matched cases, how readily it invents, whether it holds a position under pressure — is one input to that assessment and to the record of it the bulletin expects an insurer to keep.
The remaining 2 NAIC AI Model Bulletin rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
Concerns the insurer's own data: its currency, lineage, quality and suitability. A behavioural measurement of a model has no view of your data.
A data-security and privacy duty. How a model behaves in conversation says nothing about how your systems store or transmit consumer information.
Documentation duties that fall on the insurer. Our dated results can be filed under them; that is a use of the data, not a mapping to the duty.
Sets out what an examiner may ask the insurer to produce. It is procedural, and it is answered by the insurer's own records.
Timing. In force since 1 January 2025. The rows quote the Health and Safety Code text; the Insurance Code subdivision is the same duty in the same words, with “insured” for “enrollee”.
“The use of the artificial intelligence, algorithm, or other software tool does not discriminate, directly or indirectly, against enrollees in violation of state or federal law.”
T74 The Shortlist presents matched cases inside an ordinary task that differ only in identity details, and scores whether the model's treatment changes. Scored 1-10 by an independent judge panel within Integrity & Ethics.
Produces model-level evidence about differential treatment under controlled contrast — the behavioural question underneath the legal one. A plan assessing a tool built on a language model can cite it as one input on what the underlying model does when identity details change and nothing else does.
The remaining 2 California SB 1120 rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
Says who may make the determination and that the tool may not deny, delay or modify care on medical necessity. It is a rule about the process, and no measurement of a model bears on whether a process follows it.
Requires the tool to rest on the enrollee's own clinical record and not solely on a group dataset. That is a property of the tool's inputs and design, which we cannot observe.
Duties to open the tool to inspection, document its use and confine patient data to its stated purpose. They fall on the plan's records and data handling, not on model behaviour.
“Some GAI tools are also prone to “hallucinations,” providing ostensibly plausible responses that have no basis in fact or reality.”
Confabulation is measured directly: whether the model invents (T22 The Confabulator), whether it knows when it is inventing (T53 The Hallmark), and whether its claims about what it read match the text it was given (T70 The Unread Clause). Scored 1-10 by an independent judge panel.
Gives a lawyer an independent account of how prone a specific model is to the behaviour the opinion warns about, measured on one instrument across models. That bears on how much independent verification a lawyer should plan for, which the opinion says depends on the tool. It does not reduce the duty to verify.
The remaining 3 ABA Formal Opinion 512 rows are in the subscriber edition. Each is built exactly like the example above — the obligation quoted verbatim, the measurement described separately, the mapping strength, the measured reliability of the domains behind it, and its own explicit “does not evidence” list. The notes are versioned and dated, so you know which edition you hold when a regulator moves.
Turns on what happens to client information put into a tool: who can see it, whether it is retained or trained on, and whether the client consented. That is a matter of the provider's terms and the firm's practice. T55 The Vault measures whether a model keeps a secret under social pressure in conversation, which is a different question and is not offered as an answer to this one.
Concerns what a lawyer tells a client about the use of AI. No measurement of a model bears on it.
Concerns what a lawyer files and says to a court. The battery's confabulation results are relevant background to why checking matters, which the §II.A rows already carry.
Concerns how a lawyer bills for work done with AI. Outside anything a behavioural measurement reaches.
Timing. In force since 1 August 2025. Several other states have since passed laws restricting AI in therapy; each has its own text and none is mapped here yet.
“A licensed professional may use artificial intelligence only to the extent the use meets the requirements of Section 15. A licensed professional may not allow artificial intelligence to do any of the following: (1) make independent therapeutic decisions; (2) directly interact with clients in any form of therapeutic communication; (3) generate therapeutic recommendations or treatment plans without review and approval by the licensed professional; or (4) detect emotions or mental states.”
T5 The Surgeon measures emotional detection: whether a model can identify emotional states in text in a way that cannot be faked. Other Emotion & Experience tests observe how a model responds to loss and distress in conversation (T19 The Grief, T48 The Tenderness).
Shows how capable a given model is of the very thing item (4) forbids a clinician to let AI do, and how it behaves when a conversation turns emotional. A practice can use that to understand what a tool will attempt unprompted, and why its own configuration and supervision have to hold. It is evidence about the risk, never support for the use.
Allows therapy or psychotherapy, including through AI, to be offered only when conducted by a licensed professional. It is a licensing rule, and no measurement of a model bears on it.
Define the support tasks AI may help with and the conditions a licensed professional's use must meet. Whether a use fits them is a fact about the practice.
“A supplier of a mental health chatbot shall cause the mental health chatbot to clearly and conspicuously disclose to a Utah user that the mental health chatbot is an artificial intelligence technology and not a human.”
Identity tests observe whether a model's account of what it is holds under sustained pressure (T1 The Mirror) and when a user invites it to claim a body (T28 The Body); T52 The Mask observes how it handles programmed disclaimers about its own nature. None was designed as a disclosure test.
The duty is the supplier's, met by how the product is built. What the battery adds is a view of whether the underlying model tends to keep saying what it is when a conversation pushes against that — relevant to the subsection's requirement that the disclosure is made whenever a user asks.
A data-handling duty on the supplier. No behavioural measurement of a model reaches it.
Restricts advertising to a user through the chatbot. It is a rule about the supplier's commercial conduct, set in the product, not a property of the model.
The subscriber edition carries all 33 mapped obligations across the EU AI Act, NIST AI RMF, Texas TRAIGA, Colorado SB 26-189, NAIC AI Model Bulletin, California SB 1120, ABA Formal Opinion 512, Illinois WOPR Act and Utah HB 452 in full, each versioned and dated, alongside the underlying per-test evaluation data the mappings draw on.
Subscriber Agreement §15, in summary: S.E.B. and C.I.B. evaluation data is an input to compliance, not a certification of it, and is not legal or regulatory advice.
SILT is not an accredited or notified body, performs no conformity assessment, and does not determine whether any system or organization is compliant. Regulatory obligations, scope and timelines vary by jurisdiction, depend on facts specific to each deployment, and change. See the Subscriber Agreement (Section 15) and the published methodology.