From Attestation to Evidence: Assuring That Human Oversight of AI Works

The case rests on a consistent body of research on how people work with automated systems. People over-rely on automated advice, accept recommendations without verification and fail to notice problems the system does not flag. These effects have been found in experts as well as novices and in clinical settings, and they are not reliably removed by training alone. Vigilance research adds that detection of rare events declines with sustained watching, which means a highly accurate system makes the reviewer’s job harder, because errors become infrequent and the habit of agreeing is reinforced by experience.

Two further findings bear on assurance. Explanations and confidence displays, often cited as supports for oversight, can raise acceptance of incorrect advice, so they should be treated as hypotheses to test and not as proof of quality. And where a reviewer carries nominal responsibility without real control, the organization can be protected by the existence of the step while the reviewer absorbs the blame for failures they could not have prevented. For this reason the paper treats assurance as having an ethical as well as a technical dimension.

A uniform requirement for performance testing would be unreasonable, since testing takes time and can burden staff. The paper proposes that expectations scale with the risk of the decision, in the way existing frameworks scale obligations to risk tiers. Low-risk uses might rest on the first three levels, moderate-risk uses on the first four, and high-risk decisions about individuals, such as eligibility, diagnosis, credit or coverage, on all five. For critical decisions, the evidence would be repeated at set intervals and after change. The dividing lines are presented as matters of policy judgment for standards bodies and regulators to settle.

The paper also notes that oversight evidence goes stale as models are retrained, case mixes shift, and reviewers turn over, and it recommends that standards name the events that trigger fresh evidence. It cautions that testing should not expose real people to harm, that results should inform systems and training and not be used to discipline individual reviewers, and that the burden should not fall so heavily on small organizations that they are driven toward the weakest level.

The paper illustrates the effect with an invented scenario in which two organizations have identical policies, workflows, training records, override rates and review times. An assessor who stops at the first three levels would write the same conclusion for both. Examining conditions and measuring performance separates them: in one, reviewers have time, authority and feedback and catch most of the errors seeded in a test, and in the other, a long queue and penalties for rejection make approval the easy path, and few of the errors are caught. The scenario is hypothetical and does not describe a real organization, and its purpose is to show why the highest level should not be reserved for exceptional cases, since it is the one that distinguishes between organizations that look identical on paper.

The paper closes with recommendations grouped by the party best placed to act. Assurance and audit providers are asked to state the evidence level behind every oversight conclusion, to treat override rates and review times as process records and say what they cannot show, to keep the AI system’s accuracy, the reviewer’s performance and the decision outcome separate in reporting, and to seek the higher levels for high-risk decisions or state plainly why they were not available. Standards drafters and regulators are asked to tie expected evidence to risk tier, to define what may be accepted as showing effectiveness, to require refreshed evidence after material change, and to require that reports state the evidence level. Deploying organizations are asked to treat reviewer workload, authority and feedback as part of the control design, to plan early for testing, and not to rely on system accuracy to justify thin oversight evidence. The paper also offers candidate language for standards and guidance, presented for discussion.

The paper is a position paper and does not report new empirical data. The hierarchy is a proposal built from published research and from the text of existing frameworks, and it has not been validated through use in assurance engagements. The paper names the open questions directly: how much evidence is enough at each tier, what error mix tests should use, whether testing changes behavior, whether organizations can game a test that becomes standard, how far evidence from other fields transfers, and whether oversight is the right control at all for a given system, since constraining what a system may do or adding independent checks may protect people better.

The paper is written for assurance and audit providers who write oversight conclusions, standards drafters and regulators who decide what an assessor may accept, and the risk and compliance leaders who commission and read the results. It is relevant to regulated settings such as health, financial services, insurance, public administration and employment, where human review is relied on as a primary safeguard.