Seeded-Error Testing: A Protocol for Assuring Human Oversight of AI

A reviewer’s performance has two separate qualities, and reporting only the share of errors caught hides the difference. Sensitivity describes how well the reviewer separates erroneous outputs from correct ones. The response criterion describes how much evidence the reviewer needs before flagging something. A reviewer with a high criterion approves most things and catches few errors, without necessarily being less able to recognize them, while a reviewer with a low criterion catches many errors at the price of many false alarms.

The distinction matters for assurance because it points to different remedies. Low sensitivity suggests problems of information, understanding or interface, where the reviewer cannot tell that something is wrong. A high criterion with adequate sensitivity suggests problems of workload, incentives, authority or trust, where the reviewer could tell and does not act. A team that looks like it lacks ability may in fact need different working conditions, and not retraining.

Any test of human performance can mislead, and the paper treats the handling of these threats as part of the method. They include reviewers who become more careful because they know they are being tested, test prevalence that differs from real prevalence, seeded errors that look unlike real ones, reviewers learning the case library, and organizations training staff to the test. For each, the paper offers a response, such as program-level disclosure with case-level blinding where appropriate, rotating and confidential case libraries, and expert review of the realism of seeded cases.

It also treats the ethics directly. Testing people at work raises questions that testing a technical control does not. The method evaluates the oversight arrangement and not the individual, so results should be reported at the group or system level and should not be used to discipline staff, since otherwise reviewers have reason to avoid reporting problems and to game the test. Cases built from personal data need a lawful basis and careful handling, and an organization should decide in advance who must respond to an adverse finding, because a test that finds weak oversight and leads to no action creates a record of known risk.

The protocol does not depend on any single instrument and can supply evidence for several. The paper maps what a seeded-error test can and cannot contribute to provisions of the EU AI Act, the NIST AI Risk Management Framework and ISO/IEC 42001. It shows, for example, that a test can provide direct evidence that overseers in practice detect errors and rely on incorrect output, while it cannot show that the technical controls exist or that reviewers understand the system in general. It also offers candidate requirements that standards drafters could adapt, written modestly and using “should” wherever practice is not mature enough to be mandatory.

A first test does not have to be large. The paper describes a pilot-scale design that can show whether the case library is realistic, whether the safeguards work, and how much reviewer time the test needs, and it notes that pilot results should be treated as indicative because intervals are wide when error counts are small. It identifies the parties involved, from the assessor and domain experts to operational managers, data protection staff, staff representatives and a senior risk owner, and it describes features that make systems easier to test, such as logging the case shown and the reviewer’s action, marking test cases and blocking downstream effects, and replaying past cases in the current interface. It also covers variations, including team-level testing and placing seeded errors in the intermediate steps of an AI agent’s work.

The protocol is a proposal. It draws on established methods in signal detection, human factors and security testing, but it has not been standardized or validated as an assurance method, and its application to oversight of AI has not been systematically evaluated. The paper states that it tests only the error types and conditions it includes, that awareness of testing may raise vigilance and overstate day-to-day detection, that many oversight tasks involve judgment and not binary detection, and that results do not transfer automatically to other sites or later versions of a system. It does not decide how much detection is enough. Seeded-error testing is therefore offered as a starting protocol to be piloted, criticized and improved, and not as a sufficient test on its own.

The paper is written for three groups. Assurance and audit providers can use it to scope and conduct testing engagements, standards drafters can use it to decide what a requirement for oversight effectiveness might mean and what an assessor should be allowed to accept, and risk, compliance and operational leaders in organizations that deploy AI can use it to commission tests and read the results critically. It is relevant to settings such as health, financial services, insurance and public administration, where human review is relied on as a primary safeguard.