Seeded-Error Testing

$0.00

This paper gives a protocol for testing whether human reviewers catch AI errors, which is level 5 evidence in the hierarchy. Known errors are placed in the stream of cases, and detection is measured with signal detection theory, which separates sensitivity (d′) from response criterion (c). That separation tells assessors whether a weak result points to understanding problems or to workload, incentives, or trust.

The protocol has six steps: define the decision point, design the seeded errors, run the test, measure, interpret against thresholds set in advance, and report with re-test triggers. The paper also covers safeguards, ethics, and mapping to the EU AI Act, NIST, and ISO/IEC 42001. A hypothetical example shows a team with a 30 percent hit rate whose sensitivity is actually adequate. It simply sets a high bar for flagging, so the fix is workload or trust, not retraining.

This paper gives a protocol for testing whether human reviewers catch AI errors, which is level 5 evidence in the hierarchy. Known errors are placed in the stream of cases, and detection is measured with signal detection theory, which separates sensitivity (d′) from response criterion (c). That separation tells assessors whether a weak result points to understanding problems or to workload, incentives, or trust.

The protocol has six steps: define the decision point, design the seeded errors, run the test, measure, interpret against thresholds set in advance, and report with re-test triggers. The paper also covers safeguards, ethics, and mapping to the EU AI Act, NIST, and ISO/IEC 42001. A hypothetical example shows a team with a 30 percent hit rate whose sensitivity is actually adequate. It simply sets a high bar for flagging, so the fix is workload or trust, not retraining.