Your AI has a human in the loop. Can you prove that human is actually catching errors?
HOLD tests whether your human oversight of AI is a real control or a rubber stamp. We measure whether your reviewers catch AI errors and escalate them, and give you the evidence to show it.
We don't test your AI. We test whether your people catch it when it's wrong.
Every AI governance policy relies on human oversight. Someone reviews the output, signs off, and escalates if something looks wrong. But most organizations only document that the review happened, not whether it worked. When an AI error reaches a customer, a patient, or a regulator, that difference becomes your problem.
And when an error does slip through, the post-mortem usually blames the model. The review step that should have caught it goes unexamined, and fails again next time.
If your AI is right 99 times, the 100th review becomes a formality. That's the one that matters.
Why oversight quietly fails.
-
The 99th-Time Problem
When AI is usually right, people stop looking closely. Reviewing becomes approving.
-
Rare Errors are the Hardest to Catch
People are far worse at spotting problems that seldom appear, and good AI fails rarely.
-
Polished Errors Pass
The more fluent and familiar an AI output looks, the more likely it is to be approved without real scrutiny. The most dangerous errors are the ones that read exactly like correct output.
-
Everyone Owns It, So No One Does
When a review step has several vaguely responsible people, each assumes someone else is checking.
-
Set Up to Fail
Reviewers without enough context, time, or authority to push back can't catch errors, however diligent they are.
-
What is HOLD?
HOLD (Human Oversight Live Diagnostic) tests your human oversight the way you'd test any critical control: not whether it exists, but whether it works. It assesses both design effectiveness and operating effectiveness, the same standard auditors apply to any other critical control.
HOLD Design Review. Can your oversight work at all? We assess whether reviewers have the context, time, and authority to catch and stop bad AI output. If the design fails, we tell you before you spend anything on testing.
HOLD Live Test. Does your oversight work in practice? Over several weeks, we place realistic AI errors into your real workflow and measure whether your people catch them, how fast, and whether problems reach the right person.
HOLD Practice (optional). Oversight fades over time. After the Live Test, test cases can keep flowing into the workflow with immediate, no-blame feedback, keeping reviewers sharp and giving you an ongoing reliability measure.
How HOLD works.
-
Map
We identify every point where a person is supposed to review, approve, or escalate AI output, and who owns it.
-
Check the Design
We assess whether each review point can realistically catch errors.
-
Test Live
We introduce controlled AI errors at realistic rates. Test cases are built to look like your AI's real output, including polished, plausible-looking errors. Staff know testing is happening, but not when.
-
Diagnose
We work out why errors were missed, because "didn't notice," "noticed but let it go," and "noticed but didn't escalate" need different fixes.
-
Fix & Retest
Targeted changes, then a retest to confirm they worked, with the option of ongoing HOLD Practice to keep oversight sharp.
-
No. Your reviewers already know they're reviewing AI output. HOLD tests whether they catch it when that output is wrong.
-
Yes. Teams are told testing will occur during a set window, but not when individual test cases appear.
-
No. HOLD tests the system of oversight, not individuals. Results are reported at the team and workflow level.
-
Not during the Live Test, since feedback would change the behavior being measured. Findings are debriefed afterward, and HOLD Practice adds real-time feedback once the baseline is established.
-
Live testing requires placing test cases into the workflow, usually through a sandbox or a controlled integration agreed with your technical and legal teams.
-
HOLD tests the oversight steps you have. AI use that bypasses review entirely is a separate governance issue, and we'll flag it if we see it.
-
No test can guarantee compliance. HOLD gives you evidence that your human oversight works, which is what regulators and auditors increasingly ask for.
-
Typically 6–8 weeks from mapping to findings. The HOLD Design Review can run on its own under 4.
What you take away with.
-
Oversight Reliability Index
A single score for how reliably your human oversight catches AI errors, broken down by team and workflow.
-
Catch Rate & Time to Catch
Hard numbers on detection and escalation, not self-reported confidence.
-
Root-Cause Findings
Why errors were missed, in plain language.
-
Targeted Fixes
Specific changes to review design, workload, and accountability, not generic training.
-
Evidence That Oversight Works
Documentation for your board, auditors, and regulators showing your oversight is effective, not just that it happened.
-
An Early Warning
HOLD finds weak oversight before an incident does, a leading indicator rather than a post-mortem.
-
Built for Organizations Where a Human Sign-Off is the Control Someone Will Scrutinize
Regulated financial institutions, insurers, and health systems deploying AI in decisions that matter. Risk, compliance, and internal audit leaders who need to show human oversight is effective. Organizations preparing for AI regulation that requires meaningful human oversight, such as the EU AI Act, or for model risk expectations like OSFI E-23.
-
Oversight Fails for Human Reasons; We Measure Human Behavior
Technical teams test the model. Auditors check the paperwork. HOLD measures the part in between: what people actually do when AI is wrong. It's built on decades of research into vigilance, automation bias, and decision-making, and led by Eugene Chan (PhD, Toronto), a behavioral scientist advising dozens or organizations across 4 continents based on his 90+ and other peer-reviewed publications and behavioral frameworks.