Governing Agentic AI: Why Your AI Governance Checklist is Measuring the Wrong System
Author: Eugene Chan
Published: 19 November 2026 on LinkedIn
-
This is an abridged version of the white paper "Governing Agentic AI" by Behavieural now available at: https://behavieural.com/govagenticai
Every organization rolling out agentic AI right now has a governance checklist. Model evaluations, done. Capability benchmarks, passed. Compliance sign-off, secured. On paper, the deployment looks buttoned up.
First, a quick definition, since the term gets used loosely: agentic AI refers to systems that don't just generate a single output for a person to review, but plan and carry out extended sequences of actions on their own — querying systems, drafting and sending communications, coordinating with other software or other agents — often with no human watching any individual step. That's different from the AI tools most governance frameworks were built for, where a model produced one recommendation, a person reviewed it, and trust was earned or lost one decision at a time.
Once an agent is taking a dozen actions between human check-ins, the human's job quietly shifts from evaluator of outputs to supervisor of a process they can only partially see. And that shift changes what "governance" needs to mean. It's no longer enough to ask whether the model is accurate. The harder question is whether the humans around it still understand what it's doing, retain meaningful control, and can catch a problem before it compounds into something expensive. Most current governance frameworks were never built to answer that question — because it isn't really a question about the model. It's a question about the relationship between the agent and the human who no longer watches every step of it.
That's the gap my newest white paper on governing agentic AI is written to close, and it does so by leaning on decades of behavioral science rather than technical audits.
-
Where the Blind Spot Actually Lives
The commercial conversation about agentic AI has fixated on two things: what agents can now do, and how fast organizations should roll them out. Both are technology questions. Neither asks what happens on the human side once meaningful authority has been handed off.
That omission has a real cost, because the damage from an agentic failure isn't determined by whether an error occurs — it's determined by how long that error goes unnoticed. A single bad output from a traditional AI tool is one bad output. An agent can take a dozen individually reasonable-looking actions on top of one early mistake before anyone realizes the whole trajectory has drifted. In the paper, I walk through why organizations that look "well-governed" on paper can still be structurally blind to this — and what that blindness typically looks like across technologists, executive sponsors, and governance officers, each of whom is asking a version of the right question and still missing the one that actually matters.
-
Five Mechanisms, Not One Vague Risk
Rather than treating "agentic risk" as a single fuzzy category, I draw on five specific, well-established behavioral science concepts — most with decades of research behind them in aviation, clinical decision-making, and human-factors psychology — and shows how each one compounds specifically in agentic settings.
Automation Bias
Long before generative AI, researchers studying pilots and clinicians documented a consistent pattern: once an automated system has been reliable a few times, people stop scrutinizing it. Automation bias is generally defined as "the tendency to over-rely on automated cues and recommendations, and to under-scrutinize them, in proportion to how often the system has been correct in the past" (Parasuraman & Manzey, 2010; Skitka, Mosier & Burdick, 1999). In a single-turn tool, this produces one under-checked answer. In an agentic system, it compounds — the tenth action in a sequence gets far less scrutiny than the first, because the first several went fine.
Responsibility Diffusion, or the "Moral Crumple Zone"
This term comes from human-factors research on complex automated systems (Elish, 2019) and describes "the tendency for legal and organizational blame to concentrate on the person nearest to a failure, regardless of whether that person actually had the power to prevent it" — much like the crumple zone of a car absorbs impact so the occupants don't have to. Agentic AI multiplies the number of people who touched an outcome (whoever set the agent's mandate, whoever supervised it, whoever should have noticed something was off) without any single one of them controlling the whole sequence — which makes this diffusion worse, not better.
Trust Miscalibration
Trust researchers distinguish between trust that's calibrated and trust that isn't. Calibrated trust is "confidence in a system that accurately tracks that system's actual reliability — no more, no less" (Lee & See, 2004). Most people don't calibrate well, because tracking a system's real performance is mentally expensive, so they substitute easier cues — like how fluent or confident the system sounds — for the harder work of checking whether it's actually right. Agents that narrate their own reasoning in natural, first-person language are especially good at triggering this substitution, which is why trust earned on one task often gets extended, unearned, to a completely different task the agent was never tested on.
The Autonomy Paradox
Organizations grant agents more autonomy specifically to reduce human workload — but the oversight burden doesn't shrink the way they expect. This is the applied, agentic-era version of what automation researchers have long called the "ironies of automation" (Bainbridge, 1983): increasing automation tends to make the human's remaining role harder, not easier, because monitoring an autonomous process is a fundamentally different — and more mentally taxing — kind of work than doing the task directly. It draws on what's sometimes called a vigilance decrement: "the documented decline in a person's ability to detect rare, unpredictable events the longer they're required to sustain attention on a low-signal task" (Warm, Parasuraman & Matthews, 2008) — which is close to exactly what supervising an agent asks a person to do.
Failure Recovery and the Repair Window
The single biggest driver of how expensive an agentic mistake becomes isn't whether it happens — it's how long it goes undetected. An error caught two steps in is a quick correction. The same error, uncaught for twenty steps, is an incident, because everything built on top of the original mistake has to be unwound too. How wide that "repair window" stays open before someone notices turns out to matter more than how good the model is.
-
From Diagnosis to Measurement
Naming a mechanism is a diagnosis, not a fix. In the paper, I go further and translate these five mechanisms into a set of measurable dimensions of the human side of an agentic deployment — separate from, and complementary to, the technical evaluations most organizations already run on the model itself.
One dimension worth calling out here is what I term autonomy safety: essentially, whether the people supervising an agent still feel they have meaningful judgment and control over what it's doing, and whether their trust in the agent's scope of action keeps pace as that scope — or the agent's actual behavior — evolves after launch, rather than freezing at the moment someone first approved it. It's a useful example of how these dimensions work, because it makes clear that governance isn't a one-time approval event; it's a condition that has to hold up continuously.
My broader point is that these profiles are rarely uniform. A financial services deployment might score well on one dimension and poorly on another; a healthcare deployment often shows the opposite pattern. Treating agentic risk as one undifferentiated blob — which is what most governance checklists still do — hides exactly the variation an organization needs to see before it can act on it.
-
What This Changes for Leaders
I close with three organizational implications that reframe what "responsible AI deployment" actually requires — around how oversight mechanisms are designed, when accountability needs to be assigned, and how trust gets calibrated over the life of a deployment rather than just at launch. In each case, it contrasts the response most organizations reach for first with the one the underlying behavioral evidence actually calls for — and the two are not the same thing.
It also breaks down how these five mechanisms show up differently across sectors — healthcare, defense, and financial services each stress a different combination of them, for reasons specific to how authority and accountability are already structured in those industries.
The core argument is simple, even if the implications aren't: as organizations delegate more consequential work to AI agents, the object of governance has to expand beyond the model itself to include the relationship between the agent and the human who's supposed to be watching it. Organizations that keep treating agentic AI governance as an extension of model evaluation will keep being surprised by failures their checklists said shouldn't have been possible.