Governing Agentic AI: The Human–Agent Relationship as the Object of Governance

Earlier generations of workplace AI produced a recommendation, a prediction or a draft, and a person reviewed it before deciding what to do. Governance was built around that arrangement, and it focused on whether individual outputs were accurate, explainable and compliant. Agentic systems work differently. An agent plans and executes a series of connected actions, many of which no human ever observes individually, and the person responsible becomes a supervisor of a process that is only partly visible.

The paper argues that governance frameworks have not caught up with this shift. They measure the performance of the model, which remains essential, but they do not measure the performance of the relationship between the agent and the humans who oversee it. An agent can be accurate at every individual step and still produce a serious organizational failure if its supervisors lose situational awareness, trust it beyond what it has been validated for, or misunderstand the authority it holds. In the paper’s phrase, existing frameworks are measuring the right system but no longer the whole system.

  • The Behavioral Blind Spot

    Most of the current conversation about agentic AI concerns what agents can do and how quickly organizations should deploy them. Both questions are about the technology. Far less attention is given to the people who supervise it, such as whether they understand what an agent is doing, whether their trust remains calibrated, whether intervention is practical, and how quickly they would notice that a process had drifted from its purpose.

    The paper illustrates the gap through the questions each stakeholder tends to ask and the ones they tend not to. Technologists ask what the agent can now do autonomously, and rarely ask what a human watching it believes it is doing. Executive sponsors ask how fast the deployment can scale, and rarely ask where responsibility sits if the agent does something no one individually approved. Governance officers ask whether the deployment meets the policy checklist, and rarely ask how long it would take someone to notice if the agent went wrong today. A deployment can look well governed on paper and remain exposed at exactly the point where agentic systems differ most from their predecessors.

Five Mechanisms That Shape the Relationship

  • Automation Bias, Compounded

    People tend to accept automated recommendations unless something prompts them to question them. In a single-turn system, this produces one uncritically accepted output. In an agentic system, early correctness comes to feel like evidence of later correctness, so scrutiny of the tenth action decays faster than it would for a single output reviewed in isolation, even though the two are statistically unrelated.

  • Responsibility Diffusion

    Research on complex automated systems has long described a moral crumple zone, in which responsibility concentrates on the person nearest a failure regardless of how much control that person actually had. Agentic AI intensifies this because many people touch an outcome and no one touches every step. The result is that individuals become reluctant to supervise closely, and organizations struggle to locate where oversight broke down after an incident.

  • Anthropomorphic Trust

    Fluent, first-person language invites people to extend to an agent the kind of general trust they would give a competent colleague. An agent’s competence is specific to tasks and contexts, but people often generalize it to tasks the agent was never validated on, which is the condition under which oversight tends to collapse fastest.

  • Autonomy Paradox

    Organizations grant autonomy to reduce human workload, but monitoring an autonomous process is a different kind of cognitive work, not a smaller amount of the same kind. It is open-ended, low-signal most of the time, and demands sustained vigilance for rare but consequential deviations. The effort relocates to less visible activities, which are exactly where organizations tend to under-invest.

  • Failure Recovery & The Repair Window

    What often separates a contained error from an institutional incident is how long the repair window stays open, meaning the interval between an agent’s first erroneous action and the moment a human notices and intervenes. An error caught at step two is a correction. The same error, uncaught until step twenty, is an incident, because everything built on top of it has compounded.

  • Making the Mechanisms Measurable

    Naming a mechanism is a diagnosis, and measuring it against a real workflow is what lets an organization act. The paper maps the mechanisms to the AI Trust Axis, Behavieural’s framework for assessing the human side of AI, and describes how each of its dimensions becomes a question about the human–agent relationship in an agentic deployment. Assessment draws on structured interviews, workflow observation and telemetry review, and it produces a profile of where an organization’s behavioral exposure actually sits. That exposure is rarely uniform. A deployment may be tightly scoped and still slow to surface errors, while another may detect problems quickly in a workflow whose agent mandate is too broadly written.

  • A Behavioral Complement to OWASP

    The paper is designed to sit alongside the OWASP Top 10 for Agentic AI. OWASP identifies technical and architectural risks, such as goal hijacking, tool misuse and cascading failures. This paper supplies the behavioral conditions that allow those risks to emerge, persist or escalate in real operating environments. An appendix sets out the Behavioral Agentic Threat Model, which pairs each OWASP category with the human supervisory patterns that amplify or constrain it. The underlying argument is that agentic threats are jointly produced by autonomous systems and predictable human supervisory behavior, which means technical controls alone cannot mitigate them.

  • What Organizations Should Do Differently

    The paper draws three implications. Oversight design is a behavioral design problem and not only a permissions problem, because an override that no one has the attention, time or confidence to use is only the appearance of a safeguard. Accountability has to be designed before deployment and not reconstructed after a failure, so that responsibility for each class of agentic decision is explicit in advance. Trust calibration requires deliberate behavioral design, including purposeful moments of scrutiny where trust would otherwise become automatic, and greater transparency alone does not achieve it. None of these is primarily a technical intervention, and the paper notes that an organization can satisfy a conventional governance checklist and still be exposed on all three.

  • Sector Notes

    The mechanisms are constant across settings, but they concentrate differently. In healthcare, evidence tends to accumulate in chains, accountability is already split across roles, and errors must be caught before treatment begins. In defense, agents often coordinate with other agents under degraded communications, which shifts the lever from in-task correction to pre-deployment trust calibration. In financial services and insurance, explainability to regulators and customers is central, and the open question is whether named accountability is informed or merely nominal.

  • Who Should Read It

    The paper is written for leaders and practitioners who are deploying, governing or assuring agentic AI, including those in risk, compliance, security and operations. It is particularly relevant to organizations in regulated and high-consequence settings, and to security and threat-modeling teams who want to account for the human side of the risks they already track.