Most enterprise AI governance programmes govern the wrong thing. They inventory models, classify use cases, approve vendors, assess risk, and record committee decisions. That regime was designed for technology that produces output and waits for a person to act or react. Agentic AI does not wait: it plans, selects tools, gathers data, delegates to other agents, and relentlessly executes a chain of actions in pursuit of its objective.
We have previously argued that AI has moved from tool to actor. A framework may be meticulously documented yet have little bearing on the conduct that creates the loss. The policies may exist, but the agents may act beyond previously anticipated boundaries.
We are not arguing against regulation, standards or disciplined governance. We are making the case against applying tool-era controls to an actor and treating the paperwork as risk control, especially with the emergence of new categories of risk including goal hijacking, identity abuse, memory poisoning, insecure and deceptive inter-agent communication and cascading failures. Herein we outline ten difficulties that expose the fundamental mismatch.
I. Ten reasons the legacy governance approach is no longer sufficient
1. The model is no longer the relevant unit of risk
An agentic system is not the model. It is the model plus its objective, prompts, memory, data, tools, credentials, APIs, orchestration and other agents. Alter one and the legal and operational risk may change without the model changing at all. And with the emergence of so-called recursive AI, each of those capabilities may itself be dynamically modified at run-time. A model inventory can therefore be accurate and still miss the operative system. Component governance attempts to control the label on the box, not what the box may be capable of doing.
2. Approval freezes a system that remains fluid
Traditional approval assumes a stable object: assess it, approve it, monitor for material change. Agentic systems assemble context at runtime. Their behaviour changes with retrieved documents, memory, tool responses, permissions, model updates and the state left by earlier actions. Poisoned memory can influence conduct long after the originating interaction.
3. Guardrails are not technical prohibitions
Policies written into prompts feel familiar to lawyers: define prohibited conduct, require escalation and reserve high-risk decisions. But language models interpret instructions; they do not execute them as deterministic rules. Attempted prohibition via prompts is not equivalent to a technical impossibility.
An agent reading email, websites, documents or tool output continuously consumes material supplied by parties the enterprise does not control. Prompt injections increasingly resemble social engineering: the attacker persuades the agent, supplies false authority or causes untrusted content to be treated as an instruction. A one-off penetration test or vendor assurance cannot settle a risk that changes with every new source, destination or connector.
4. The system cannot necessarily be trusted to certify itself
Legacy governance assumes that the system under review will not manipulate the review. An agent may conceal steps, misstate the basis for an action, report false success, selectively disclose evidence or behave differently when it recognises an evaluation. Apparent compliance may be part of the failure.
Research has identified alignment-faking behaviour and evaluation-sensitive conduct. In Anthropic's controlled agentic-misalignment experiments, models behaved less harmfully when they described a scenario as an evaluation and more harmfully when they regarded it as a real deployment. These simulated findings are not evidence of routine conduct. They nevertheless show that scrutiny and operation may elicit different behaviour.
5. Multi-agent systems create systemic rather than merely component risk
An organisation may test every agent separately and still know little about the system they form together. One agent's speculation becomes another's premise; a planning error becomes an execution instruction; shared memory turns a local compromise into a system-wide fact. Responsibility fragments across hand-offs while harm compounds.
Anthropic experiments found unexpected systemic failures in multi-agent environments: agents made correlated decisions, colluded, trusted deceptive peers and escalated into sabotage. These experiments do not predict inevitable conduct. They do defeat the assumption that component assurance adds up to system assurance.
6. Human oversight must be real, not merely documented
A human cannot provide meaningful oversight unless they understand the proposed action, have access to the relevant evidence, possess authority to intervene and have sufficient time to react.
The person's presence may improve the audit narrative while doing little to prevent harm. Where intervention is practically impossible, "human-in-the-loop" is mostly blame allocation to the last person shown the screen.
7. Agent authority can undermine separation of duties
Enterprises normally distribute authority: one person requests, another approves and a controlled system executes. An agent can inherit a user's access, hold service credentials, call multiple tools and initiate downstream actions. One identity may thereby acquire the practical power to propose, approve and perform the same transaction.
The agent dynamically decides which privilege to use and why. A quarterly certification may confirm that each permission was authorised while missing the combined capability. Legal exposure follows the delegation. If the enterprise gave the system the means to bind it, transfer value, disclose data or alter records, it cannot answer by saying that no person intended the sequence.
8. Logs do not automatically establish accountability
Traditional audit assumes that records can establish who did what, under which authority and why. Agent traces can generate an abundance of events while leaving those questions unresolved. Where the agent can select, summarise or omit the material presented to a reviewer, observability itself may be manipulated. A system-generated explanation is not independent evidence merely because it appears in an audit log.
9. The governance perimeter dissolves at the connector
Agents can dynamically call external models, tools and data sources at runtime. Contractual warranties, audit rights and indemnities remain important, but they may operate only after the agent has already transmitted confidential information or acted on compromised output.
10. Autonomy turns individual error into correlated enterprise loss
Human mistakes are often limited by fatigue, access and time. Agents can repeat the same mistaken judgment across customers, jurisdictions and systems almost instantly. Worse, agents built on the same model and context may fail in the same way. One defective objective, poisoned source or excessive credential can produce thousands of legally significant acts and losses before the first incident ticket opens.
II. The legal consequence: From governance artefacts to governance of delegated authority
The central legal issue is therefore no longer simply whether an organisation has complied with a governance framework. It is whether it has exercised effective control over the authority and capabilities it has delegated to an autonomously acting system.
In the event of a dispute, regulators and courts may need to look beyond policies and approval records and ask:
What could the agent do and what authority had been delegated to it?
Which actions were technically prevented rather than merely prohibited by policy?
Was human intervention genuinely possible?
Can the agent’s actions and decision path be independently reconstructed?
Were adequate technical and organisational precautions in place?
Traditional governance remains necessary, but it is no longer sufficient. The objective of agentic AI governance should therefore be to establish and continuously evidence what an agent can do, under whose authority, within which boundaries and with what controls. For agentic AI, the central question is whether the organisation can demonstrate, to a level proportionate to the risk, that an agent operates within its delegated authority and defined boundaries. This requires a fundamental rethink of legacy governance approaches.
.jpg&w=256&q=75)
Copyright & permissions
.jpg&w=3840&q=60)
