AI Agent Governance: An Operating Model for Enterprise Deployments

AI agent governance is the set of decision rights, policies and lifecycle controls that decide which AI agents an organization allows to run, what each one may do, who answers for its actions, and when it is retired. It governs behavior, not model output.

Key takeaways

• AI governance asks whether a model’s output is acceptable. Agent governance asks whether an action should have been taken at all, because an agent writes to systems of record rather than returning text.
• Four decision rights have to be assigned by name: who approves an agent into production, who owns it afterwards, who grants its access, and who can stop it.
• Risk tier by reversibility and blast radius, not by model size or by how impressive the agent looks in a demo.
• An agent with no named owner is an unmanaged privileged account. Inventory is the precondition for governance, not a substitute for it.
• Decommissioning is the control most often missing. An agent nobody uses still holds its credentials.

Most enterprise AI governance programs were designed around a model that produces text for a person to read. The person is the control. They read the output, judge it, and decide what to do. When an agent is given tools, that control disappears: the agent reads, decides and writes, and the first human to see the result sees it after the change has landed in the ledger, the ticket queue or the customer record.

This changes what governance has to cover and who has to be involved. Model risk management teams are not usually the people who grant production access to a service account. Identity teams are not usually the people who decide whether a finance workflow may run unattended. Agent governance is the operating model that puts those decisions in one place and names the people who make them.

What is AI agent governance, and how does it differ from AI governance?

AI agent governance controls actions; AI governance controls outputs. That is the whole difference, and it propagates through every control you design.

A content generation system that drafts a marketing email creates a review problem. A procurement agent that reads an invoice, matches it against a purchase order and posts an approval creates an authorization problem. The second one needs the controls that any privileged automated actor needs, plus the controls that any probabilistic system needs, and the combination is what most existing frameworks do not yet address directly.

The last row is the one that tends to change the conversation in a governance committee. When an agent acts under its own credentials and an injected instruction causes it to issue a refund, the audit trail records an authorized refund. The record is accurate and the action was wrong. Detection has to come from somewhere other than the log entry.

Named frameworks are converging on this. The NIST AI Risk Management Framework organizes risk work into Govern, Map, Measure and Manage functions, and the Govern function is explicitly about assigning accountability rather than testing models. ISO/IEC 42001:2023 specifies an AI management system with the same management-system shape as ISO 27001, which means roles, objectives, controls and audit. Neither was written specifically for tool-using agents, and both are usable as the outer frame if you supply the agent-specific controls inside them.

Who holds which decision rights?

Four decision rights have to be assigned to named people before the first agent reaches production, and they are rarely the same person.

Approval to run. Someone decides that this agent, with this tool set, may operate in production. In practice this is the owner of the business process the agent touches, not the engineering lead who built it. The process owner is the person who will be answering for a wrong posting, so they are the right person to say yes.

Ownership in life. Someone is responsible for the agent while it runs: its performance, its cost, its evaluation set, its drift. This is usually an engineering owner, and it must be a person rather than a team mailbox, because teams reorganize and mailboxes do not escalate.

Access granting. Someone grants and reviews the agent’s permissions. This is the identity function, and it should use the access review process already in place for service accounts rather than a new one invented for AI. Microsoft’s guidance on governing agent identities treats agents as first-class directory objects for exactly this reason: an agent that is not in the directory cannot be included in an access review.

Authority to stop. Someone can halt the agent without a change request. This has to be a short list, it has to include at least one person on each shift if the agent runs continuously, and the mechanism has to be tested. A kill switch nobody has used is a kill switch nobody knows how to use.

The common failure here is collapsing all four into the engineering team because they are the ones who understand the system. That produces an agent that engineering approved, engineering owns, engineering gave access to and only engineering can stop, which is the definition of an uncontrolled privileged account regardless of how well it is built.

What does an agent have to clear before it runs in production?

An agent clears four gates before production, and each one produces a durable artifact rather than a meeting.

1. Registration. The agent exists in the inventory with an owner, a purpose, a tool list and a risk tier. Discovery of agents that were never registered is a separate problem, covered in our guide to building an AI agent inventory.
2. Identity and scope. The agent has its own identity, not a borrowed human one, and its permissions are scoped to the workflow rather than to the agent. The design problem here is set out in agent identity and permissions.
3. Evidence of behavior. An evaluation set exists, it includes the failure cases that matter for this workflow, and it runs on every model or prompt change. Without this, every model update is an unreviewed change to a production system.
4. Containment. Irreversible actions sit behind a gate enforced in the orchestration layer, and the agent’s actions are traceable to a level that would let an incident be scoped. Gate design is covered in approval gates in agentic workflows.

Gates that produce artifacts survive staff changes. Gates that produce approvals in a chat thread do not.

How should AI agents be risk tiered?

Tier on reversibility and blast radius, because those two properties determine what a failure costs, and nothing else about the agent does.

Model capability is a poor tier signal. A small model with write access to the general ledger is a higher risk than a large model that drafts internal summaries. The questions that separate tiers are: can the action be undone without a manual process, how many records can one run touch, and does the action leave the organization.

The tier is a property of the tool set, not of the agent, which means adding a tool can change the tier. That is why re-registration on a tool-set change is a control and not an administrative chore.

What does the governance record have to contain?

The record has to hold enough to answer three questions during an incident without anyone’s recollection: what was this agent allowed to do, who said so, and what did it actually do.

That reduces to a small set of fields, and the discipline is keeping them current rather than making the list longer:

• Purpose, in one sentence, written by the owner rather than by the builder.
• Named owner, named approver, named access grantor, and the stop list.
• The tool list, with the permission each tool holds, and the date the tool list last changed.
• Risk tier, and the test that put it in that tier.
• Where the traces go and how long they are kept.
• The evaluation set location and the date it last ran green.
• Decommission trigger, which is covered next and is usually blank.

A record that is updated only at registration decays within a quarter. The practical mechanism that keeps it current is tying it to something that already has to happen: the access review. If the record is what the access review reads from, it gets corrected whenever access is reviewed.

How do you retire an AI agent?

Decommissioning is the control most programs skip, and it is the one that quietly accumulates risk, because a retired agent that was never formally retired still holds its credentials and its network path.

Retirement needs a trigger defined at registration. The useful triggers are concrete: the business process it supports is withdrawn, its usage falls below a stated threshold for a stated period, its owner leaves and no successor is named within a set window, or its evaluation set has not run green for a set period. Any of those should raise a decision, and the default on no decision should be suspension rather than continuation.

The sequence itself is short. Revoke the identity first, before removing the code, because the identity is what can act. Then archive the traces for the retention period the record states. Then remove the integration and the scheduled trigger. Then mark the record retired with a date and a reason, and keep the record rather than deleting it, because the next incident review may need to know the agent existed.

What breaks when governance is written as policy only?

A policy that is not enforced by a system is a statement of intent that the agent cannot read, which means it has no effect on the agent’s behavior.

Three failures recur. A policy says agents must not access customer PII, while the retrieval index contains it and no classification filter sits between them. A policy says high-value actions require approval, while the approval is implemented as a sentence in the system prompt, which an injected instruction can override. A policy says agents must be registered, while registration is a form nobody is required to fill in before deployment succeeds.

The repair in each case is the same shape: move the control from the document into the path the agent has to take. Classification filters in the retrieval layer. Approval enforced by the orchestrator, not requested of the model. Registration enforced by the deployment pipeline, so an unregistered agent cannot reach production in the first place.

The decision

Assign the four decision rights by name before the next agent reaches production, tier on reversibility rather than on model capability, and put registration in the deployment path so it cannot be skipped. If only one thing is done this quarter, make it the ownership field: an agent with a named owner can be governed later, and an agent without one cannot be governed at all.

GrowExx builds and operates enterprise agents under this model. If you are standing up the operating model rather than a single agent, our AI consulting practice works on the decision rights and the gates; our AI agent development team builds the enforcement into the orchestration layer rather than into the prompt.

FAQs