Securing an enterprise GenAI system means controlling what it can reach, what it can do, and what it can reveal — because the model itself cannot be relied on to enforce any of those. The controls sit around the model, not inside it.
This is the shift that catches security teams applying conventional application security practice. In a normal application, input validation happens at the boundary and the code then behaves deterministically. In a GenAI system, the input reaches a component that interprets instructions from whatever text it is given, including text that arrived in a retrieved document nobody reviewed.
The model is not a trust boundary. Every control has to assume it can be talked into anything.
What the standards actually say to do
Two references carry most of the weight, and they do different jobs.
The OWASP Top 10 for LLM Applications, 2025 edition is the risk taxonomy: Prompt Injection (LLM01), Sensitive Information Disclosure (LLM02), Supply Chain (LLM03), Data and Model Poisoning (LLM04), Improper Output Handling (LLM05), Excessive Agency (LLM06), System Prompt Leakage (LLM07), Vector and Embedding Weaknesses (LLM08), Misinformation (LLM09) and Unbounded Consumption (LLM10). Use it as the checklist against which a design is reviewed.
The NIST AI Risk Management Framework, released 26 January 2023, with its Generative AI Profile (NIST-AI-600-1), published 26 July 2024, is the governance structure: the Govern, Map, Measure and Manage functions, and a generative-AI-specific set of risks and suggested actions. Use it to decide who is accountable and what gets documented.
Neither is a compliance regime. Where regulation applies, the EU AI Act implementation timeline is the one with dates attached: obligations for general-purpose AI models under Chapter V began applying on 2 August 2025 (Article 113(b)), with providers of models placed on the market before that date required to comply by 2 August 2027 (Article 111(3)). The remainder of the Act, including the Article 50 transparency obligations covering synthetic content, applies from 2 August 2026 (Article 113), and systems already on the market before that date must meet Article 50(2) by 2 December 2026 (Article 111(4)). Confirm the current position with counsel rather than a blog — including this one.
Secure Your Enterprise GenAI at Production Scale
Put practical security controls in place across your AI applications, models, data, integrations, and workflows as usage expands.
The trust boundary moves
In a GenAI system, the boundary sits between the model’s output and anything that acts on it — not between the user and the application.
This single reframing resolves most design arguments. The question stops being “how do we stop the model saying something bad” and becomes “what is the worst thing that can happen if the model says exactly the wrong thing, and what stands between that output and the consequence.”
Three implications:
Model output is untrusted input to the next component. If the output is rendered as HTML, it can carry script. If it is passed to a shell, a database, or a templating engine, it is an injection vector. OWASP files this as Improper Output Handling (LLM05:2025), and it is the most conventional vulnerability class in the list — the mitigation is the same encoding and parameterization you would apply to any untrusted input.
Retrieved content is an input channel. An agent that reads documents, emails or tickets is processing text that an attacker may have authored. NIST has published work on agent hijacking evaluations noting that instructions embedded in processed content can influence agent behavior. Content the system reads deserves the same suspicion as content a user types.
System prompts are not secrets. Treat anything in the system prompt as eventually public — OWASP lists System Prompt Leakage separately (LLM07:2025). Credentials, internal URLs, business rules that reveal commercial terms, and the names of tools the user should not know about do not belong there.
Excessive agency is the control that matters most
The highest-consequence security decision in an enterprise GenAI deployment is what the system is permitted to do without a human, and most teams make it implicitly.
OWASP’s Excessive Agency (LLM06:2025) covers three separate over-grants, and they need separating because the fixes differ:
| Over-grant | What it looks like | Fix |
|---|---|---|
| Excessive functionality | A tool that can do more than the use case needs — a “run query” tool that accepts arbitrary SQL | Narrow, purpose-built tools with explicit schemas |
| Excessive permissions | The agent’s identity can reach systems irrelevant to its task | Least-privilege role scoped to the workflow, reviewed |
| Excessive autonomy | High-consequence actions execute with no approval | Approval design per decision, keyed to reversibility |
The practical test for the third is reversibility, not importance. An action that can be undone cheaply tolerates autonomy. An action that posts to a closed period, moves money, sends a message to a customer, or deletes a record does not — regardless of how confident the system is, because confidence is not calibrated to consequence.
A single agent should operate at different autonomy levels across its own workflow. Designing this per decision rather than per system is what makes the deployment approvable in a regulated environment.
Identity, and why shared service accounts fail the audit
Every agent authenticates as itself, with a dedicated service identity and least-privilege roles — never through a human user’s credentials and never through a service account shared with an integration.
Without this, three things become impossible and one becomes inevitable. Impossible: distinguishing agent actions from human ones in an audit trail, revoking the agent’s access without breaking something else, and scoping permissions to the agent’s actual task. Inevitable: the agent inherits a permission bundle designed for a trusted human, which is how an automated step ends up able to satisfy both sides of a segregation-of-duties control.
Practical requirements:
- One identity per agent, not per team or per platform.
- Roles derived from the workflow’s actual steps, reviewed when the workflow changes.
- Credential rotation that does not require a code change.
- Read and write capability separated, with different approval paths.
- Every action logged against that identity with enough context to reconstruct the decision.
That last point is architectural rather than operational. Logging that records “agent updated record 4471” without the retrieved context, the prompt version and the reasoning is not an audit trail; it is a receipt.
The supply chain nobody inventories
A GenAI system’s supply chain includes model providers, model weights, embedding models, vector stores, orchestration frameworks, prompt libraries and any MCP or tool server it connects to — and most organizations have no inventory of it.
OWASP lists Supply Chain (LLM03:2025) and Data and Model Poisoning (LLM04:2025) as separate risks because the exposure is different at each layer. What to establish:
Know what models are in use, including indirect ones. Embedding models and rerankers are frequently pulled from public repositories by a library default and never recorded.
Pin versions. A framework or model that updates silently changes system behavior outside your release process.
Treat third-party tool servers as remote code. A tool the agent can call is functionality inside your trust boundary, governed by whoever maintains it.
Record where training or fine-tuning data came from. If the organization fine-tunes, the provenance of that data is a poisoning surface and a licensing question at once.
The related discipline — securing code that AI systems generate rather than the systems themselves — is covered in AI-generated code and the OWASP Top 10.
What to monitor, and what monitoring will not tell you
Monitor behavior at the boundaries: what the system retrieved, what it was asked, which tools it called with what arguments, and what it returned. Model-level metrics tell you very little about whether something has gone wrong.
A workable minimum:
- Tool invocation logs with full arguments — a recurring production incident in agent systems is a tool called with plausible but wrong arguments, and this is the only place it is visible.
- Retrieval logs showing which documents were returned for which query and under whose permissions.
- Refusal and escalation rates, tracked over time — a sudden change usually indicates something upstream changed.
- Spend and volume per tenant and per identity, with limits that trigger before the budget does.
- Output-handling failures — encoding errors, rejected outputs, schema violations.
What this will not give you is detection of a well-formed malicious instruction that produced a well-formed harmful action. That is what the autonomy design is for, which is why approval gates are a security control rather than a usability compromise.
Build Secure, Governed AI Workflows
Integrate GenAI into enterprise systems and business processes with clear access controls, human oversight, and operational visibility.
Who owns this
Assign an owner before the first system ships, because GenAI security falls between existing functions by default.
Application security owns the code. Infrastructure security owns the platform. Data governance owns the data. A GenAI system’s risks sit in the gaps: a retrieval layer that respects permissions, a tool schema that constrains agency, an approval design that matches consequence. Nobody’s existing remit covers those.
The workable pattern is a named owner in security who is embedded with the delivery team from design, plus a standing review at two points — before write access is granted, and before autonomy is increased. Both are decision points with a clear before and after, which makes them enforceable in a way that periodic review is not.
For how these controls are built into agent systems in delivery, see AI agent development; for the identity, permission and audit path into systems of record, AI integration services.
Frequently asked questions
What is LLM security?
LLM security is the practice of controlling what a language-model-based system can reach, do and reveal — through permission scoping, tool design, output handling and approval gates — on the assumption that the model itself cannot be relied on to enforce any restriction it is given in text.
Why is the model not a trust boundary?
Because a model interprets instructions from any text it processes, including text in retrieved documents an attacker may have authored. Controls therefore sit between the model's output and the components that act on it, not inside the model.
What is excessive agency in an AI system?
OWASP LLM06:2025 covers three over-grants: excessive functionality (tools that can do more than the task needs), excessive permissions (identity reaching irrelevant systems), and excessive autonomy (high-consequence actions running without approval). Reversibility, not importance, is the practical test for the third.
How should an AI agent authenticate to enterprise systems?
Through a dedicated service identity with least-privilege roles scoped to the workflow — never a human user's credentials and never a service account shared with an integration. Without this the audit trail cannot distinguish agent actions from human ones.
What should be monitored in a production GenAI system?
Tool invocations with full arguments, retrieval logs with the permissions applied, refusal and escalation rates over time, spend and volume per tenant and identity, and output-handling failures. Model-level metrics say very little about whether something has gone wrong.
Which standards apply to enterprise GenAI security?
The OWASP Top 10 for LLM Applications (2025) as the risk taxonomy and the NIST AI Risk Management Framework with its Generative AI Profile (NIST-AI-600-1, July 2024) as the governance structure. Where the EU AI Act applies, its obligations carry specific dates and should be confirmed with counsel.
Strengthen Your GenAI Security Controls
Talk to an AI Expert
