Enterprise AI architecture is the shared platform that several AI systems run on: the model access layer, the retrieval and data plane, the identity and policy plane, the integration layer and the evidence layer. It is what stops the fifth use case rebuilding the first four.
Key takeaways
- The architecture question is not how one agent is built. It is which parts are built once and which are built per use case.
- Five planes are worth centralizing: model access, retrieval and data, identity and policy, integration, and evidence. Everything else belongs to the use case.
- Centralize a component when a mistake in it is expensive and repeated. Leave it local when the use cases genuinely differ.
- The integration layer is the one that decides delivery speed. Most AI programs slow down at the third system of record, not at the third model.
- A platform with no funded owner is a set of libraries. Funding decides whether it stays coherent after the second team adopts it.
The first enterprise AI use case does not need an architecture. It needs a model, a data source and a person who can ship. The second one can usually be built the same way. By the fourth or fifth, three things have happened: each team has made its own decision about how to call a model, nobody can answer which systems hold embeddings of customer data, and each new integration into the ERP takes as long as the first one did.
That is the point at which the architecture question becomes real, and it is a question about what is shared rather than about what is built.
What is enterprise AI architecture, and what does it actually decide?
Enterprise AI architecture decides which capabilities are provided once to every AI system in the organization, and which are the responsibility of each use case. Everything else follows from that split.
It is not a diagram of one agent, and it is not a vendor stack. The useful form is a statement of ownership: this team runs model access for everyone, this team runs the retrieval platform, this team owns the connectors into the systems of record, and this is what a use-case team has to bring. The layers inside any individual agent, from foundation model to human oversight, are a separate subject, covered in our anatomy of a production AI agent.
| Question the architecture answers | Consequence of leaving it to each team |
|---|---|
| How does a system call a model? | Per-team key management, no central cost view, and a model deprecation becomes several migrations |
| Where do embeddings live, and under what access policy? | Nobody can answer which stores hold regulated data |
| What identity does a system act as? | Every team invents its own service account pattern |
| How does a system reach the ERP? | The third integration costs as much as the first |
| Where do traces go? | Incidents cannot be investigated across systems |
| What has to be true before production? | Readiness means something different on every team |
Which planes should be shared?
Five planes earn centralization because a mistake in each is both expensive and repeated; the rest should stay with the use case.
Model access. One gateway through which systems reach models. It carries credentials, routing, quota, cost attribution and version pinning. The argument for it is not cost savings; it is that a provider deprecating a model becomes one change rather than eleven, and that somebody can answer what the organization spends and on what.
Retrieval and data. Shared embedding infrastructure, a connector set to the governed sources, and the entitlement check applied at retrieval time. This is the plane with the strongest centralization case, because it is where access control and classification live, and getting it wrong per team means getting it wrong repeatedly.
Identity and policy. Agent identities issued from the corporate directory, permission scoping patterns, and the enforcement point for approval gates. Centralizing the pattern matters more than centralizing the service: each team can run its own orchestrator provided the gate is enforced in code rather than in a prompt.
Integration. Connectors to the systems of record, with their authentication, rate handling, idempotency and error semantics solved once. This is the plane that determines how fast the sixth use case ships.
Evidence. Trace collection, the evaluation harness, and the record that satisfies audit. Shared because an incident crosses system boundaries and a per-team trace format makes that investigation impossible.
What should stay with the use case?
Everything that encodes how a specific business process works should stay local, because centralizing it produces a platform team that becomes a queue.
The prompts and the task decomposition belong to the people who understand the process. The evaluation cases belong to them too, because only they know what a wrong answer costs in that workflow. The workflow logic, the human handoff points and the user interface are theirs. So is the decision about which tools the agent needs, within the permission patterns the platform sets.
The line is worth stating precisely, because getting it wrong in the centralizing direction is the more common failure and the harder one to reverse:
| Capability | Shared | Per use case |
|---|---|---|
| Model credentials, routing, version pinning | Yes | |
| Prompt content and task decomposition | Yes | |
| Embedding infrastructure and connectors to governed sources | Yes | |
| Which documents this use case indexes | Yes | |
| Agent identity issuance and scoping patterns | Yes | |
| Which tools this agent holds | Yes | |
| Approval gate enforcement mechanism | Yes | |
| Which actions need a gate, and who approves | Yes | |
| Trace collection and storage | Yes | |
| Evaluation cases and pass criteria | Yes |
A platform team that owns the right column becomes a bottleneck within two quarters. A use-case team that owns the left column rebuilds it badly.
How do you decide when to centralize something?
Centralize when the cost of getting it wrong is high and the chance of getting it wrong is repeated across teams. Both conditions have to hold.
Model access qualifies: a leaked key or an unnoticed spend is expensive, and every team faces the same decision. Entitlement at retrieval qualifies strongly: a wrong answer there is a data incident, and every team implementing it separately will implement it differently. Prompt content does not qualify: a bad prompt is cheap to fix and the decision is not the same across teams.
A third test is worth applying before building anything: would a use-case team adopt this if it were optional? A platform component that teams route around is not a platform component, it is an internal product with no users, and the usual reason is that it was built to a governance requirement rather than to a delivery need.
Where do enterprise AI programs actually slow down?
They slow down at the third system of record, not at the third model, and the architecture should be shaped around that fact rather than around model choice.
Calling a different model is a configuration change. Reading from and writing to an ERP, a CRM, a ticketing system and a data warehouse, each with its own authentication, its own rate behavior, its own idempotency story and its own release calendar, is engineering work that does not get cheaper by changing model. The programs that ship their sixth use case quickly are the ones that treated connectors as a platform asset from the second one.
This is also where the quarterly release cycles of enterprise suites intrude. An integration that works today has to keep working after the vendor’s next update, which makes regression testing of connectors a platform responsibility rather than a use-case one.
Who funds and runs the platform?
A named team with its own budget runs it, and if that does not exist, the architecture is a document rather than a platform.
The failure mode is well known from internal developer platforms and it repeats here exactly. A platform built as a side project by whichever team needed it first serves that team’s use case and nobody else’s. When the second team adopts it, they ask for changes the first team has no time to make, so they fork it. Within a year there are three variants, which is the state the platform was meant to prevent.
The test is simple to apply: does a named person own this, with budget, and are they measured on adoption by other teams rather than on their own use case shipping. If not, plan for divergence and keep the shared surface deliberately small, because a small shared surface that holds is better than a large one that forks.
The decision
Share the five planes, keep the process logic local, and apply the two-condition test before adding anything else to the shared surface. If the program is at its second or third use case, the component to build first is the integration layer, because that is where the sixth use case will be delayed, and it is the one component a use-case team cannot build well enough on its own.
GrowExx builds the shared planes and the connectors that sit under them. Our AI consulting practice works on the shared-versus-local split before anything is built; our AI agent development team builds the platform components and the use cases that run on them.
FAQs
What is enterprise AI architecture?
Enterprise AI architecture is the shared platform several AI systems run on, covering model access, retrieval and data, identity and policy, integration with systems of record, and the evidence layer of traces and evaluation. It defines which capabilities are provided once to every AI system and which belong to each use case.
How is an AI reference architecture different from an individual agent's architecture?
An agent's architecture describes the layers inside one system, from the model through context, tools, orchestration and oversight. An enterprise AI architecture describes what sits beneath several such systems and is built once: the model gateway, the retrieval platform, identity issuance, the connector set and trace collection.
Which parts of an enterprise AI platform should be centralized?
Centralize model access, retrieval and data infrastructure, identity and policy patterns, integration connectors and the evidence layer. Keep prompts, task decomposition, workflow logic, tool selection, approval criteria and evaluation cases with the use-case team that understands the business process.
When is it too early to build an AI platform?
At the first or second use case. The platform question becomes real when a third or fourth team starts making the same decisions independently, and the signal to watch for is a second team asking for an integration that already exists somewhere else in the organization.
Why do enterprise AI programs slow down as they scale?
Because the constraint is integration with systems of record rather than model capability. Each additional ERP, CRM or ticketing integration brings its own authentication, rate behavior, idempotency requirements and release calendar, and that work does not get cheaper by changing model.
