AI Agent Deployment Architecture: Cloud, On-Prem and Hybrid

An enterprise AI agent has three deployable components: the inference endpoint that runs the model, the orchestration runtime that holds the loop and the state, and the data the agent reads and writes. Deployment architecture is the decision about where each one lives — and they do not have to live together.

Key takeaways

• An agent is not one deployable thing. Inference, orchestration, and data each have their own placement decision.
• Most enterprise deployments are hybrid by construction: orchestration near the systems of record, inference wherever the residency rules allow.
• Residency is a hard constraint and settles the question. Latency, cost, and control are trade-offs that get optimized afterwards.
• Self-hosting a model moves the data boundary and moves the operating burden with it. It does not, on its own, make a deployment safer.
• Write down where data crosses a boundary. That diagram, not the deployment label, is what a security review actually assesses.

Treating “cloud versus on-prem” as one choice is what produces the two common failure modes. A team concludes that data sensitivity requires on-premise inference, self-hosts a model, and inherits a GPU operations problem it has no team for. Or a team deploys everything on a vendor’s cloud and discovers at security review that the orchestration layer’s logs hold regulated content in a region the policy does not allow.

This piece separates the three placement decisions, gives the constraint order that resolves them, and sets out what each model actually costs to run.

What are the three placement decisions?

Inference, orchestration and data. Each has its own drivers.

The second row is the one that gets least attention and causes the most rework. Orchestration is where tool calls originate, so it needs network reach to the ERP, the document store and the identity provider. Placing it far from those systems adds a round trip to every tool call and usually a firewall exception to every integration. Orchestration wants to be close to the systems of record; inference does not care where it is called from.

The third row is where compliance findings come from. Retrieval indexes, memory stores and trace logs contain content drawn from the systems the agent reads. They inherit the classification of that content, and they are frequently deployed with the runtime rather than with the data policy.

The four deployment models

Provider API. The model runs on the provider’s infrastructure; you call an endpoint. Lowest operational burden, fastest access to new model versions, and the highest bar to clear on data handling — contractual terms on training, retention and processing region have to satisfy your policy for the real data, not the pilot’s.

Cloud-hosted in your tenant. A managed model service inside your own cloud account and region. Data stays within your cloud boundary and your existing controls — network policy, key management, logging — apply. This is where a large share of regulated enterprise deployments land.

Self-hosted in your infrastructure. You run the model on your own GPUs, in your data center or a private cloud. Maximum control over the data path, and a genuine operating commitment: capacity planning, serving stack, upgrades, and a capability ceiling relative to frontier hosted models.

Air-gapped. No egress at all. Reserved for environments where it is mandatory. Everything — model weights, updates, evaluation data — moves by controlled process, and the operating cost reflects that.

How do you choose a deployment model?

In constraint order. One of these is a hard stop; the rest are trade-offs.

1. Residency and regulatory placement.

Where is this data legally and contractually permitted to be processed? This is a yes-or-no question and it eliminates options rather than scoring them. Where an EU regulatory regime applies, the governing text is the place to check rather than a summary of it — the EU Artificial Intelligence Act is published in full on EUR-Lex.

2. Network reach.

Which systems must the orchestration layer call, and where do they sit? Systems inside a private network effectively decide where orchestration runs.

3. Latency budget.

What is the end-to-end target, and how much of it is model time versus tool time? In most enterprise agents, tool round trips and retrieval dominate the budget — moving inference closer is rarely the largest available saving.

4. Capability requirement.

Does the workflow need a frontier model’s reasoning, or is it extraction, classification and routing that a smaller open-weight model handles well? This often splits: a hosted model for reasoning, a local one for high-volume utility steps.

5. Cost at volume.

Per-token pricing against the fully loaded cost of self-hosting — GPUs, serving stack, utilization, and the people. Self-hosting economics depend on sustained utilization; intermittent workloads rarely justify it.

6. Operating capability.

Does a team exist that can run a model serving stack, and is running one a good use of them?

The hybrid pattern, and why most enterprises end up there

Orchestration inside the enterprise boundary, inference wherever policy allows, data governed where it is generated.

The common shape: the orchestration runtime runs in the cloud tenant or data center where the systems of record are, so tool calls are local and existing network controls apply. The retrieval index and audit log sit inside the same governed boundary. Inference is called out to a managed endpoint in an approved region for reasoning steps, with a smaller self-hosted or in-region model handling classification and extraction volume.

This shape is not a compromise between two ideals. It reflects the fact that the three components have genuinely different drivers, and forcing them into one location optimizes none of them.

Two rules keep it defensible:

• Redact before the boundary, not after. If content is classified above what the inference endpoint is approved for, it should be redacted or tokenized by the orchestration layer before the call, not filtered from the response.
• Every fallback inherits the same policy. A compliant primary model with a non-compliant fallback is a compliance gap that appears under load, which is exactly when nobody is reviewing it. This is the same failover discipline described in the model layer walkthrough.

Draw the data-crossing diagram

The artefact a security review actually needs is a diagram of where data crosses a boundary, annotated with what data and under which control.

It is a short exercise and it surfaces most findings before the review does. For each arrow in your architecture, record four things: what data class travels on it, which boundary it crosses, what protects it in transit and at rest on the far side, and how long it is retained there. Six arrows is typical for a first agent — request content to the inference endpoint, retrieved documents into the prompt, tool arguments to each system of record, results back, trace data to the logging store, and durable memory writes.

Two of those are usually the surprise. Trace and observability data contains prompt and tool-argument content, so it inherits the classification of whatever the agent read. Durable memory accumulates the same, as covered in AI agent memory. Both routinely land in a logging or storage tier chosen for operational convenience rather than data policy.

The control set that has to hold across all of it is set out in securing enterprise GenAI.

What each model costs to operate

The asymmetry to plan for: hosted inference costs scale with success, and self-hosted inference costs are largely fixed against provisioned capacity. That makes self-hosting attractive at sustained high volume and expensive at low or spiky volume — which is the profile of most first deployments. Measuring for a quarter before committing capital is cheaper than the reverse.

Where this leaves an architecture decision

Place the data first, the orchestration second, and the inference last.

That order inverts how the decision is usually framed, and it is the order that produces architectures which pass review. Data placement is constrained by policy and is therefore not really a choice. Orchestration placement follows from what it has to reach. Inference placement is the one with genuine optionality — and it is the one most teams decide first, then work backwards from.

GrowExx designs and implements enterprise agent deployments across all four models, including the data-crossing analysis that a security review will ask for. See AI implementation services and AI agent development, or talk to us about a deployment and data-boundary review before your architecture goes to security.

Frequently asked questions