Enterprise Multi-Agent Architecture

Enterprise AI deployments rarely fit the single-agent model. When an organization needs AI to answer HR questions, resolve IT tickets, and reconcile financial transactions while enforcing policy, identity, and data residency requirements, the architecture must distribute intelligence across purpose-built agents while maintaining a coherent user experience. This chapter presents the enterprise multi-agent pattern: a gateway-centric design that routes user intent to domain-specific agents through a policy engine, with observability, circuit breakers, and audit trails that make the system governable at scale.

Foundations: The Enterprise Multi-Agent Architecture

Architectural Overview and Design Principles

The enterprise multi-agent architecture resolves a fundamental tension: users want a single intelligent interface, but enterprise data governance requires strict domain isolation. A layered design lets the shared gateway handle cross-cutting concerns — authentication, intent routing, policy enforcement — while domain agents operate in isolated environments with scoped identities, constrained tool sets, and dedicated data stores.

Five design principles guide this architecture: identity at every boundary (no inter-agent call without a verifiable credential); least-privilege tool access; immutable audit trails written before actions take effect; fail-safe defaults that escalate to humans rather than retry indefinitely; and domain data boundaries enforced by separate Azure OpenAI deployments, AI Search indexes, and storage accounts per domain.

The physical topology spans four layers: User Access (Azure API Management (APIM) + Entra External ID), Gateway (router, intent classifier, policy engine), Domain Agent (HR, IT, Finance Container Apps), and Data Boundary (per-domain storage and vector indexes). Azure Service Bus connects the gateway to domain agents asynchronously; Azure Monitor Application Insights collects telemetry from every component.

Key Azure Services and Their Roles

The table below maps the principal Azure services to their architectural layer and primary responsibility in the multi-agent stack.

The Enterprise AI Gateway Pattern

Agent Router Design

The agent router normalizes incoming requests into a canonical schema, consults the intent classifier to determine target domain, and dispatches messages to the appropriate Service Bus queue. Its stateless function-as-a-service pattern allows horizontal scale-out without session affinity; session continuity is handled via a session token that retrieves conversation state from Cosmos DB without exposing conversation content to the router.

Hard routing rules (policy-driven, principal-group-based) take precedence over soft intent-classification rules. Only ambiguous natural-language queries at /api/agents/chat invoke the intent classifier, keeping deterministic organizational rules out of the LLM decision path.

# Create resource groups (CAF naming)
az group create \
  --name rg-enterprise-multi-agent-architecture-prod-001 \
  --location eastus2 --tags env=prod project=apaf-ch09

# Deploy Service Bus namespace (Premium for VNet integration)
az servicebus namespace create \
  --name enterprise-multi-prod-eastus2-001 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --sku Premium --premium-messaging-partitions 1

# Create domain queues with dead-letter configuration
for domain in hr it finance; do
  az servicebus queue create --name "agent-${domain}-inbound" \
    --namespace-name enterprise-multi-prod-eastus2-001 \
    --max-delivery-count 3 --dead-lettering-on-message-expiration true
done

Intent Classifier Design

The intent classifier is a dedicated low-latency Azure OpenAI deployment (e.g., GPT-4o-mini) that outputs only a domain label (hr, it, finance, ambiguous, out-of-scope) and confidence score. The router acts on the label only when confidence exceeds 0.85 (configurable). Ambiguous cases — such as "I can't log into Workday" which could route to IT or HR — are resolved through explicit policy-encoded few-shot examples, not LLM inference.

Classifier accuracy improves through a shadow-log pattern: every routed message records the label and confidence alongside the actual handling agent. Cases where the agent redirected the user (indicating misroute) are periodically sampled to update the few-shot bank without fine-tuning.

Policy Engine Design

The policy engine enforces rules too sensitive for model prompts: data residency constraints, information barrier policies, time-of-day restrictions, and budget caps. Rules are stored as JSON documents in Cosmos DB with the schema { policyId, principalGroup, targetDomain, allowedActions, deniedPatterns, budgetLimitTokensPerDay, requiresHumanApproval }. A deny decision returns a structured error immediately; an approval-required decision suspends the request and notifies the approver via Azure Logic Apps.

Domain Agent Architecture

HR Agent: People Data with Privacy Controls

The HR Agent runs as an Azure Container App with a user-assigned managed identity holding Storage Blob Data Reader and a custom HR-AI-Search-Reader role. Privacy constraints are enforced at three levels: the system prompt (soft), an APIM policy that inspects outbound tool-call requests (medium), and Azure Policy deny effects on Container App network egress (hard).

The HR Azure AI Search index is populated by an Azure Data Factory pipeline that reads from Workday, strips high-sensitivity fields, and upserts documents. No credential appears in pipeline config — the Workday API key is accessed from Key Vault through managed identity.

# Create Container App environment and managed identity for HR agent
az containerapp env create --name cae-hr-agent-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --location eastus2 --logs-destination azure-monitor

az identity create --name id-hr-agent-prod-001 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001

HR_AGENT_IDENTITY=$(az identity show --name id-hr-agent-prod-001 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --query principalId -o tsv)

az role assignment create --assignee $HR_AGENT_IDENTITY \
  --role "Storage Blob Data Reader" \
  --scope "/subscriptions/<SUB_ID>/.../storageAccounts/sthr<suffix>"

IT Agent: Helpdesk Automation with Action Gating

The IT Agent requires write operations (ticket creation, password reset, license assignment), so its tools are organized into two tiers. Tier 1 (auto-approved) covers read operations; Tier 2 (requires explicit approval) covers mutations. The requiresApproval metadata field in each tool definition triggers the gateway policy engine's approval workflow before execution.

Integration with ServiceNow or Jira uses an APIM facade that translates agent tool-call schemas to downstream ITSM APIs, injects credentials from Key Vault, and logs every mutation to the audit trail. The IT Agent's code never contains ITSM credentials.

Finance Agent: Regulated Data with Immutable Audit

The Finance Agent enforces two controls absent from other domain agents. First, every response containing a numeric figure is tagged requiresReview: true, routing it through an async human-review workflow before delivery; approved responses are cached for 15 minutes. Second, all Azure AI Search queries execute against a read-replica index populated from Azure Synapse Analytics, preventing any direct or indirect modification of financial records.

# Create AI Search for Finance Agent with system-assigned identity
az search service create --name srch-finance-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --sku Standard --replica-count 2

az search service update --name srch-finance-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --identity-type SystemAssigned

FINANCE_SEARCH_ID=$(az search service show \
  --name srch-finance-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --query identity.principalId -o tsv)

az role assignment create --assignee $FINANCE_SEARCH_ID \
  --role "Search Index Data Reader" \
  --scope "/subscriptions/<SUB_ID>/.../srch-finance-prod-eastus2"

Agent Identity and Authorization

Entra ID Managed Identity and RBAC Scoping

Every component runs under a distinct user-assigned managed identity. User-assigned identities decouple the identity lifecycle from the compute resource lifecycle: when a Container App revision is replaced during deployment, the new revision inherits the same identity without re-assignment, eliminating the privilege gap that briefly opens under system-assigned patterns.

Role assignments follow minimum viable permission. The HR Agent needs Storage Blob Data Reader on a specific container, not Contributor on the resource group. A permission matrix stored in the repository and enforced by Azure Policy ensures infrastructure-as-code PRs that expand role scope trigger a mandatory security review.

Tool-Level Permission Scoping

RBAC at the Azure control plane is necessary but insufficient. Each tool definition carries a permission annotation evaluated by the policy engine at invocation time: { "allowedGroups": [...], "deniedGroups": [...], "requiresApproval": false, "auditLevel": "full" }. The auditLevel field controls logging granularity — full records complete inputs and outputs; summary records only tool name and timestamp.

Permission decisions are cached in-memory per agent container with a 60-second TTL to avoid round-trips on every tool call. Cache invalidation is triggered proactively by a Service Bus message published whenever a policy document is updated.

Delegation Chains and the OBO Flow

Cross-domain workflows — for example, an IT provisioning request that must first verify HR employment status — use the OAuth 2.0 On-Behalf-Of (OBO) flow. The gateway acquires a fresh, narrowly scoped OBO token per agent call; no agent receives a token broader than its own application scope. The full delegation chain (user → gateway → HR Agent → gateway → IT Agent) is recorded in Cosmos DB with the correlation ID linking it to the original user request.

Agent-to-Agent Communication

Message Schema and Protocol Design

All agent-to-agent messages use a standardized four-section schema transported over Azure Service Bus. The envelope carries routing metadata; the identity section carries delegated principal info including the delegation chain array; the payload section carries the typed request/response (discriminated by messageType); and the policy section carries the policy engine's decision for this message.

All messages are serialized as JSON and signed with HMAC-SHA256 using a Key Vault-stored key. The receiving agent verifies the HMAC before processing, preventing forged message injection — an attack vector that could impersonate a user with elevated privileges in a multi-agent system.

Audit Trails and Immutable Logging

Every Service Bus message is appended to an audit log in Cosmos DB before the message is removed from the queue. The container is configured with append-only semantics enforced by a stored procedure, and point-in-time restore is enabled with 30-day retention. Audit records capture the full processing result: response time, sanitized tool call inputs (PII stripped), and response classification (answered, escalated, denied, error).

# Create Cosmos DB with continuous backup for audit log
az cosmosdb create --name cosmos-gateway-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --default-consistency-level Session \
  --backup-policy-type Continuous \
  --continuous-tier Continuous30Days \
  --locations regionName=eastus2 failoverPriority=0 isZoneRedundant=true

az cosmosdb sql container create \
  --account-name cosmos-gateway-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --database-name AgentGateway --name AuditLog \
  --partition-key-path "/correlationId" --throughput 400

Agent Observability and Safety

Distributed Tracing and Metrics

Azure Monitor Application Insights provides end-to-end trace visibility through W3C Trace Context propagation. Every service — including Service Bus producer/consumer code — propagates the traceparent header. Custom dimensions (agentId, toolName, tokenCount, policyDecision) enable filtered queries answering operational questions like P99 latency from user query to HR Agent response or intent classifier ambiguity rates.

Token consumption is tracked as a custom metric per Azure OpenAI caller with dimensions including agentId, modelDeployment, and userId (hashed). This feeds an operational dashboard (cost per domain per hour, circuit breaker events) and a governance dashboard (daily token consumption by department, policy denial rates).

Circuit Breakers and Budget Controls

The circuit breaker is backed by Azure Cache for Redis: each agent increments a rolling-window counter per Azure OpenAI call. When the counter exceeds the threshold (e.g., 100 calls in 60 seconds per session), the circuit opens and subsequent calls return CircuitOpenError immediately. The circuit resets to half-open after a cooldown period and fully closes once a probe call succeeds.

Per-domain daily token budgets are stored in the Cosmos DB policy documents. Before each Azure OpenAI call, the agent compares current-day consumption from the audit log against the budget. When exhausted, the agent enters degraded mode — serving queries from Azure AI Search retrieval only, with no generation — and logs a budget-exhaustion event for the platform team's daily report.

Human Escalation and Kill Switches

The human escalation workflow triggers on three conditions: the policy engine's requiresHumanApproval flag, the Finance Agent's requiresReview tag, or an explicit user request. The assigned reviewer accesses the conversation through a dedicated portal showing the full agent reasoning chain, tool calls, and data accessed — not just user-facing messages.

Kill switches operate at three scopes: a session kill switch (broadcasts CancelSession to all queues for the correlation ID), a domain kill switch (sets disabled in the policy engine for one agent domain), and a global kill switch (disables the APIM product subscription, returning 503 to all clients). All three are tested in the quarterly disaster recovery drill.

# Create Application Insights for distributed tracing
az monitor app-insights component create \
  --app appi-agent-gateway-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --location eastus2 --kind web --retention-time 90

# Alert on circuit breaker open events
az monitor metrics alert create \
  --name "alert-circuit-breaker-open-prod" \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --condition "count customEvents/name eq 'CircuitBreakerOpen' > 0" \
  --window-size 5m --severity 1

Lab

# 1. Create Azure OpenAI for intent classifier
az cognitiveservices account create --name aoai-gateway-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --kind OpenAI --sku S0 --location eastus2

# 2. Deploy intent classifier model (GPT-4o-mini)
az cognitiveservices account deployment create \
  --name aoai-gateway-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --deployment-name intent-classifier-v1 \
  --model-name gpt-4o-mini --model-version "2024-07-18" \
  --sku-name GlobalStandard --sku-capacity 10

# 3. Deploy APIM with managed identity
az apim create --name apim-agent-gateway-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --sku-name Developer --enable-managed-identity true \
  --publisher-email chakravarthy_b@infosys.com \
  --publisher-name "APAF Enterprise"

# 1. Create Redis for circuit breaker state (TLS only)
az redis create --name redis-circuitbreaker-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --sku Standard --vm-size c1 --redis-version 7
az redis update --name redis-circuitbreaker-prod-eastus2 \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --set enableNonSslPort=false

# 2. Alert on token budget exhaustion events
az monitor metrics alert create \
  --name "alert-token-budget-exhausted-prod" \
  --resource-group rg-enterprise-multi-agent-architecture-prod-001 \
  --condition "count customEvents/name eq 'TokenBudgetExhausted' > 0" \
  --window-size 15m --severity 2

Summary

Chapter: 9 of 12  |  Status: v0.1 Draft  |  2026-07-08