Responsible AI, Governance, and LLMOps

As AI systems move from isolated experiments into the operational core of enterprise platforms, responsible AI transforms from a compliance checkbox into a foundational architectural concern. This chapter establishes the governance framework that makes enterprise AI sustainable — covering privacy, compliance, explainability, and human oversight controls — and introduces the LLMOps discipline that closes the loop between model development and production reliability.

Foundations of Enterprise AI Governance

Enterprise AI governance is a multi-layer control architecture spanning data handling, model behavior, operational telemetry, and organizational accountability. It draws from established disciplines — data governance, software supply chain security, risk management — and adapts them for the unique characteristics of probabilistic AI systems: non-determinism, emergent behavior, and the difficulty of explaining individual outputs.

The Five Pillars of Responsible AI on Azure

Azure Responsible AI is organized around five interdependent pillars: Fairness, Reliability and Safety, Privacy and Security, Inclusiveness, and Transparency. Each maps to concrete Azure services that architects must wire into designs from the start. Fairness uses Azure Machine Learning (Azure ML)'s Responsible AI Dashboard with Fairlearn to quantify performance disparities across sensitive features; fairness metrics are encoded as automated gates in the model evaluation pipeline. Transparency requires Azure Model Cards and a model transparency portal aggregating evaluation reports and lineage graphs for governance stakeholders.

Privacy by Design in AI Architectures

Privacy by design for AI requires applying its seven foundational principles at every layer: data ingestion, training, evaluation, deployment, and inference. The most common failure is treating privacy as a data-at-rest concern while ignoring that trained models can memorize PII, fine-tuned models can be inverted, and RAG systems can surface confidential documents through indirect injection.

For training pipelines: data minimization, anonymization (differential privacy via SmartNoise SDK), and label-based access policies in Azure Data Lake Storage Gen2. For inference pipelines: input PII redaction, output PII filtering, and immutable audit logging of every request-response pair with caller identity.

Compliance Frameworks and Regulatory Alignment

The 2026 regulatory landscape spans the EU AI Act (effective February 2025, phased obligations through August 2026), NIST AI RMF 1.0, ISO/IEC 42001, and sector-specific requirements (HIPAA, PCI DSS, FedRAMP). The EU AI Act's risk classification — Unacceptable, High Risk, Limited Risk, Minimal Risk — drives conformity assessment requirements. Most enterprise AI use cases (AI-assisted hiring, credit scoring, medical diagnosis support) fall into High Risk.

Content Safety, Model Governance, and AI Risk Management

Content safety in enterprise AI is a multi-layer defense architecture. Azure AI Content Safety provides the foundational moderation service, but architects must design content safety as a pipeline spanning input validation, system prompt enforcement, output classification, and human escalation — each layer catching what the previous layer misses.

Azure AI Content Safety Architecture

Azure AI Content Safety provides moderation APIs covering Hate, Sexual, Violence, and Self-Harm categories with configurable severity thresholds (0–7, typically enforced at ≥ 2 for enterprise). The pipeline pattern: user inputs pass through Content Safety before reaching the LLM; system prompts enforce role boundaries and prohibited topics; outputs pass a second Content Safety pass plus custom domain-specific classifiers. Groundedness detection addresses factual hallucinations in RAG — a low grounding score on a response signals the model is not citing retrieved documents.

Model Governance: Lifecycle, Versioning, and Retirement

Azure ML Model Registry is the governance hub. Every registered model carries mandatory metadata: model card URI, training dataset lineage, evaluation report URI, Responsible AI Dashboard snapshot, approvals with identity and timestamp, and deployment constraints. The registry enforces a lifecycle state machine: Experimental → Staging → Production → Deprecated → Retired.

AI Risk Management and the AI Risk Register

The NIST AI RMF's four functions — Govern, Map, Measure, Manage — translate directly into architectural artifacts. The AI Risk Register catalogs identified risks with likelihood, impact, Azure mitigation, and residual risk acceptance. Each entry maps to a specific Azure service implementing the control, enabling the register to serve simultaneously as a compliance artifact and operational runbook.

AI Architecture Review Framework

The AI Architecture Review Framework (AI ARF) extends the Azure Well-Architected Framework with AI-specific pillars and gates calibrated to system risk tier. It defines four lifecycle gates: Design Review, Pre-Deployment Review, Production Readiness Review, and Quarterly Operational Review.

Governance Gates and Review Criteria

The Design Review gate requires: EU AI Act risk tier classification, data flow diagram with sensitivity labels, privacy impact assessment, model selection rationale, evaluation strategy, and human oversight design. Approval is required before provisioning production Azure ML workspaces or Azure OpenAI PTU reservations.

The Pre-Deployment Review gate requires: completed Responsible AI Dashboard, content safety configuration tested, evaluation report meeting all thresholds, security threat model reviewed, compliance attestation signed by DPO (for GDPR-scoped systems), and deployment runbook reviewed by operations. Only then may a PR be merged to the deployment branch.

Review Workflows and Approval Automation

Manual review creates bottlenecks that drive bypass behavior. The AI ARF is implemented as an Azure DevOps pipeline where governance artifacts are submitted as pull requests, policy-as-code checks run automatically, and human approvals are recorded with identity and timestamp. Each lifecycle gate is a pipeline stage with environment approval gates: automated Azure Policy compliance scans, Responsible AI Dashboard completeness checks, and evaluation metric threshold validation all run before the human approval gate opens.

Versioning Strategy: Prompts, Models, and Datasets

Versioning in LLMOps extends software version control to three new artifact types — prompts, models, and datasets — each with distinct versioning semantics. Without a coherent versioning strategy, AI system changes are unauditable, regression testing is impossible, and rollback under incident conditions fails.

Prompt Versioning and Management

Prompts are software artifacts that directly determine model behavior, yet are typically stored as plain text strings with no versioning discipline. A Prompt Registry implemented as Azure Cosmos DB stores prompts with schema fields: prompt_id, version (semantic version), content, model_compatibility, status (Draft / Review / Approved / Deprecated), approved_by, and approval_timestamp. Applications fetch prompts by name and explicit version — never a floating latest reference in production.

Model Versioning and Lineage

Azure ML Model Registry provides native version tracking. The lineage discipline connects each model version to its training dataset version, training code Git commit hash, hyperparameter configuration, and evaluation results. MLflow captures this automatically: mlflow.log_param(), mlflow.log_artifact(), mlflow.log_metrics(), and dataset version tags form the evidentiary chain regulators require to reconstruct how a production model was built and validated.

Dataset Versioning and Data Lineage

Three dataset types require versioning: training datasets, evaluation datasets (golden sets, adversarial test sets, RAG benchmarks), and retrieval datasets (RAG corpora). Azure ML Data Assets provide immutable versioning for training and evaluation datasets; Microsoft Purview provides the end-to-end lineage graph from source systems through transformation pipelines to final training datasets.

AI CI/CD: Evaluation Pipelines, Safety Gates, and LLMOps

LLMOps applies DevOps principles — continuous integration, continuous delivery, automated testing, observability, and incident response — to the lifecycle of AI systems. The AI CI/CD pipeline is the executable implementation of the AI ARF, automating every governance check that can be automated and surfacing human review at gates that cannot.

Evaluation Pipeline Architecture

The evaluation pipeline runs on every pull request modifying a prompt, model, or retrieval corpus. Five stages: Dataset Preparation (fetch pinned evaluation dataset version, run contamination check); Model Inference (batch inference via Azure ML); Metric Computation (RAG: Answer Relevance, Groundedness, Context Precision, Recall; classification: Accuracy, F1, AUC); Threshold Validation (compare metrics against governance YAML thresholds); Report Generation (HTML artifact uploaded to Azure ML, MLflow run tagged PASS/FAIL).

RAG Quality Testing

RAG evaluation uses question-answer-context triples and measures four metrics: Retrieval Recall@k, Retrieval Precision@k, Answer Groundedness, and Answer Correctness. The Azure AI Evaluation SDK provides RelevanceEvaluator, GroundednessEvaluator, CoherenceEvaluator, and FluencyEvaluator as base implementations. Extend these with domain-specific evaluators — ComplianceEvaluator, CitationEvaluator — as custom Python classes following the SDK evaluator interface.

Safety Gate Automation

Safety gates are binary: a pipeline producing content safety violations must not proceed to production regardless of quality score improvements. The safety evaluation dataset is a versioned adversarial test set covering jailbreak attempts, indirect injection patterns, boundary-testing queries, and domain-specific harmful queries. The gate passes only if the harmful response rate on the adversarial dataset is zero for high-risk systems.

Lab

SUBSCRIPTION_ID="<your-subscription-id>"
LOCATION="eastus2"
RG_PROD="rg-responsible-ai-governance-llmops-prod-001"
AI_CONTENT_SAFETY_NAME="responsible-ai-prod-eastus2-001"

az account set --subscription "$SUBSCRIPTION_ID"
az group create --name "$RG_PROD" --location "$LOCATION" \
  --tags environment=prod complianceframework=eu-ai-act dataclassification=confidential

az policy assignment create --name "restrict-ai-data-residency-prod" \
  --policy "e56962a6-4747-49cd-b67b-bf8b01975c4f" \
  --resource-group "$RG_PROD" \
  --params '{"listOfAllowedLocations":{"value":["eastus2","eastus"]}}'

az cognitiveservices account create \
  --name "$AI_CONTENT_SAFETY_NAME" --resource-group "$RG_PROD" \
  --kind ContentSafety --sku S0 --location "$LOCATION" --yes

az cognitiveservices account update \
  --name "$AI_CONTENT_SAFETY_NAME" --resource-group "$RG_PROD" \
  --public-network-access Disabled
# Store key in Key Vault — retrieve with az cognitiveservices account keys list

ML_WORKSPACE="aml-responsibleai-prod-eastus2-001"
MODEL_NAME="customer-support-rag-v2"

az ml workspace create --name "$ML_WORKSPACE" --resource-group "$RG_PROD" \
  --managed-network allow_only_approved_outbound --public-network-access Disabled

az ml data create --name "rag-golden-evaluation-dataset" --version "3.1.0" \
  --workspace-name "$ML_WORKSPACE" --resource-group "$RG_PROD" \
  --type uri_file --tags contamination_check_passed=true

az ml model create --name "$MODEL_NAME" --version "2.0.0" \
  --workspace-name "$ML_WORKSPACE" --resource-group "$RG_PROD" \
  --tags base_model="gpt-4o-2024-11-20" lifecycle_stage=Production \
         aarb_approval_id="AARB-2026-0614-007"
# Submit evaluation pipeline job and gate on EVAL_STATUS == "Completed"

COSMOS_ACCOUNT="cosmos-promptregistry-prod-eastus2-001"

az cosmosdb create --name "$COSMOS_ACCOUNT" --resource-group "$RG_PROD" \
  --locations regionName="$LOCATION" failoverPriority=0 isZoneRedundant=true \
  --enable-public-network false --backup-policy-type Continuous

az cosmosdb sql container create --account-name "$COSMOS_ACCOUNT" \
  --resource-group "$RG_PROD" --database-name promptregistry \
  --name prompts --partition-key-path "/promptName"

az monitor scheduled-query create \
  --name "alert-content-safety-high-severity" \
  --resource-group "$RG_PROD" --scopes "$LA_WORKSPACE_ID" \
  --condition "count > 0" --severity 1 \
  --window-size 5 --evaluation-frequency 5
# Prompt v1.3.0 registered via Cosmos DB data plane REST API

Summary

Chapter: 11 of 12  |  Chapter Status: v0.1 Draft  |  2026-07-08