Most AI governance lives in a slide deck or a policy PDF nobody reads at deploy time. That document does not stop a model from leaking PII, does not catch silent drift, and does not tell an auditor who approved which model on what data. Real governance is a stack of controls wired into the pipeline, where each control removes a specific, named risk. If a control is not enforced in code at a point where the pipeline can block, it is theater.
The eight controls and the risk each removes
Map every governance claim to a tool and the failure it prevents. If you cannot name the failure, delete the control.
- Data lineage: trace every feature to source so you can answer "why did the model say that" and reproduce a decision. We use OpenLineage with dbt and Airflow metadata.
- PII redaction at ingestion: strip sensitive data before any model or log sees it. Microsoft Presidio at the ingestion boundary, not as a later filter. In our EU insurance work, redaction ran before any model touched a claim.
- Access control: scope who reads which dataset and who promotes a model. Postgres row-level security plus a registry permission model, not a shared admin token.
- Model and version registry: pin every deployed model to a version, training data snapshot, and approver. MLflow Model Registry with stage gates.
- Drift and performance monitoring: detect input drift and live metric decay before users do. Evidently or whylogs feeding alerts; backtesting on a schedule.
- Human-in-the-loop: route low-confidence or high-impact outputs to a reviewer. A confidence threshold plus a review queue, sized to volume.
- Audit logging: append-only record of inputs, model version, output, and reviewer for every decision. This is what an auditor actually asks for.
- Eval gates in CI: block promotion when accuracy, a fairness slice, or a safety check regresses against a held-out set.
What to build first, and what is theater
Sequence by regulatory and reputational blast radius, not by ease. Build the controls that prevent an irreversible incident before the ones that make reporting prettier.
- PII redaction at ingestion and access control. A leak is irreversible and the most common cause of a real incident.
- Model registry and audit logging. Without these you cannot answer the two questions every regulator asks: which model, and who approved it.
- Eval gates and drift monitoring. These stop slow degradation, which is the failure mode nobody sees until a metric has already moved.
- Human-in-the-loop last, scoped to the decisions that actually warrant a reviewer.
Theater is a 40-page responsible-AI charter with no enforcement hook, a fairness dashboard nobody is paged on, and a model card written once and never regenerated. The decision rule: a control counts only if it can block a deploy or a request, and only if someone is paged when it fires.
Implementation: enforce at the boundaries
Two boundaries carry most of the load. At ingestion, redaction and lineage run before data lands in any store, so logs and feature tables are clean by construction rather than scrubbed after the fact. At promotion, the registry refuses to advance a model that lacks an approver, a data snapshot, and a passing eval; CI runs the eval gate on every model PR and fails the build on a regressed slice. In our government IDP work on heterogeneous bank statements, the same audit trail that satisfied compliance also gave us the per-document record we needed to debug extraction errors. Governance and observability are the same pipe.
One anti-pattern to avoid: redaction as a downstream filter. Once raw PII is in a log line or a feature store, you have already created the liability. Move it to the boundary.
The business case
A guardrail stack is what lets a regulated client say yes. In finance, insurance, and government engagements, the audit trail, redaction, and eval gates were not overhead, they were the reason the system shipped to production instead of staying a proof of concept. The stack also lowers the cost of every future change: when each control is enforced in code, a new model or feature inherits governance instead of triggering a fresh review. Write the policy if you must, but spend the budget on the eight controls. The PDF documents the stack; it is not the stack.
