Get in touch

Why enterprise AI governance fails when it lives in prompts

A specific class of AI agents is capable enough to deploy and not reaching production. These are not writing assistants or document summarizers. Those ship without incident. These are agents that query production databases, take actions on behalf of authenticated users, and touch regulated data. The block is not capability. It is governance: specifically, governance implemented entirely in the prompt layer, which enterprise security teams correctly identify as no governance at all.

We have had versions of the same conversation with contacts at large organizations across resources, mining, and professional services. The pattern is consistent: AI tools in active use internally, proof-of-concept agents built for privileged workflows, none of them through the governance gate. The review questions are not about model quality. They are: Who authorized this tool call? What was actually executed? Can we see it? Can we block it if we need to?

A system prompt that says "only access data the user is authorized to see" does not answer any of these questions. It describes an intention. It does not constitute a control.

What governance-first architecture looks like

In our work on an income verification platform built for the Australian lending market (one that extracts and validates financial data from payslips on behalf of lenders), we faced this directly. The naive framing is to give the model the documents and a system prompt describing assessment criteria. The problem is that this treats the model as the decision-making layer. In Australian consumer lending, "the AI decided" is not a defensible regulatory position. The model is an extraction layer. The decision belongs in a separate, deterministic layer.

The architecture that actually holds:

After the LLM extracts fields from a document (gross income, payment period, employer, year-to-date figures), a deterministic validation pipeline runs: consistency checks, fraud indicators, income calculations, representativeness scoring. The model's output feeds this pipeline; it does not determine the verdict. A prompt that says "be conservative in your assessment" does not produce an auditable income figure. A deterministic calculation the lender can trace and explain does.

The decision vocabulary is enforced at the code level. The platform returns SATISFACTORY, REVIEW, or UNSATISFACTORY. Never APPROVED or DECLINED. APPROVED would imply a credit decision the platform has no regulatory licence to make. This is not enforced via a prompt that says "don't use the word APPROVED." It is a code-level check on every response, unconditionally.

Every document processed runs exclusively through a regional cloud endpoint that satisfies data sovereignty requirements. Not via a prompt. A credential check that fails the job if the compliant endpoint is not available, regardless of what the model requests.

None of these are prompt instructions. All of them are auditable. All of them are independent of what the model outputs. A compliance team can read them, test them, and verify that they hold for all inputs, not most inputs.

Application-code controls are one tier of the enforcement layer. The harder line is credential-level: an agent that authenticates with a service account scoped to the permitted systems cannot reach infrastructure outside its boundary regardless of what it generates or what the application checks. Application checks establish deterministic logic; scoped credentials make the boundary hold even if the application has a bug. Both belong to the enforcement layer; which depth you build to depends on the risk classification of the constraint.

The frameworks gap

The governance landscape enterprise teams are working from is more developed than it was twelve months ago, and it matters to characterize it accurately.

OWASP released the Top 10 for Agentic Applications in December 2025, a peer-reviewed framework developed with over a hundred industry contributors. It treats agents as execution principals with access to identities, secrets, APIs, and data paths, and it is specifically about post-authorization behavior: what the agent does once it is inside the perimeter, not just how it gets there. Its prescribed mitigations are recognizable: scoped agent identities, least privilege, isolation, observability, human-in-the-loop for high-impact actions. This is the security- engineering artefact the space was missing.

ISO/IEC 42001 is the certifiable management-system standard: third-party audit, a Statement of Applicability, per-control evidence. It requires organizations to define roles, assign control ownership, and demonstrate that AI risks are handled in a controlled, repeatable manner. That is precisely a "who owns it, how do you show an auditor" mechanism.

NIST's Generative AI Profile (AI 600-1) maps twelve risk categories and over two hundred specific actions. A dedicated AI agent interoperability profile is planned for late 2026, narrower than agent governance, but the body of work is accumulating.

The gap is not that nobody has addressed this. The gap is that these three layers are not connected.

OWASP Agentic gives you the threat taxonomy and technical mitigations. It does not give you a build order, or explain that a scoped service credential is categorically harder than an application-code check, or show how the enforcement tiers map to specific agent behaviors. ISO 42001 gives you the certifiable management shell (control ownership, audit evidence, role assignment), but is deliberately implementation-agnostic: it tells you to have control ownership, not what to own or how to build it. NIST is building toward the agent case.

None of them is the practitioner playbook that connects the security layer to the management layer with a concrete build order and a clear answer to: what teams are involved, what gets built when, and how do you demonstrate to an auditor that a scoped credential is the control that holds for all inputs rather than a prompt that holds for most. With EU AI Act enforcement powers beginning August 2026, enterprises are working out what "demonstrable controls" means for agent systems right now. The urgency is real. The integrated answer does not yet exist.

The counterargument is real

Enforcement-layer governance costs more to implement. A code-level check requires a code change and a deployment. A prompt instruction takes minutes to update. For many AI applications (a writing assistant, a summarisation tool, a Q&A interface over documentation), probabilistic behavioural guidance is appropriate. The risk profile does not justify the overhead.

The counterargument is right for those applications. But enterprise governance teams are not reviewing document summarisers. They are reviewing agents that query production databases, call external APIs on behalf of users, and access regulated data. For those use cases, the risk profile is identical to any other privileged software system. And privileged software systems have code-level controls, not prompt-level intentions.

The practical split

The actionable version of this: classify constraints by what happens if they are violated.

Behavioural guidelines (tone, format, whether to ask a clarifying question before acting) belong in prompts. The consequence of a violation is a worse user experience. Iteration is fast. Probabilistic is fine.

Compliance and safety constraints (data access scoping, authorization boundaries, operation restrictions, anything with legal or audit implications) belong in the enforcement layer. The consequence of a violation is a data incident, a compliance breach, or a production failure. You need to be able to demonstrate to an auditor that the control holds for all inputs.

The two layers serve different purposes. The prompt shapes what the model tries to do. The enforcement layer determines what it can do. Governance lives in the second layer.

Knowing that is necessary but not sufficient. The harder questions follow immediately: when in a project do you build the enforcement layer, and what does it actually look like in practice? Those are sequencing and implementation questions, and they have concrete answers the current frameworks don't provide.

case studies

Learn how Australian businesses are maximising value from their AI investments with Evolve bespoke solutions

$1M

ROI

How a Non-Bank Lender Unlocked $1M in Revenue by modernising their risk scorecards in 8 weeks

View case study

120X

faster invoice processing

From Hours to Seconds: Automating Payment Reconciliation in Debtor Finance

View case study
AI Consulting Australia

Looking for a custom product made to fit your business need?

We build a custom solution to maximise your business revenue, reduce costs and add operational efficiency

Speak to an expert