Prompts Are Not Governance: Policies That Survive Model Upgrades

Someone on the team rewrites the system prompt, ships it, and feels like governance is done. I get the temptation. Prompts are visible. They feel like control. Then the vendor ships a new model name, behaviour shifts, and the carefully worded instructions stop doing what you thought they did.

Prompts are instructions to a model. Governance is how the organization decides who may use what, on which data, with what review, and how you prove it later. Those are different jobs.

What collapses when the model changes

Model upgrades can change tone, refusal patterns, tool-calling habits, and how strictly the model follows long instructions. Your “always cite the policy PDF” line might get softer. Your length limits might be ignored more often. Edge-case refusals might flip.

If the only control is the prompt text, you discover drift when a customer already saw the bad output. That is late.

Treat prompts as configuration, not policy. Configuration gets versioned and retested. Policy states outcomes and boundaries that stay true even if you swap models next quarter.

Usage rules that people can actually follow

Write rules in the language of work, not model internals:

• Which data classes may enter which tools (public web chat, company tenant, fully internal systems).
• Which tasks are allowed for assist-only drafts versus direct send to clients or systems of record.
• What must never be pasted into consumer tools (credentials, personal data, unreleased financials, client secrets).
• Who is accountable for the final decision when AI helped draft it.

Keep the list short enough that a busy manager can explain it in five minutes. Long policy PDFs that nobody opens do not govern anything.

Logging is how you learn and defend

You do not need to store every token forever. You do need enough history to answer: what was asked, what context was attached, what came back, who approved it, and which version of the app and model was in play.

Useful minimums for many mid-market setups:

• User or service identity (or a team proxy where individual tracking is too heavy).
• Timestamp, feature or workflow name, and model identifier used.
• Redacted or classified handling of sensitive fields.
• Link to the business object when the output touched a ticket, claim, or case.

Without logs, “the AI did something weird” stays a story. With logs, it becomes a fixable defect. You also get a clearer picture of which features people actually use, which is useful when someone asks for more budget without evidence.

Approval standards that do not depend on model branding

Define approval by risk of the outcome, not by which logo is on the API:

• Low risk: internal brainstorming, first drafts, summarization for the author only.
• Medium risk: customer-facing text that a human must review before send.
• High risk: anything that updates money, eligibility, legal positions, or regulated records needs named approvers and a clear definition of done.

When a new model lands, re-run the same acceptance checks. Do not rewrite governance from scratch because marketing renamed the model. If you evaluate outputs with BA-style criteria, the checks travel better than “it felt smarter this week.”

Version prompts, evals, and policies as separate layers

A pattern that holds up:

1. Policy layer: allowed uses, data rules, review requirements (slow-changing).
2. Prompt and workflow layer: instructions, tools, retrieval settings (change with features).
3. Evaluation layer: fixed scenarios and pass/fail rules you re-run on upgrades.

When the vendor pushes a new default model, you retest layer three against layer one. You may tweak layer two. You should not invent a new philosophy of control every release.

Company knowledge retrieval needs the same discipline. A RAG setup can still surface stale or wrong documents if nobody owns content quality. Governance includes who updates the knowledge base, not only who wrote the system message.

A practical takeaway

If your control story is “we have a strong prompt,” you have a starting point, not a control framework. Add usage rules people can recite, logging you can query, and approval standards tied to business risk. Re-test when models change. Keep data ownership and access rules explicit so tools do not become a quiet second copy of the enterprise.

Hype cycles will keep renaming capabilities. Policies that describe work, risk, and accountability tend to survive the rename. Prompts alone usually do not.