The most dangerous words in public-sector AI: “let the model decide”
A large language model is the most persuasive intern you have ever hired: articulate, tireless, endlessly confident — and perfectly willing to make something up rather than admit it does not know. That is fine when it is drafting an email. It is dangerous when it is answering for a council.
The failure mode of AI in the public sector is almost never that the model is stupid. It is that the model is fluent and wrong. It will invent a fee that does not exist, promise a callback no one ever scheduled, reassure a resident that a crew is on the way when nothing has been dispatched, and it will do all of it in a warm, plausible, grammatically perfect sentence. The very quality that makes these systems feel magical — that they always have an answer — is the quality that makes them hazardous when the answer has to be true.
Improvisation is a decision to leave things to chance
“Let the model decide” sounds like sophistication. In practice it means handing a probability engine the decisions a resident actually feels: what gets promised, what gets dispatched, what gets escalated, what a fee is, whether an emergency reaches the on-call officer. Each of those is a load-bearing outcome, and a generative model produces a slightly different answer every time you ask. You would never let a new staff member freelance those decisions on day one with no rules and no supervision. Ad-lib AI is exactly that, at scale, in every conversation at once.
Why government has no tolerance for it
In a consumer app a hallucination is an annoyance you laugh about. In local government it is a vulnerable person given the wrong information at 2am, a false promise that erodes trust in the institution, or a genuine emergency that never reaches someone who could act. The public does not get to opt out of its council. That lack of an exit is exactly why the standard has to be higher than “usually right.” Usually right, delivered with total confidence, is the fastest way to burn public trust — because people believe it.
And you cannot patch your way out of it with a better prompt. Telling the model “do not make promises you cannot keep” is still leaving the promise to the model. The instruction is itself a suggestion it may ignore under the right phrasing. If the outcome matters, an instruction is not a control.
The line that keeps you safe
The discipline is simple to state and hard to hold: the language model may own how something is said — the tone, the warmth, the acknowledgement — but it must never own what is done. What gets promised, dispatched, escalated or quoted belongs to deterministic rules the organisation can inspect and stand behind. Give the AI freedom where the stakes are lowest and bind it tightly where they are highest. Its output should be a suggestion the system is free to reject, never a contract the resident is left holding.
Used that way, a language model is a genuine gift to public service. Used as an oracle you trust to improvise the important parts, it is a liability wearing a friendly voice. The difference is not the model. It is whether you were willing to leave the things that matter to chance.