45 posts tagged with "governance"

Your agent works. It resolves the ticket, issues the refund, updates the record, closes the loop — unattended, at a quality your metrics say beats the median human doing the same job. You are ready to take the human out of the loop. And then the deployment stalls, not in engineering, but in a meeting where someone from legal or finance asks a single question you cannot answer: when it's wrong, who eats the loss?

This is the part of agent autonomy that the capability curve doesn't touch. You can push accuracy from 95% to 99% and the question doesn't move. Because the blocker was never "is the model good enough." The blocker is that a probabilistic system taking irreversible actions is a risk somebody has to hold, and right now nobody wants to hold it. Your errors-and-omissions policy was written for human mistakes and increasingly carves out automated ones. Your model vendor's contract disclaims the whole thing. And no carrier has an actuarial table for a system whose failure rate quietly drifts every time you edit a prompt.

Last month I needed to reconcile two CSV exports with slightly different column conventions — a task I'd normally solve by finding a diff tool, reading its docs, and fighting its assumptions for twenty minutes. Instead I asked an agent to write me a fifty-line script. It ran once, produced the answer, and I deleted it. Total elapsed time: ninety seconds. The script never entered version control, never got a name, and never will be seen again.

That transaction — write it, run it, delete it — is quietly becoming the default mode for a whole class of work. When code generation costs approach zero, the cheapest correct move is often a bespoke one-off, not a general-purpose tool. Finding an existing utility means searching, evaluating, installing, configuring, and trusting it. Generating a disposable one means describing what you want. For narrow, well-specified tasks, the second path now wins on every axis except one: nobody is watching what gets created.

Two years ago, standing up an AI center of excellence was the responsible move. Nobody knew how to evaluate a model, procurement had no idea what an inference contract should look like, and legal wanted one throat to choke. Concentrating the ten people who understood any of it into a central team was obviously correct.

Today that same team is the reason your product engineers wait six weeks to change a prompt. The CoE reviews every system-message edit, owns the only eval harness in the company, and gates model upgrades behind a committee that meets biweekly. Teams have noticed. They ship on personal API keys, run evals in notebooks the CoE never sees, and paste customer data into whatever tool answers fastest. You did not prevent shadow AI — you created it, and gave it a governance body to hide from.

A platform team I know spent two quarters building a model registry. It had everything the org chart asked for: a promotion workflow from dev to staging to prod, a CODEOWNERS-style approval matrix, lineage tracking, eval-score gates, a deprecation policy with a 30-day window, and a Backstage tile that showed which version of every model was live in which service. They cut a launch announcement, ran a brown bag, and added a row to the compliance binder.

Six months later, the highest-traffic agent in the company was running on a model card whose "owner" field still pointed at someone who had left, whose eval score was from a benchmark the team had since deprecated, and whose "approved by" name was the platform tech lead — who had never used that agent, never read its eval set, and had pressed approve at 11:43pm on a Thursday because the producer had pinged him in DMs saying the launch was tomorrow.

The registry was not broken. The promotion gates fired. The audit log was intact. Everything the launch announcement had promised was true. And the org had less real oversight of its production models than it had had eighteen months earlier, when the same decisions were made by an ML engineer reading the eval output by hand before pasting the model URI into a config file.

Finance sent a memo on a Monday. By Friday, every product team had shipped a prompt change, and on the following Tuesday the support queue grew by a third. Nobody had touched the model. Nobody had touched the product. The only thing that had changed was that the LLM bill was now flowing back to the teams that issued the calls — and the teams had responded the way any rational cost center responds to a new line item on its P&L. They cut it.

The story that gets told inside the company afterwards is a story about prompt engineering, or about the model regressing, or about a noisy week of user traffic. The truer story is that finance, through a chargeback policy, had quietly become a product manager. The cost-attribution dashboard was a product-quality lever that nobody had reviewed, nobody had instrumented for, and nobody owned. When it moved, every prompt in the company moved with it, and the trade-offs that produced the quality regression were never seen by the people whose job it was to see them.

The auditor's question sounds simple. She has your appeals log open, points at a row from eight months ago, and asks which model decided that case. Your engineer pulls up the schema: there is a model column, and every decision in the audit window says v1. Then someone from the platform team mentions, almost in passing, that the alias behind v1 rotated four times during the audit period — a base model upgrade, a fine-tune refresh, a vendor-side capacity move, and one rollback that lasted six hours during an incident. The honest answer is that you cannot say which checkpoint produced that decision. The auditor writes something down. That phrase is not a regulator-acceptable answer, and you have just learned that the system you shipped has been failing an audit requirement it was never designed to meet.

The gap here is not a missing log line. The gap is between two different ideas of what "model" means. To the engineers shipping the system, v1 is an endpoint — a stable contract callers can point at while the thing behind it gets upgraded for free. To the auditor, "the model that produced this decision" is a specific artifact: a weight checkpoint, a hash, a thing you could in principle re-run on the same input and get a defensibly similar output. Endpoint aliases were invented to hide checkpoint rotation from callers. Audit-grade provenance demands the opposite — that every decision be attributable to exactly the checkpoint that produced it. The two ideas were on a collision course from the start; the audit just happened to be where they met.

Six months ago, your team shipped a support agent that handles refunds. There was a one-page Notion doc describing what it should do. Today, the doc still says what it said, but the agent does not. The prompt has 47 edits in its history. Three tools were added — one of them quietly bypasses a finance check that the doc still asserts exists. The model was swapped twice. A retry policy was hardened after an incident nobody wrote up. And when somebody on the data team asks "what are the actual rules for issuing a refund here," the honest answer is: read the system prompt and the tool registry, because that is the spec now.

This is the quiet failure mode of agentic systems in production: the agent's behavior IS the runbook nobody wrote. The prompt got treated as a configuration value — a string in a YAML file, edited by whoever owned the feature, reviewed like a copy change — when it was actually the most authoritative description of a multi-step business process in the company. The org accumulated process logic the way legacy codebases accumulate behavior: through edits, not design. And the people who would historically own that process — a product manager, a compliance lead, an ops director — never realized they had lost the artifact, because there was never a document to lose.

A user types "remind me every Tuesday to check that integration." The agent creates a cron entry, returns a polite confirmation, and the session closes. Six months later the user has changed teams. The integration was deprecated last quarter. The cron is still firing, hitting an API key that was rotated in April, into a Slack channel that was archived in May, charged to a project budget that nobody reviews. The agent did exactly what was asked. The asking is what aged badly.

This is not a bug in any particular agent. It is the shape of a category. The moment we gave agents the ability to schedule durable side effects — cron jobs, webhooks, polling loops, workflow triggers, calendar invites, recurring queries — we created a class of infrastructure that is born without a lifecycle. The create primitive is loud and easy. The delete primitive, the audit primitive, the inheritance primitive — they don't exist on equal footing, so they don't get used.

The cost is invisible until you go looking, which is exactly when nobody is looking.

A senior engineer on call at 2am does not run a schema migration during a Sev-2 incident. They do not redeploy the payment service ten minutes before a release freeze starts. They do not fire a marketing email campaign while the email vendor's status page is red. None of this is in their job description. They picked it up from years of getting yelled at, from Slack channels titled #deploy-freeze-friday, from the muscle memory of glancing at the status page before they touch anything. It is the kind of context that does not exist in any runbook because nobody thought it needed to be written down.

Now hand the same job to an agent. The agent has tools. The agent has a multi-step plan. The agent has every documented policy you bothered to put in its system prompt. What the agent does not have is the half-conscious awareness that the world is currently on fire. So it executes the plan. Cleanly. Confidently. Into the maintenance window. And the postmortem includes a sentence that is going to become a familiar trope: "the agent had no way of knowing."

The eval set is the most-read artifact your AI team produces, and you almost certainly don't know who's reading it. The repo is private, the CI job is internal, the file is one directory above prompts/ — and yet a sales engineer scraped six cases for a demo last quarter, a marketing analyst pulled three failure cases into a "look how robust our system is" deck, customer success cited eval pass-rates verbatim in a renewal call, and product treats the file as the hidden spec the AI team won't share. The case files are read by more people than the code that generated them, and nobody on the AI team has noticed.

This isn't a permissions failure. The eval set is on the same Git server as the rest of the codebase, with the same access controls as every other engineering artifact. The problem is that the AI team is the only group that treats the eval set as code. Everyone else treats it as documentation, as marketing material, as a product spec, or as a customer complaint log — and each of those readings extracts a different slice of the same file, packages it for a different audience, and ships it somewhere the AI team isn't watching.

The dataset your AI team calls "the eval set" is, in most companies shipping LLM features, a collection of real customer conversations pulled from production logs. Nobody on the team thinks of it as a privacy event. The data never left the cluster. No new system was provisioned. No vendor was added. An engineer wrote a query, exported a few thousand traces into a labeling tool, and the team started grading model outputs against them. The legal team never heard about it because, from the inside, nothing changed — the same conversations that already lived in the same database were now also being read by a few engineers and a judge model.

That is the privacy boundary nobody reviewed. Customers gave you their messages so you could answer them. They did not give you their messages so you could measure your model against them. The two uses look identical at the storage layer and feel identical at the inference layer, but they are different processing purposes under every modern privacy regime — and the gap between the two is where the next round of compliance pain is going to land.

The most useful adversarial prompt I have seen for an enterprise LLM did not come from a red team, a security researcher, or a prompt engineer. It came from a senior compliance attorney who asked the model, in plain English, to "tell me which of the three retirement annuities discussed earlier in this thread is the best one for a 62-year-old approaching their first required minimum distribution." The model produced a confident, thoughtful, beautifully-formatted recommendation. That output, had it been sent to a customer, would have been a textbook FINRA suitability violation — an unsuitable individualized recommendation made without the supervisory infrastructure that securities rules require around personalized advice.

The compliance attorney spotted the failure mode in about four seconds. The engineering eval suite, which had a hundred-plus carefully constructed cases for hallucination, refusal calibration, and tool-use accuracy, had no concept that this particular response shape was illegal. Not low quality. Not a hallucination. Illegal. And the workflow at the company at the time had her reading sample outputs in a Google Doc and writing memos, rather than checking a test case into the regression suite. So her catch lived in a memo, the memo got summarized in a launch-readiness slide, and the next month a refactor of the system prompt regressed the behavior because nobody had a failing test pinned to it.

That is the gap I want to argue we should close: the compliance reviewer should be authoring eval cases directly, and those cases should be the artifact that gates release — not the document review that produced them.