Your Agent Has Two Release Pipelines, Not One

A team I worked with shipped a "small prompt tweak" on a Wednesday afternoon. The same PR also added one new tool to the agent's registry — a convenience wrapper around an internal admin API that the prompt would now occasionally invoke. The eval suite passed. The canary looked clean. By Thursday morning a customer's billing record had been mutated by an agent acting on a prompt-injected support ticket, the audit trail showed the admin tool firing exactly as designed, and the on-call engineer's first instinct — roll back the prompt — did nothing useful, because the credential had already been used and the row had already been written.

The post-mortem framed it as a security review failure. It wasn't. It was a release-pipeline failure. The team had shipped two completely different asset classes — a behavioral nudge to the model and a new authority granted to the agent — through the same review, the same gate, and the same rollback story, as if they were the same kind of change. They aren't. And once you see them as two pipelines, most "agent governance" debates become much less mysterious.

Two asset classes that look identical on disk

Open the repo of any non-trivial agent and you'll see two folders that, structurally, look indistinguishable. Both are versioned text. Both are reviewed in pull requests. Both deploy with the same CI job. But they sit on opposite sides of a fault line that networking architects identified forty years ago and that we keep re-discovering badly.

The data plane of an agent is the model's behavioral shape: system prompts, instructions, few-shot examples, output schemas, persona descriptions, refusal guidelines, the whole literary corpus that nudges the model toward one distribution of outputs over another. Edits here change what the model says.

The control plane is the agent's reachable action space: the tool registry, tool schemas, capability scopes, auth boundaries, rate-limit budgets, the list of MCP servers the agent can talk to, the policies that gate which tools fire under which conditions. Edits here change what the agent can do.

These are not two flavors of the same thing. They have different blast radii, different rollback semantics, different reviewers, different audit obligations, and different failure modes — and treating them as one pipeline is the architectural mistake at the root of most "the agent did something wild in production" incidents.

The asymmetry of failure

A bad data plane change produces a quality regression. The agent gets too verbose, refuses requests it shouldn't, hallucinates a field name, repeats itself, switches tone mid-conversation. You see it in eval deltas, user complaints, satisfaction scores. You roll the prompt back to the previous version and the regression goes with it. The blast radius is bounded by your monitoring window, and the recovery is mechanical: every conversation after the rollback uses the new behavior, and the only residue is a few hours of mediocre transcripts.

A bad control plane change produces an authority regression. The agent didn't say something you wished it hadn't — it did something you wished it hadn't. Sent the email. Cancelled the subscription. Wrote the row. Granted the access token. Posted to the channel. The model can roll back; the side-effects can't. Your "rollback" is now an incident response that involves credential rotation, communications to affected users, possibly a regulator phone call, and a lawyer asking pointed questions about what your audit log actually captured. You don't recover from a bad tool registration the way you recover from a bad prompt. You contain it, and you live with the residue.

OWASP's LLM06 guidance frames this as "excessive agency" and explicitly calls it a fundamental design tension rather than a configurable knob. That language matters. It says the right control isn't a setting on a dashboard — it's an architectural choice about which side of the data/control divide a change lives on, and what review it gets accordingly.

Why one pipeline is the default

Almost every team starts with one pipeline because, on day one, the asset classes haven't differentiated yet. The "agent" is a single Python file with the system prompt at the top, a TOOLS list near the bottom, and a main() function that wires them together. One repo, one CI, one reviewer. Treating it as two release surfaces would be ceremony for ceremony's sake.

The trouble is that the two asset classes start to grow at very different rates and into very different shapes — and the pipeline that was correctly fused on day one stays fused long past the point where the fusion does any work. The system prompt grows from 30 lines to 300, with regression tests, eval suites, behavioral diffs, A/B harnesses, and a versioning story. The tool registry grows from three functions to thirty, with auth scopes, rate-limit budgets, blast-radius classifications, and the long quiet drift of "we added one more capability for that one Q3 use case and never re-audited the whole list."

A year in, you have one pipeline doing two jobs poorly. The prompt edits go through eval-delta review (good — that's what they need) and the tool additions go through eval-delta review (bad — that's not what they need; what they need is capability-scope review, taint analysis, blast-radius classification, and a security signature). And because the same PR template, the same reviewers, and the same merge button cover both, the engineer adding a tool gets the prompt-engineer's gate by accident. The review never asks the right questions. The eval suite measures behavior, not authority — and the new tool, by definition, expands authority.

Loading…