The Rollback That Couldn't: Prompts, Tools, and Memory Version Together or Not at All

The incident channel says the new prompt is hallucinating refund amounts, so you do the obvious thing: repoint the production tag to last week's prompt version. Thirty seconds, no redeploy, textbook rollback. Except the agent gets worse. Last week's prompt references a lookup_order tool that the platform team renamed to orders.search on Tuesday. The memory store is full of preference summaries written by the new prompt's format, which the old prompt reads as user instructions. You didn't roll back the agent. You built a chimera — one-third last week, two-thirds today — and shipped it to production without ever testing that combination.

This is the failure mode nobody's runbook covers: an agent deployment is not an artifact, it's a triple — prompt version, tool contract, accumulated memory state. Rolling back one leg while the other two advance doesn't restore a previous state. It creates a new state that has never existed before, never passed an eval, and belongs to no team's on-call rotation.

The prompt registry vendors have taught everyone that rollback should be "a configuration change, not a redeploy" — repoint the tag, done in seconds. That advice is correct and dangerously incomplete. It's correct for the one leg of the system that is genuinely stateless. It's incomplete because it implies the other two legs will follow along. They won't.

An Agent Deployment Is a Triple, Not an Artifact

Break down what actually determines an agent's behavior in production:

• The prompt — system prompt, few-shot examples, output format instructions. Usually owned by the AI or product team, versioned in a prompt registry, deployed by tag.
• The tool contract — every tool name, parameter schema, and description string the agent can see. Usually owned by a platform or backend team, deployed with the services that back them. In an MCP world, some of these are owned by teams outside your company entirely.
• The memory state — retrieved summaries, learned preferences, vector store contents, scratch records from previous runs. Owned by everyone and no one, and unlike the other two legs, it accumulates rather than deploys.

These three legs ship on three different cadences, through three different pipelines, with three different owners. The prompt changes daily. The tool schemas change weekly with service deploys. The memory changes continuously, on every conversation.

The behavior your evals blessed before launch was a property of one specific combination: prompt v41, tool snapshot from Tuesday, memory as it existed that afternoon. Model version is arguably a fourth leg — pinning claude-sonnet-4-6-20250514 rather than an alias matters for the same reasons — but the triple is the part your own organization mutates, so it's where the coordination failures live.

When the pager goes off and someone rolls back "the agent," they roll back the leg their team owns. That's the trap.

How One-Legged Rollbacks Fail

The insidious part is that each mismatched pairing fails differently, and none of them fail loudly.

Old prompt, new tools. The rolled-back prompt references tools by the names and shapes it knew. If a tool was renamed, the agent either fails to find it or — worse — picks the closest-sounding alternative and calls it with confidence.

And renames are the easy case, because tool contracts break in ways schema diffing never catches. The description string of a tool is part of the contract: change the wording of search's description and you've changed the probability the model selects it, even though no code changed. There's also logic drift — the schema is byte-identical, but the tool that did fuzzy matching last week does exact matching this week, and the rolled-back prompt's retry strategy was tuned for the old behavior. A REST API breaks loudly with a 400. A tool contract breaks silently, as a slightly wrong answer.

New prompt, old tools. The mirror image, common when a platform team reverts a flaky service while the prompt stays current. The prompt's few-shot examples demonstrate calls against parameters that no longer exist. The agent imitates the examples, gets errors, and burns its loop budget retrying — or silently degrades to answering from its own weights instead of calling the tool at all.

Either prompt, wrong-era memory. The subtlest one. Memory written by prompt v42 — its summary format, its abbreviations, its implicit assumptions about what fields mean — gets retrieved into prompt v41's context. The old prompt never established the conventions the new memory relies on, so retrieved content lands as noise at best and as instruction injection at worst. Nothing errors. The agent just behaves in ways that neither version's evals ever saw.

Now consider who debugs this. The prompt team reproduces with the current prompt against current tools: works fine. The platform team replays the tool calls: all valid. The memory infra team checks the store: no corruption. Every team's leg is healthy in isolation. The failure exists only in the pairing, which means it exists in nobody's test environment and reproduces on nobody's laptop. Three green dashboards, one broken agent.

Memory Is the Leg That Can't Roll Back

Prompts and tool schemas are at least reversible — they're stateless artifacts, and pointing at an old version genuinely restores it. Memory is different in kind, and the database people figured out why decades ago: state only moves forward. A database "rollback" is really a roll forward to a state that happens to resemble the past. Migration tools that advertise rollback support handle schema reversal; the data itself — a lossy type change, two columns merged into one — cannot be un-migrated without a backup, only compensated for.

Agent memory is a database with worse hygiene. During the window when the bad version ran, it wasn't just misbehaving in responses — it was writing. Wrong preference summaries, polluted vector entries, scratch records encoding conclusions the bad prompt reached.

Loading…