6 posts tagged with "versioning"

When you deprecate an API for human developers, you have a whole ceremony for it. You bump the version, add deprecated: true to the OpenAPI spec, ship a Sunset HTTP header, send an email to the developer mailing list, post to the changelog, and give people six months to migrate. The signal reaches a human who reads it, files a ticket, and updates their client before the old path disappears.

Now point that same deprecation at an agent. The model calling your tool does not read your changelog. It does not subscribe to your mailing list. It never sees the Sunset header unless you deliberately put it somewhere the model looks, and even then it has no reliable habit of acting on it. The deprecation notice you so carefully authored lands in a mailbox with no reader. The agent keeps calling the old shape of the tool until the shape is gone, and then it fails — often silently, often in production, often at 2 a.m.

This is the quiet asymmetry of building tools for agents instead of people. Every discipline we built over two decades of API evolution assumes a human sits between the deprecation and the migration. Take the human out, and the entire mechanism goes dark.

You ran the deprecation playbook flawlessly. Announcement email six months out. A migration guide with code samples in four languages. Sunset headers on every v1 response. A banner in the developer dashboard. Two reminder emails. Then you turned off v1 — and instead of the quiet cutover the playbook promised, your error rate went vertical and stayed there. Traffic didn't migrate. It just kept arriving, failing, and retrying, at higher volume than before, because every failure triggered another attempt.

The playbook didn't fail because you executed it badly. It failed because it assumes a human is on the other end — someone who reads email, skims changelogs, and files a Jira ticket to migrate before the deadline. A growing share of your traffic has no such person behind it. It comes from agents whose knowledge of your API is frozen in system prompts, tool schemas, and model training data scattered across thousands of repositories you can't see and will never reach. Your sunset notice was written for a reader. Your biggest client is a prompt.

A platform team ships a one-line tweak to the system prompt of their summarization service. No code review, no migration guide, no version bump — it's "just a prompt." Two weeks later, the legal product team finds out their compliance auto-redaction has been silently letting names through. The investigation eats a sprint. The fix is trivial. The damage is the trust.

This is the AI changelog problem in miniature. Behavior is now a first-class output of your system, and behavior changes when prompts, models, retrievers, or tool schemas change — none of which show up in git diff of the consuming application. Teams that treat AI updates like backend deploys, where a Slack message in #releases is enough, end up reinventing the worst parts of the early-2010s "we'll just push and tell QA later" workflow.

Your /v1/summarize endpoint worked perfectly for eighteen months. Then you upgraded the underlying model. The output format didn't change. The JSON schema was identical. But your downstream consumers started filing bugs: the summaries were "too casual," the bullet points were "weirdly specific," the refusals on edge cases were "different." Nothing broke in the traditional sense. Everything broke in the AI sense.

This is the versioning problem that REST and GraphQL were never designed to solve. Traditional API contracts assume determinism: the same input always produces the same output. An AI endpoint's contract is probabilistic — it includes tone, reasoning style, output length distribution, and refusal thresholds, all of which can drift when you swap or update the underlying model. The techniques that work for database-backed APIs are necessary but not sufficient for AI-backed ones.

You shipped an agent last Tuesday. Nothing in your codebase changed. On Thursday, it started refusing tool calls it had handled reliably for weeks. Your git log is clean, your tests pass, and your CI pipeline is green. But the agent is broken — and you have no version to roll back to, because the thing that changed wasn't in your repository.

This is the central paradox of agent versioning: the artifacts you track (code, configs, prompts) are necessary but insufficient to define what your agent actually does. The behavior emerges from the intersection of code, model weights, tool APIs, and runtime context — and any one of those can shift without leaving a trace in your version control system.

In 2023, a team at Stanford and UC Berkeley ran a controlled experiment: they submitted the same prompt to GPT-4 in March and again in June. The task was elementary — identify whether a number is prime. In March, GPT-4 was right 84% of the time. By June, using the exact same API endpoint and the exact same model alias, accuracy had fallen to 51%. No changelog. No notice. No breaking change in the traditional sense.

That experiment crystallized a problem every team deploying LLMs in multi-service architectures eventually hits: model aliases are not stable contracts. When your downstream payment processor, recommendation engine, or compliance system depends on structured JSON from an LLM, you've created an implicit API contract — and implicit contracts break silently.