Tool Behavior Drift: The Schema Held, the Semantics Didn't

Your contract tests are green. The schema validator is happy. The tool returns the same shape it did last quarter. And the user-facing answer has been quietly wrong for six weeks.

This is the failure mode that contract testing was never designed to catch. Contract tests verify that the wire format hasn't changed — that search() still returns { results: [{ id, title, score }] }, that create_event still accepts an ISO 8601 string, that the geocoder still emits { lat, lng }. What they don't catch is the moment the search endpoint starts ranking by recency instead of relevance, the calendar API silently snaps your 14:07 start time to 14:00 in the EU region, the geocoder picks a different point inside the same ambiguous polygon, or the LLM-classifier-as-a-tool is upgraded to a new model behind a stable endpoint and the false-positive rate moves four points in a category your eval set never sampled. The schema held. The behavior didn't. Your agent kept reading green checkmarks and produced regressed answers no error log captured.

The pattern is so consistent across vendor relationships that it deserves its own name: tool behavior drift. It is to AI agent systems what API behavior changes have always been to client-server systems — except the agent has fewer ways to notice and fewer ways to recover. A 2026 study on silent tool failures found that smaller models are particularly poor at detecting that a tool returned the right shape with the wrong content; they trust the structure and reason on top of it. The harness has no concept of a post-condition that compares "what the tool said today" against "what the tool said for the same input last week." So drift accumulates until a human notices a regression and walks the trace back through every layer that didn't fire an alert.

The Contract You Wrote and the Contract You Depend On

The first move toward catching behavior drift is admitting that the schema is not the contract. The schema is the envelope of the contract. The contract is everything you depend on the tool doing — including the things you never wrote down because they seemed too obvious to specify.

A search endpoint's contract isn't returns: list of results. It's: results are ranked by relevance to the query; the rank-1 result for query Q has been R for six months; results for queries with no plausible match return an empty list rather than the closest-by-string-distance match; pagination tokens are stable for at least 60 seconds. Your prompt was tuned against those properties. Your eval set was scored against those properties. Your downstream tool selection — "if the rank-1 score is below 0.4, fall back to a different retrieval path" — assumed those properties.

The vendor's release notes will say the new version "improves freshness signals." Your schema validator will say nothing changed. Your agent will start surfacing newer-but-less-relevant results, your prompt's calibration on score thresholds will silently be wrong, your downstream fallback will trigger less often, and the answer quality will drift in a direction that doesn't show up in any metric you currently log.

This is the gap practitioners have been talking about for years in non-AI systems. As one PactFlow engineer puts it: "Behavioral changes that preserve the schema can still break clients — for instance, if a list endpoint silently switches to eventual consistency and sometimes misses the newest records, many clients will break in production workflows even though the JSON looks fine." Agents are the most behavior-sensitive clients we've ever shipped. They reason on the content. The structure is the least of it.

Five Drift Modes That Don't Trip Schema Tests

Practitioners building agentic systems in production are repeatedly bitten by the same handful of patterns. Naming them helps because each has a different detection strategy.

• Ranking-policy drift. A retrieval tool changes its ranking function — pure relevance becomes blended-with-recency, BM25 becomes hybrid with a vector reranker, the tie-break shifts from alphabetical to popularity. The list shape is identical. The list content shifts. Anything downstream that depends on rank-1 stability degrades.
• Quantization and rounding drift. A calendar API begins snapping times to 15-minute boundaries in one region. A pricing API switches from float-with-eight-decimals to integer-with-cents. A geocoder rounds coordinates to four decimals instead of six. The agent's tool call produces values that look right and are subtly wrong — until they hit a system that cares about the precision.
• LLM-as-tool model drift. This is the most insidious. A vendor swaps the model behind a stable endpoint — same name, same schema, new weights. False-positive rate moves a few points. Verbosity changes. Refusal patterns shift. A category your eval suite never sampled regresses materially. Independent research has documented that 91% of production LLMs experience silent behavioral drift within 90 days of deployment, and detection lags average 14–18 days between onset and the first user complaint.
• Side-effect drift. A "create" call now also sends a notification email by default. A "delete" call now soft-deletes with a 30-day window instead of hard-deleting. A "transcribe" call now also generates a summary that the agent didn't ask for. The response shape is unchanged. The world the response describes has more or fewer effects in it.
• Disambiguation drift. A tool that takes an ambiguous input — a partial name, a polygon with multiple matching addresses, a query with multiple plausible interpretations — changes how it picks the winner. The geocoder used to return the centroid of the largest matching polygon; now it returns the most-recently-edited one. The user query "Springfield" used to default to the most-populous city; now it defaults to the user's nearest. Your agent's downstream reasoning was tuned against the old default.

None of these are schema changes. All of them break agents.

Semantic Golden Traces, Behavior Changelogs, and a Canary Suite

The fix is not to stop using third-party tools. It is to build a second layer of monitoring that watches behavior the way contract tests watch shape. The vocabulary already exists in the canary-analysis literature; it just needs to be applied to the tool surface rather than to your own deploys.

Semantic golden traces are fixed query/response pairs that exercise behavior, not shape. For each tool, pick 10–50 inputs whose correct outputs are stable and known. "For this fixed query, the rank-1 result has been document D for six months." "For this fixed address, the geocoder returns (lat, lng) to four decimal places." "For this fixed prompt, the classifier returns category: A with confidence above 0.7." Run these on a schedule. Diff against the baseline. Alert when the diff exceeds a band you've calibrated against natural noise. This is the behavioral analogue of a contract test, and it catches every drift mode in the previous section that doesn't change the wire format.

Behavior changelogs are a per-tool feed your team subscribes to with the same care as a security advisory. The vendor's release notes, the API status page, the model-card revision history, the model name and quantization fingerprint exposed in response headers if the vendor provides one. When the vendor doesn't publish a changelog (and many don't, especially for LLM-backed endpoints), your golden traces become the changelog — a regression on the canary suite is the first signal a behavior change happened, and your incident response is to backfill the "what changed" question against the vendor's communication channels.

Per-tool canary suites run continuously. The pattern from production canary analysis applies directly: a small number of strongly-correlated metrics is more useful than a large number of weakly-correlated ones. For each tool, pick the two or three properties your agent's downstream reasoning actually depends on, write probes that test exactly those, and run them every hour. Track the metric. When it crosses a band, page someone. The Google SRE canary literature is explicit: too-strict thresholds give false positives until the team turns the alert off; too-loose thresholds let bad rollouts through. The tuning is real work and is itself part of the system.

Tool-level circuit breakers are the action layer. A behavioral metric crossing a threshold is no different from an error rate crossing a threshold — it should flip the agent to a fallback path, just as a 5xx rate flip would. The mechanics are the same as any circuit breaker: closed → half-open → open, with the metric that drives the transition being a behavioral one (false-positive rate, rank-1 stability, output length distribution) rather than a transport one (HTTP status, timeout). Production agent harnesses are starting to ship this primitive — every agent in a mesh gets a configured fallback sequence, and the router demotes the primary when a behavior-quality metric crosses its band.

The Trace That Tells You the Tool Drifted, Not the Model

When an agent's quality regresses, the team's instinct is to blame the model. The model is the most visible component, the one that gets the most attention, and the one whose providers ship visible upgrades. So the team rolls back the prompt, A/B-tests a different model version, runs the eval suite — and finds nothing. The model didn't move. The tool did.

This is the failure mode that costs the most cycles, because the team is debugging the wrong layer. The architectural fix is to instrument the tool surface independently of the model surface. Every tool call gets a trace span with the full input, the full output, the tool's reported version (if any), and a hash of the response that lets you compare against the golden trace baseline. When a regression lands, the first dashboard you open isn't "model behavior over time" — it's "tool behavior over time, by tool, by input class." The drifted tool will show up as a step function in its golden-trace diff rate; the model, if it didn't change, will show flat.

The deeper realization is that your prompt is co-tuned with the tool's behavior, not just with the model's behavior. A prompt that says "summarize the rank-1 result" is implicitly assuming the rank-1 result is what you want to summarize. When the ranking changes, the prompt is suddenly wrong, even though the prompt's text is unchanged. Practitioners increasingly think of the "system prompt" as everything in the agent's context that shapes its output — and the tool's behavior, baked into the responses it returns, is part of that context. Monitor it accordingly.

What the Discipline Looks Like When It's Working

A team that's gotten this right has a small number of recognizable habits. Every external tool has an owner — a single engineer or sub-team responsible for its behavioral health, the same way every database table has an owner. The owner maintains the golden trace set, watches the canary, and gets paged when the behavior metric crosses a band. The team's runbook for "the agent regressed in production" starts with "check the tool dashboard before you touch the model." The vendor relationship includes a behavior-change channel — a Slack connect, a mailing list, a dedicated contact — and not just a status page that fires on outages.

When the team adds a new tool, the integration checklist includes "what's the behavioral contract here, beyond the schema?" — and the answer is captured as a set of probes before the tool ships behind the agent. The probes are versioned alongside the prompt and the eval set, on the same release cadence. Behavior drift is a planned-for category of incident, not a surprise that gets explained in a postmortem.

The team that monitors only the schema is monitoring half the contract. The half they're not watching is exactly the half a 2026-era tool surface keeps changing — because the vendors shipping these tools are themselves shipping ML systems, and the only thing they can promise to keep stable is the wire format. The semantics are the product, and the product is iterating. Your agent's reliability depends on whether your monitoring can see that iteration before your users do.