Prompt-Version Skew Across Regions: The Unintended A/B Test Your CDN Ran for Six Hours

You shipped a system-prompt change at 09:14. The rollout dashboard turned green at 09:31. By 11:00 your eval tracker still looked clean, the cost dashboard was unremarkable, and a customer-success engineer pinged the team: structured-output errors on the parser side were up about three percent in Asia-Pacific only. Nothing in North America. Nothing in Europe.

The rollout had paused itself at 67% region coverage because a non-load-bearing health check on one POP flapped during the cutover, and nobody had noticed. For six hours, us-east and eu-west were running prompt v47 while ap-south and ap-northeast were still on v46. You were running a live A/B test split by geography — except you didn't design the test, you couldn't see the test, and the eval suite that was supposed to catch quality regressions was hitting the new version in one region and shrugging.

This failure mode is not a bug in any single tool. It is the predictable consequence of pushing prompts through deployment systems built for a different kind of artifact.

Prompts Are Not Static Assets, but Your Rollout System Thinks They Are

The default machinery for distributing config across regions — CDN edge configs, key-value config services, sidecar pollers reading from object storage — was designed for assets where eventual consistency is fine. A 200ms window where one POP serves a stale logo is invisible. A 200ms window where one POP serves a stale CSS file is a cache miss. A six-hour window where one POP serves a stale system prompt is a behavioral fork in your product.

Static assets carry no semantic coupling to the model that consumes them. Prompts do. A new prompt version often coordinates with a new output schema the parser expects, a new tool definition the agent calls, a new safety rail the downstream filter relies on. When v47 lands in one region but v46 stays in another, the parser sees one schema in us-east and a different schema in ap-south. The cost dashboard sees one token distribution in one region and a different one in another. The eval suite sees whichever region its harness happens to point at.

Three reinforcing properties make this worse than the equivalent CDN-asset divergence:

• Behavioral divergence is silent. The model still returns 200s. The latency is still in band. Nothing in the wire-level signal says "this region is on a different prompt." The only evidence is downstream — parsing failures, eval drift, weird user complaints clustered by geography.
• Quality regressions can take hours to surface. Token-level A/B differences in a chat product accumulate over conversations, not over single requests. A prompt that subtly degrades multi-turn coherence shows up as a Day-2 retention dip, not a Day-1 latency alert.
• Eval suites assume one config. Most eval harnesses talk to a single endpoint with a single prompt. They cannot detect "the prompt my users saw in Seoul is not the prompt my evals tested in Virginia."

The Latency-Routing Trap That Breaks Per-User Assignment

Here is where it gets adversarial. Modern LLM-serving stacks pair regional rollout with latency-aware routing — Anycast DNS, GSLB, or an ingress that picks the nearest healthy POP for each new TCP connection. The routing layer optimizes for round-trip time per request, which is the right thing to do for latency.

It is the wrong thing to do for prompt-version stability.

A single user, in a single session, can land on us-east for the first message (closest healthy POP at the moment of TLS handshake) and us-central for a follow-up message twelve seconds later because a transient capacity event in us-east made us-central look closer to the GSLB. If those two regions are at different stages of a prompt rollout, that user just experienced two prompt versions inside one conversation. Whatever conversation history was bound to v47 is now being interpreted by v46.

The eval pipeline that slices regressions by user_id sees this as user behavior shift. The user "switched topics," the user "started speaking less formally," the user "dropped follow-up depth." The team chases user-segment hypotheses for two days before someone notices that the affected users all bounced between POPs mid-session.

The fix is not to abandon latency routing. The fix is to recognize that prompt version, not just network proximity, is part of the routing decision. A POP serving v46 cannot be considered "the same backend" as a POP serving v47 for purposes of session affinity, even if the URL is identical.

Treat Prompts Like Database Migrations, Not Like Static Configs

The discipline that holds up under this load is the same one that the database community converged on a decade ago for schema changes — and for the same reason. Schemas, like prompts, carry semantic coupling to code. You don't run a rolling deploy of a schema migration where some app servers see the new column and some don't. You sequence it: schema is live globally, then code rolls forward.

For prompts, the analogous discipline has three rules.

One version live globally before the next ships. This is the analog of "expand-then-contract" for schema migrations. The new prompt has to reach 100% in every region before the controller will queue the next change. Mid-rollout pauses are normal; the pathological state is concurrent mid-rollouts where two prompt revisions are partially deployed across the same fleet.

Atomic per-region cutover, not rolling-within-region. Inside a region, every replica flips at the same time. This is what teams sometimes call a "drain-and-cutover" gate: traffic in flight finishes against the old version, then the entire region's pool moves to the new version in a single instant. The alternative — gradually replacing pods, each pod independently picking up the new prompt — produces the same skew problem at a smaller scale (intra-region instead of inter-region) and is harder to detect because most observability dashboards aggregate per region.

Region-by-region cutover, with a kill switch on regional drift. The controller advances regions sequentially: a region cuts over, soak time elapses, eval delta in that region is checked against the production baseline, then the next region is queued. If the soak window detects a quality regression, the rollout halts and existing already-cut-over regions roll back together. The unit of rollout is "one region cleanly," not "1% of fleet across regions."

This is slower than your current rollout. That is the point. The reason database migrations are slower than code deploys is not engineering laziness — it is the recognition that the cost of a half-applied schema is much higher than the cost of waiting an extra hour.

Make Prompt Version a First-Class Observability Dimension

The reason the six-hour skew was invisible is that no dashboard surfaced it. The cost graph sliced by model and by region, but not by prompt revision. The eval tracker showed pass-rate by topic, but did not annotate which prompt version produced each trace. The SLO board reported error rates by service, but had no way to ask "what fraction of errors in ap-south were generated under v46 vs v47 in the last hour?"

The instrumentation that closes this gap is mundane. Every trace, every log line, every metric tag carries a prompt_version field that propagates from the gateway down through the model invocation, the tool call, the parser, and the eval scorer. Once that dimension exists, three things become possible:

• The cost dashboard can split by prompt version. A spike in tokens-per-request in one region under one version is a real signal, not noise. Without the dimension, the same data looks like a regional traffic anomaly.
• The eval tracker can join eval runs to the production version distribution. If 12% of production traffic in the last hour ran on v46 but the eval scorer only sees v47, the dashboard says so. The team knows the eval is not representative.
• The SLO board can express drift as an alert. "More than one prompt version active in production" is a paged condition, not a Slack-channel-of-the-week conversation. The system that detects it is the same system that detects multiple model versions in production, which most teams have already wired up for canary safety.

The deeper org artifact this enables is a single source of truth for the question that should never be hard to answer in a postmortem: for this user, in this session, in this region, at this timestamp, which prompt revision did they see? When the answer is in one place, ranked by trace ID, the rollback debate ends. When the answer is reconstructed from three half-trustworthy log streams, the rollback takes longer than the original incident.

The Rollback Problem That Skew Creates

Skew makes rollback harder than the original deploy. With a clean global rollout, rolling back is "pin every region to v46." With a partial rollout, rolling back has to answer first: which regions actually took v47? If the rollout controller's record of who-took-what disagrees with the runtime reality (because of a flapping health check, a pod that restarted mid-cutover and pulled a stale value, an aggressive sidecar cache), the rollback may pin v46 in a region that is already on v46 — and miss the region that quietly drifted forward.

The mitigation is to make the runtime — not the rollout controller's intent — the authority. The controller's job is to propose a state. The runtime's job is to report the state it actually has. Rollback queries the runtime, not the controller. A region whose runtime says "I'm on v47" gets rolled back. A region whose runtime says "I'm on v46" is left alone, regardless of what the controller thought it had pushed.

The same principle handles the more annoying case: the v47 prompt itself was fine, but it was paired with a parser update that only landed in two regions. Now the rollback target depends on which regions also took the parser change. Without runtime-authoritative version tracking — for prompt, parser, tool definitions, and the model itself, all four dimensions — rollback is a guess. With it, rollback is a query.

What Holds Up

The teams that don't get burned by this share a small set of choices. They reject the framing that prompts are configuration; they treat prompts as code that happens to be a string. They couple prompt rollouts to model and parser rollouts inside a single deployment unit, so the four dimensions move together or not at all. They invest in regional cutover discipline that looks expensive on the surface but pays for itself the first time a partial rollout would have produced a six-hour ghost A/B test. They make prompt_version a top-level dimension in every dashboard, every log, every trace, every eval scorecard.

And they treat "the rollout dashboard says green" with the same skepticism a database team treats "the migration script exited zero." The script exiting is a precondition for success, not the same thing. The success criterion is the runtime — across every region, every replica, every active session — agreeing on what version it is running.

The infrastructure for this exists. Most of it is the same infrastructure your team already runs for code deploys, just applied to a class of artifact that the community has been treating as "just a string in a config file." It is not just a string. It is the behavior of your product, distributed across a global fleet, and the rollout system you use should reflect that.