Where Your Tokens Sleep at Night: Data Residency for LLM API Calls

A customer support transcript leaves a server in Frankfurt, gets concatenated into a prompt, and crosses the Atlantic to a GPU in Virginia. The model thinks for 800 milliseconds. A response comes back. From the user's perspective, nothing happened — the chat just worked. From your regulator's perspective, you transferred personal data to a third country, and you may not be able to name the legal basis for it.

This is the part of LLM adoption that demos hide. Prototypes call api.openai.com and ship. Then a procurement questionnaire from a German bank, a French hospital, or your own legal team asks a question that the prototype never had to answer: where does the inference happen, and who can compel access to it? "The provider is SOC 2 compliant" is the reflexive answer, and it is the wrong one — it answers a question about the provider's internal controls, not about which jurisdiction's courts can reach into your prompts.

Data residency for inference is not a checkbox you flip on at the end. It is an architectural property you either designed in or didn't. The good news is that the building blocks now exist across every major provider. The bad news is that they interact with each other and with extraterritorial law in ways that "we host in the EU region" does not capture. This post is about closing that gap before a regulator does it for you.

Residency Is Not Sovereignty, and SOC 2 Is Neither

Three words get used interchangeably and mean different things, and the confusion is where most compliance failures begin.

Data residency is a statement about geography: the bytes are processed and stored inside a boundary you specify — say, the EU. It is necessary for most regulatory regimes and trivially insufficient on its own.

Data sovereignty is a statement about jurisdiction: which government's laws can compel access to the data, regardless of where it physically sits. A prompt stored on a server in Paris is resident in France. If the operator of that server is a U.S.-incorporated company, the data is still subject to U.S. legal process. Residency without sovereignty is a comforting map of the wrong territory.

SOC 2 is a statement about operational controls: the provider has documented and audited processes for security, availability, and confidentiality. It tells you the vendor is unlikely to leak your data through negligence. It tells you nothing about where the data lives or whose subpoena can reach it. A provider can be flawlessly SOC 2 Type II certified and still process every one of your prompts in a region your regulator forbids.

When a questionnaire asks about data residency and someone on your team answers with a SOC 2 report, they have changed the subject without noticing. The interesting question is not "is this vendor careful?" It is "if a foreign court orders this vendor to hand over my customers' prompts, can the vendor comply — and will I even be told?"

The CLOUD Act Is Why "Hosted in the EU" Isn't the End of the Sentence

The reason residency and sovereignty diverge has a name: the U.S. CLOUD Act of 2018. It lets U.S. law enforcement compel any U.S.-based provider to produce data the provider controls, regardless of where that data is stored. Jurisdiction follows the provider, not the server.

Concretely: if your LLM vendor is a U.S. company, hosting your inference in their Frankfurt region does not place the data beyond the reach of a U.S. warrant. The data is in Germany; the company that operates it answers to a court in the United States. Microsoft's own legal officer in France testified under oath that the company could not guarantee EU data would be safe from U.S. access requests. That is not a Microsoft problem — it is structural, and it applies to every U.S.-headquartered hyperscaler and model provider.

This collides head-on with GDPR. Article 48 says a foreign court's order to disclose personal data is only enforceable in the EU if it rests on an international agreement like a mutual legal assistance treaty. The CLOUD Act asserts the opposite — that U.S. providers must comply with U.S. orders directly. Your vendor can be caught between two legal systems that each insist they get to decide. And under the CLOUD Act, the provider can be legally barred from telling you the request happened, which breaks the transparency obligations you owe your own data subjects under GDPR Articles 13–14.

Two practical consequences follow. First, "where does the data rest" is the wrong unit of analysis; "who controls the entity that operates the infrastructure" is the right one. Second, this is exactly why we now see a wave of sovereign-cloud offerings and EU-incorporated entities — AWS launched a European Sovereign Cloud in early 2026 operated through a German-incorporated company, physically and logically separated from its other regions, precisely to sever the jurisdictional thread. Whether a U.S. parent's "sovereign" subsidiary truly escapes the CLOUD Act is a live legal debate. Treat the marketing label as the start of due diligence, not the conclusion.

What the Providers Actually Give You

The encouraging development is that regional processing for inference has gone from "enterprise sales call" to "documented feature" across the board. The shapes differ, and the differences matter.

• OpenAI offers EU data residency for API traffic through a regional endpoint (eu.api.openai.com) and a region selection at project creation. You opt into where data is stored and processed.
• Azure OpenAI introduced Data Zones — a US zone and an EU zone — that process and store data within the boundary while spanning multiple regions inside it. This is the pragmatic middle ground between a single pinned region (control, but capacity and latency risk) and global routing (cheap and fast, but residency-blind).
• Google Vertex AI exposes EU regions in Belgium, the Netherlands, and Finland for regional processing.
• AWS Bedrock lets you pin inference to a region, and the new European Sovereign Cloud goes further on the jurisdictional question with a separate operating entity.

A subtlety that trips people up: a region for storage is not automatically a region for processing. Even data that exists only for milliseconds during inference — never written to disk, never logged — counts as a cross-border transfer if it's processed outside your boundary. The definition regulators apply is "where the data moves," not just "where the data sleeps." A configuration that stores logs in Frankfurt but routes the actual token generation to a global pool has not achieved residency; it has achieved the appearance of it.

Retention is the other axis. Knowing how long the provider keeps prompts and completions is part of the residency story — shorter, region-pinned retention windows shrink both your exposure surface and the volume of data any subpoena could reach. Read the data-handling terms, not the homepage.

Build the Boundary at the Gateway, and Make It Fail Closed

You cannot enforce residency by asking every engineer to remember to call the right endpoint. The control has to live in infrastructure, and the natural place is an AI gateway — a proxy that sits between your application and every model endpoint, making the routing decision on policy rather than on whoever wrote the call.

A residency-aware gateway does a few things no individual SDK call does well:

• Region-locked routing. The gateway decides, per request, which endpoint serves it based on the data's jurisdiction — EU-tagged traffic only ever reaches EU endpoints. The application doesn't get a vote.
• No silent cross-region fallback. This is the one that actually saves you. When the EU endpoint is over capacity, the tempting behavior is to fail over to a US region so the user isn't blocked. That exception path is where most real-world residency violations happen — not in the steady state, but in the 3 a.m. degraded mode nobody tested. A correct gateway fails closed: if the request can't be served inside the boundary, it errors rather than quietly crossing the border.
• Region-local audit logs. The record of who sent what, when, and where it was processed has to stay inside the boundary too. An audit trail that ships to a US logging cluster reintroduces the transfer you spent all this effort avoiding.
• One enforcement and record surface. Whether the traffic terminates at OpenAI, Anthropic, Bedrock, or an internal model, the policy and the logging are uniform. Residency becomes a property of the platform, not of each team's discipline.

The deployment location of the gateway itself is part of the boundary. If the proxy that enforces your EU policy runs in us-east-1, the prompts traverse it on US soil before reaching the EU model. Run the gateway where the data is required to stay.

When Geography Isn't Enough: Encryption and Key Control

Sometimes the jurisdictional problem can't be solved by moving servers, because the only acceptable provider is a foreign-controlled one, or because the workload genuinely needs a model that only runs in a region you'd rather avoid. The supplementary measure the post-Schrems II world settled on is technical rather than territorial: encrypt before the data crosses, and hold the keys in a jurisdiction you control.

If the prompt is encrypted with keys held by an EU-controlled entity, encrypted data can sit on third-country infrastructure because the operator there has no ability to decrypt it. A CLOUD Act order served on the US provider yields ciphertext. This is harder to apply to inference than to storage — the model has to see plaintext to generate a completion — but it shapes the architecture around the call: customer-managed keys, split-key or m-of-n approval for any decryption, and keeping the cleartext window as narrow and as local as possible. For the highest-sensitivity workloads, this is also the argument for self-hosting an open-weights model inside your own boundary, where no third party ever holds the plaintext at all.

The point of a transfer impact assessment is to document exactly this reasoning: what data moves, where, under whose jurisdiction, and what technical measures make the residual risk acceptable. It is not data localization for its own sake — it's a defensible, written argument that survives a regulator reading it two years later.

The Question to Ask Before You Ship

The market is moving in one direction. Gartner reported inquiries about cloud sovereignty rising over 300% in the first half of 2025; analysts expect a large share of countries to lock into regional AI infrastructure rules by the end of 2026; the EU AI Act's core obligations became enforceable in August 2025 with penalties reaching into the tens of millions of euros. Residency is not a niche enterprise concern drifting toward the mainstream — it is already there, and retrofitting it onto a system that assumed a single global endpoint is far more expensive than designing for it on day one.

So before the next feature ships, ask the question the demo never had to: for this specific prompt, name the jurisdiction it is processed in, the entity that operates that infrastructure, and what happens on the day the EU region is full. If the honest answer to any part of that is "I'm not sure" or "it falls back to US," you don't have a residency story — you have a latency optimization that a regulator will eventually reclassify as an unlawful transfer. Build the boundary now, put it in the gateway, and make it fail closed. Your tokens should sleep somewhere you can point to on a map and defend in a deposition.