19 posts tagged with "gdpr"

A deletion request lands in your queue. A user has invoked their right to erasure, and legally you have a month to make their personal data disappear. In a normal system this is a DELETE statement and a smug audit-log entry. In an AI system it is the moment you discover that your data doesn't live in one place — it has been smeared across a fine-tuned model's weights, baked into a vector index, cached in a dozen retrieval snapshots, and copied into last quarter's evaluation set. There is no single row to delete. The user is, in a very literal engineering sense, undeletable.

This is not a hypothetical. In March 2025 the European Data Protection Board launched a coordinated enforcement action across thirty national authorities focused specifically on the right to erasure. Regulators have converged on an uncomfortable position: including someone's data in training is processing, so Article 17 applies to the model, not just the database. The question every AI team eventually faces is whether output suppression — teaching the model to refuse to talk about someone — is enough, or whether you actually have to remove the influence of their data from the system. The honest answer is that most teams have never designed for either.

Somewhere in your company's data classification policy, there is a table. Raw customer text — support tickets, medical notes, chat transcripts — sits in the "sensitive" row, wrapped in encryption requirements, access controls, and deletion SLAs. And then there is your vector store, holding embeddings of that exact text, classified as... nothing. Derived data. Anonymous math. Just floats.

That classification is wrong, and it is wrong in a way that is now experimentally demonstrated. Inversion attacks can reconstruct the original text from its embedding — in the best-studied setting, recovering 32-token inputs exactly in 92% of cases, including full names from clinical notes. If an attacker with your vectors can read your customers' words, your vectors inherit the sensitivity of those words. The regulators have started saying this out loud, and most retrieval architectures are not ready for what follows: a deletion request that has to reach every index, every snapshot, and every derived artifact — not just the row store.

A customer support transcript leaves a server in Frankfurt, gets concatenated into a prompt, and crosses the Atlantic to a GPU in Virginia. The model thinks for 800 milliseconds. A response comes back. From the user's perspective, nothing happened — the chat just worked. From your regulator's perspective, you transferred personal data to a third country, and you may not be able to name the legal basis for it.

This is the part of LLM adoption that demos hide. Prototypes call api.openai.com and ship. Then a procurement questionnaire from a German bank, a French hospital, or your own legal team asks a question that the prototype never had to answer: where does the inference happen, and who can compel access to it? "The provider is SOC 2 compliant" is the reflexive answer, and it is the wrong one — it answers a question about the provider's internal controls, not about which jurisdiction's courts can reach into your prompts.

A compliance program is a description of the systems your company had on the day the auditor signed off. The systems your company has today are a different set, and the gap is the surface area of every release that shipped a new persistent store between then and now. The deletion guarantee you sold your customers is a guarantee against the first set, and the regulator who eventually asks about it will be asking about the second.

The failure mode is not a bug in the deletion code. The deletion code is correct. The saga fans out across every storage system named in the data inventory, calls each one's erasure endpoint, collects a receipt per system, and reports success when every receipt comes back signed. The saga is doing exactly what it was built to do. The problem is that the saga is iterating over a list of storage systems that was true eighteen months ago, and the agent platform team shipped a long-term memory feature six months ago that nobody added to the list.

Your residency audit traced every outbound request from the tenant's traffic, watched it terminate on a hostname in Frankfurt, and signed off. The audit was correct about everything it measured. It was also looking at the wrong layer. The request went to the EU. The bytes that satisfied the request — the cached prefix the provider hashed and pulled from the nearest available node — lived in us-east-1. Your regional endpoint promised you a destination. The cache promised nothing, because the cache was a different product, governed by a different SLA, designed for cost rather than for compliance.

The customer's auditor caught it. Not yours. A different vendor's incident report mentioned that prompt cache placement was decoupled from inference region, and the customer's GRC team asked the obvious follow-up question: where do our prefixes go? The contract amendment to close the gap took ninety days. The renewal got suspended. The team that wrote the integration had done nothing wrong by the documentation they were handed.

A PII redactor that runs only on inbound traffic is a one-way valve installed at the wrong end of the pipeline. It catches user-submitted names, emails, and account numbers before they reach your logs. It does nothing about the model's own outputs — the place where the same model is now actively assembling text that may contain those same identifiers, drawn from RAG retrievals, tool returns, conversation history, or content the user pasted from another tenant's data. Every team I've watched ship an input-side redactor has a follow-up ticket in the backlog labeled "output-side parity." Most of those tickets never close, because no incident surfaces the gap for six months, and after six months the ticket has accumulated enough re-prioritization to look like a feature request rather than a missing half of a security control.

The failure mode is invariant: input redaction is treated as the canonical control because it is the easier engineering problem and the easier audit story. You wrote a regex set, you ran a labeled benchmark, you proved precision and recall on a fixed corpus, you shipped it behind a feature flag, and the security review accepted it as the PII boundary. The output side has none of that benefit. The model's response is generative, the surface area is unbounded, and the test methodology — "what should it not say in any of infinitely many contexts" — is structurally harder than "what should we strip from a known input." So the team that ships the inlet treats the outlet as future work and the future never arrives until a customer reports another customer's email landing in their transcript.

A customer audit finds eleven months of verbatim user PII sitting in a Redis cluster nobody on the residency team knew existed. No system was compromised. No attacker got in. The data was written there on purpose, by a service the inference team built and named "prompt cache," as a performance optimization. The redactor on the analytics path worked perfectly the entire time. The redactor was simply not on this path.

The breach is real anyway. Under GDPR, retention beyond the contracted thirty days is enough; the data does not need to have leaked to trigger Article 33 notification obligations. The residency team's inventory listed every log, every warehouse, every queue — and missed the cache because the cache was on the inference team's side of the org chart. The privacy boundary that everyone trusted ran straight down the analytics pipeline and stopped at the wall where the LLM stack began.

The team that ships "AI for EU customers" usually ships exactly one residency control: an inference endpoint pinned to an EU region. The procurement team gets a DPA, the architecture diagram gets a green checkmark next to "model hosted in Frankfurt," and the launch proceeds. What the diagram doesn't show is that the customer's verbatim query gets vectorized by a US-hosted embedding API on its way to the model, that the vector store the query is matched against has its operational plane in us-east-1, that the rerank model is a third-party SaaS deployed wherever the vendor chose, that the prompt cache is keyed regionally on hits and globally on misses, and that the trace store logging the retrieved chunks has a 30-day retention bucket that replicates cross-region for redundancy.

The inference layer respects residency. The retrieval pipeline doesn't even know it's a participant.

This is the gap where most "GDPR-compliant" RAG deployments fail an audit the team didn't realize was coming. The fix isn't another control on the model call — it's recognizing that data residency is a property of every component the customer's bytes touch, and that the team owning "the LLM" owns at most one of the six surfaces involved.

The first compliance escalation against your agent memory layer almost never arrives as a regulator's letter. It arrives as a Jira ticket from your enterprise sales engineer that says "the customer's privacy team is blocking the contract — they want to know what 'forget my user' actually means in your system, and they want a written answer by Friday." That ticket lands six to twelve months after the memory layer shipped, and the engineering team that built it discovers, in the time it takes to read the question, that they accidentally built a records-management system without any of the primitives a records-management system is supposed to have.

This is the structural problem with long-term memory in agentic products. The team building it optimizes for the things memory is sold to do — retrieval quality, latency, storage cost, the felt-personalization that makes the assistant feel like it knows the user. Nobody in the design review prices the parallel system being built at the same time: a per-user, per-tenant, multi-region data store with retention obligations, deletion semantics, audit export requirements, and a regulator's clock that starts the moment the first user's data lands in it. Memory is not a feature. It is the operational surface that every privacy regime, every enterprise procurement questionnaire, and every right-to-erasure request will eventually find.

Your golden eval set is a privacy boundary your security team didn't know existed. It is built by sampling production traces, which means it is a curated collection of real customer queries — often containing names, emails, account numbers, transcripts of frustrated calls, half-typed credit card digits — paired with the canonical correct response on top, and then committed to whatever bucket the eval pipeline reads from.

That last part is what makes eval data uniquely dangerous. A raw production trace is sensitive because it captures what the customer said. An eval case is sensitive in a new way because it captures what the customer said plus the labeled correct answer. The label is a derivative work that someone, often an annotator or a domain expert, applied with intent. It signals "this is canonical." It gives the trace a longevity that the original log never had — log retention will eventually rotate the trace out, but the eval case is now a permanent test fixture that the team is committed to keeping green.

A customer files a subject-access request asking for their data to be deleted. The data engineer purges the production database, the analytics warehouse, the support ticket archive, the cold-storage backups. Every system the legal team listed in the data inventory comes back clean. Then somebody in the room asks the question that nobody wants to answer first: what about the model?

Three months ago that customer's support transcripts went into a fine-tuning run. The resulting adapter has been serving predictions to other customers ever since, with their phrasing, their account names, occasionally their literal sentences embedded in the weights. You can prove deletion in the warehouse. You cannot prove deletion in the model — and the more honest member of the team is the one who says so out loud.

Your AI feature launched. Users are engaging with it. The gap between what it does and what it should do is visible in every session replay, every thumbs-down, every request that returns a wrong answer. You have the signal. The question is whether you can legally act on it.

This is where teams hit the compliance wall. Not a theoretical wall — a concrete one. In 2024 alone, European regulators issued over €1.2 billion in GDPR fines, with OpenAI, Meta, and LinkedIn among the named defendants. The common thread across most enforcement actions: using behavioral data in ways that weren't explicitly scoped at collection time, or collecting more than was necessary to operate the feature. The fact that your intent is model improvement rather than advertising doesn't move regulators the way engineers assume it does.