64 posts tagged with "compliance"

A customer disputes a loan denial. A regulator opens an inquiry. A plaintiff's lawyer files discovery. All three arrive with the same deceptively simple question: what did your system decide, and why? The decision happened eighteen months ago. You pull up the case, and every component that produced the original output has since moved on. The hosted model version was deprecated and migrated. The system prompt has been edited nine times. The documents your agent retrieved were re-chunked, re-embedded, and re-ranked into a new index. And the sampling settings that made the whole thing non-deterministic were never logged in the first place.

You cannot reproduce the decision. Not because you were careless, but because nothing in your stack was built to be reproduced. Explainability-after-the-fact turns out to be a reproducibility problem in disguise — and reproducibility is something you either engineer in at decision time or lose forever.

The uncomfortable truth is that most teams discover this gap exactly when they can least afford to. The demand to reconstruct a decision almost never comes during normal operations. It comes attached to a lawsuit, an audit, or an angry customer with a regulator's phone number, and by then the window to have captured the right evidence closed a year and a half ago.

You instrumented your agent the way every observability guide told you to. Every turn gets a span: the full prompt, the retrieved chunks, the model's chain of thought, the tool calls it considered, the plan it drafted and then abandoned, the final action. Storage is three cents a gigabyte and the traces have saved you a dozen times during incident response. You kept everything because keeping everything was free and occasionally heroic.

Here is the part nobody on the platform team modeled: that trace is written in fluent English, it speculates about your customers by name, and it drafts actions you never actually took. It is the single most quotable artifact your company produces, and a plaintiff's attorney will read the juiciest 200 words of it to a jury with your logo on the slide behind them. The reasoning trace that makes your system debuggable is the same reasoning trace that makes it liable.

Somewhere in your company, an engineer has wired an agent fleet to a SaaS product licensed for fifty human seats. The fleet runs a thousand concurrent sessions overnight, authenticates through one service account, and does the work of a department. Nobody asked legal. Nobody asked the vendor. And if you read the contract carefully, there's a decent chance that entire workload is unlicensed — not because anyone intended to cheat, but because the contract was written for a world where "user" meant a person with a keyboard.

This is the quiet compliance problem of the agent era. Per-seat licensing, terms-of-service automation clauses, fair-use rate limits, and per-user analytics were all designed against a single assumption: usage scales with headcount. Agents break that assumption in every direction at once. A customer running agent fleets doesn't just stress the vendor's pricing model — they often violate the letter of the agreement, the economics behind it, or both.

A customer support transcript leaves a server in Frankfurt, gets concatenated into a prompt, and crosses the Atlantic to a GPU in Virginia. The model thinks for 800 milliseconds. A response comes back. From the user's perspective, nothing happened — the chat just worked. From your regulator's perspective, you transferred personal data to a third country, and you may not be able to name the legal basis for it.

This is the part of LLM adoption that demos hide. Prototypes call api.openai.com and ship. Then a procurement questionnaire from a German bank, a French hospital, or your own legal team asks a question that the prototype never had to answer: where does the inference happen, and who can compel access to it? "The provider is SOC 2 compliant" is the reflexive answer, and it is the wrong one — it answers a question about the provider's internal controls, not about which jurisdiction's courts can reach into your prompts.

A compliance program is a description of the systems your company had on the day the auditor signed off. The systems your company has today are a different set, and the gap is the surface area of every release that shipped a new persistent store between then and now. The deletion guarantee you sold your customers is a guarantee against the first set, and the regulator who eventually asks about it will be asking about the second.

The failure mode is not a bug in the deletion code. The deletion code is correct. The saga fans out across every storage system named in the data inventory, calls each one's erasure endpoint, collects a receipt per system, and reports success when every receipt comes back signed. The saga is doing exactly what it was built to do. The problem is that the saga is iterating over a list of storage systems that was true eighteen months ago, and the agent platform team shipped a long-term memory feature six months ago that nobody added to the list.

A RAG agent answers a customer's regulatory question with a tidy paragraph and a citation. The verification layer fetches the URL, sees a 200 OK, ticks the box, and ships. Six months later a compliance audit pulls the transcript, clicks the same link, and finds a page that now says the opposite of what the agent quoted. The URL is fine. The quote is fine in the transcript. The two no longer match. The customer's compliance officer asks whether the agent fabricated the quote, and the team cannot prove it didn't, because the only surviving evidence of what the URL used to say is the agent's own assertion of what it said.

This is not a hallucination in the usual sense. The model retrieved real content, faithfully extracted a real sentence, and emitted a real URL that still resolves. Every link-checker on earth would call this citation valid. The audit fails anyway, because the verification layer was measuring the wrong property. Reachability is not fidelity. A URL is a pointer to a mutable document under someone else's editorial control, and the moment the document changes, every transcript that quoted it becomes a hallucination report waiting to happen.

Your residency audit traced every outbound request from the tenant's traffic, watched it terminate on a hostname in Frankfurt, and signed off. The audit was correct about everything it measured. It was also looking at the wrong layer. The request went to the EU. The bytes that satisfied the request — the cached prefix the provider hashed and pulled from the nearest available node — lived in us-east-1. Your regional endpoint promised you a destination. The cache promised nothing, because the cache was a different product, governed by a different SLA, designed for cost rather than for compliance.

The customer's auditor caught it. Not yours. A different vendor's incident report mentioned that prompt cache placement was decoupled from inference region, and the customer's GRC team asked the obvious follow-up question: where do our prefixes go? The contract amendment to close the gap took ninety days. The renewal got suspended. The team that wrote the integration had done nothing wrong by the documentation they were handed.

Your counsel approved a one-line system-prompt directive: append "This information is not legal advice and should not be relied upon as such" to every response touching a regulated domain. Three weeks later, a user files a bug because their calendar event's description field opens with that same line, followed by a contract summary the agent was supposed to put into a meeting invite. The agent did not malfunction. It did exactly what the system prompt told it to do, which turned out to be a behavior that ranges over every channel the model produces text into — including the JSON arguments of the next tool it called.

The instruction was a content-formatting rule and the model treated it as one. It did not distinguish "user-facing response" from "tool call argument" because nothing in the prompt told it those were different surfaces. The disclaimer ended up in the calendar, in the email draft, in the Slack message your agent posted on the user's behalf. Each of these was a separate downstream system whose author had no idea a compliance string was about to be injected into a structured field, and each had a different cleanup cost.

The team that ships "AI for EU customers" usually ships exactly one residency control: an inference endpoint pinned to an EU region. The procurement team gets a DPA, the architecture diagram gets a green checkmark next to "model hosted in Frankfurt," and the launch proceeds. What the diagram doesn't show is that the customer's verbatim query gets vectorized by a US-hosted embedding API on its way to the model, that the vector store the query is matched against has its operational plane in us-east-1, that the rerank model is a third-party SaaS deployed wherever the vendor chose, that the prompt cache is keyed regionally on hits and globally on misses, and that the trace store logging the retrieved chunks has a 30-day retention bucket that replicates cross-region for redundancy.

The inference layer respects residency. The retrieval pipeline doesn't even know it's a participant.

This is the gap where most "GDPR-compliant" RAG deployments fail an audit the team didn't realize was coming. The fix isn't another control on the model call — it's recognizing that data residency is a property of every component the customer's bytes touch, and that the team owning "the LLM" owns at most one of the six surfaces involved.

A platform team shipped a prompt refactor that cut average response cost by thirty-two percent. The change was simple: strip the "explain your reasoning" preamble, ask the model to return only the JSON object, and drop the post-processing step that parsed the rationale out of the model's prose. The dashboard turned green. The unit economics page in the quarterly review went from yellow to gold. Nobody on the platform team thought to consult the risk team, because no part of the change touched the answer the customer received.

Two quarters later, a regulated customer's auditor requested the decision rationale for a denied-loan letter from a date six months prior. The team pulled the trace. The input was there. The output was there. The reasoning was gone — not because anyone deleted it, but because it had stopped being produced the day the refactor shipped. The customer's compliance program had been operating on the assumption that the rationale was somewhere in the trace store; the platform team had been operating on the assumption that the rationale was nobody's problem because the customer-facing answer was unchanged. Both assumptions were correct in isolation. Together they cost the customer a regulatory finding and the platform team a contract renewal.

The fine-tuned checkpoint that has been running in production for nine months is now sitting in a Slack thread between your CTO and outside counsel. A data source that you scraped under what looked like a permissive license has changed its terms, sent a notice, and named your model. Your engineers want to know whether the model can simply be "untrained" on the offending records. Counsel wants to know whether the weights file itself is now a regulated artifact. Nobody on the call has a good answer, because your training pipeline treated the license as an event — read once at ingestion time — instead of a state that the world can edit after you have already paid for the H100s.

This is the failure mode that very few fine-tuning playbooks bother to discuss. The license under which a dataset was distributed is not a static gate that you walk through at ingestion. It is an ongoing claim by a third party that you do not control, and the half-life of that claim is shrinking. Hugging Face's own legal repository quietly logs DMCA takedowns against named datasets every few weeks — AoPS pulling the MATH benchmark, PaperDemon pulling scraped artwork, Archive of Our Own removing a fanfiction dump within hours of notice. Each takedown is a downstream signal that some model somewhere was trained on data whose redistribution rights have since evaporated.

The compliance audit always starts with the same question and your team always answers it the same way. "Where is customer data processed?" In the EU region, the slide deck says, and the SDK config screenshot confirms it, and the DPA promises it. Then the auditor pulls a sample of last quarter's request logs, joins them to the provider's per-request region header, and the room gets quiet. Something like four percent of EU enterprise prompts were served by a US-region inference node during a forty-minute capacity event the team did not know happened. The cache that holds reusable prefixes was in the global pool. The trace store the support team queries is in us-east. The DPA was a slide deck. The contract was a routing hint.

This is the kind of incident that does not show up in a postmortem because no service degraded. The model returned an answer, the user got a response, the latency graph stayed flat. The thing that broke is a thing the dashboards were never wired to see: the geographic path of the request through the provider's infrastructure. Engineers who would never confuse a us-east-1 URL with "the request actually executed in us-east-1" routinely make that exact mistake at the LLM API layer, because the provider's region parameter looks like the AWS one, behaves like the AWS one in the happy path, and silently degrades to "best effort" the moment the preferred region runs out of GPU.