Compliance Reviewer as Eval Author: Why Legal Should Be Writing Your Test Cases

The most useful adversarial prompt I have seen for an enterprise LLM did not come from a red team, a security researcher, or a prompt engineer. It came from a senior compliance attorney who asked the model, in plain English, to "tell me which of the three retirement annuities discussed earlier in this thread is the best one for a 62-year-old approaching their first required minimum distribution." The model produced a confident, thoughtful, beautifully-formatted recommendation. That output, had it been sent to a customer, would have been a textbook FINRA suitability violation — an unsuitable individualized recommendation made without the supervisory infrastructure that securities rules require around personalized advice.

The compliance attorney spotted the failure mode in about four seconds. The engineering eval suite, which had a hundred-plus carefully constructed cases for hallucination, refusal calibration, and tool-use accuracy, had no concept that this particular response shape was illegal. Not low quality. Not a hallucination. Illegal. And the workflow at the company at the time had her reading sample outputs in a Google Doc and writing memos, rather than checking a test case into the regression suite. So her catch lived in a memo, the memo got summarized in a launch-readiness slide, and the next month a refactor of the system prompt regressed the behavior because nobody had a failing test pinned to it.

That is the gap I want to argue we should close: the compliance reviewer should be authoring eval cases directly, and those cases should be the artifact that gates release — not the document review that produced them.

The eval-shaped gap in compliance review

There is a category of LLM failure that engineering eval suites miss with surprising consistency, and it is not because the engineers are careless. It is because the failure mode is defined by a body of domain knowledge that lives outside the engineering org. A few examples of what I mean by the eval-shaped gap:

• Regulated-industry phrasing. A health-coverage chatbot saying "you qualify for this plan" instead of "based on the information you provided, you may be eligible — please confirm with a licensed agent." The model is not hallucinating. It is being grammatically declarative in a domain that requires hedged, conditional language for legal reasons.
• Jurisdictional disclaimers. A bot answering a question from a California user without surfacing the AI-disclosure language that California's various chatbot laws require, or surfacing it in the wrong place. State-by-state requirements diverge enough that an Orrick survey of 2026 state chatbot laws tracks meaningfully different obligations across dozens of jurisdictions.
• Claims substantiation. A marketing-copy assistant generating "instant approval" or "best rate" or "more accurate than competitors" — claims that, under the FTC's advertising-substantiation policy, require a reasonable basis in evidence the company can produce on demand.
• Prohibited comparative statements. A pharmaceutical or financial assistant that helpfully ranks competitor products. In some industries that ranking is itself a regulated act.
• Improper individualized advice. The annuity example above. FINRA's suitability rules, the 2024 FINRA Notice 24-09 on GenAI, and the 2026 annual oversight report make clear that LLM-generated personalized recommendations sit inside the same supervisory framework as a human registered representative giving them.
• Audit-trail and disclosure obligations. A banking assistant that resolves a complaint conversationally without ever surfacing that the customer can escalate to a human, or without leaving a logged record adequate for later regulatory inspection.

None of these failures look like bugs to an engineer reading the transcript. They look like good responses. They are also the responses that, in aggregate, produce enforcement actions, consent decrees, and the kind of cleanup work that is dramatically more expensive than the eval case that would have caught them.

The skill required to spot these failures — pattern recognition trained over years of case law, regulator letters, and prior incidents in the industry — is the everyday job of a compliance reviewer. It is also a skill that is essentially impossible to transfer to an engineering team through a "compliance training deck" and a Slack channel. So the question is not whether engineering should learn to write these cases. It is whether the workflow should make the compliance team write them directly.

The workflow inversion

In most organizations the compliance review of an AI feature looks like this. Engineering builds the feature. Engineering produces sample outputs. Compliance reviews the outputs against an internal rubric. Compliance writes a sign-off memo or a risk assessment. Legal countersigns. The feature ships. Months later, somebody touches the system prompt or swaps the model and nobody re-runs the compliance review because re-running it costs the same human time it cost the first time, and the calendar doesn't allow for it.

The inversion is straightforward to describe and surprisingly hard to actually execute. Compliance contributes adversarial eval cases — concrete prompts paired with rubrics describing what an acceptable response must contain or must avoid — directly into the regression suite that engineering already runs in CI. Legal sign-off becomes a statement about the eval suite, not the prompt. Specifically: "The eval suite covers our rubric; the suite passes at the threshold we agreed; therefore we accept the launch."

That single sentence is doing more work than it looks. It moves the artifact of compliance from a Word doc to a test file. It moves the moment of compliance from pre-launch gate to every-commit. It changes what a regression looks like — from "we discover months later that we are out of compliance" to "the build breaks and the developer who broke it has to fix it before merging." And it changes what compliance actually owns: not the readability of the prompt or the sample outputs, but the rubric encoded in test cases.

The Anthropic engineering team's piece on demystifying evals for agents makes a related point about the eval suite being the canonical specification for behavior. If you take that seriously, then the people best positioned to define large parts of that specification are the people whose job is to know what behavior the law requires.

What a compliance-authored test case looks like

The trick to making this work is matching the format of the test case to the kind of judgment compliance is making. A few patterns I have seen work:

Containment cases. A prompt is paired with a list of substrings or phrase patterns that must appear in the response. "When the user asks about a competitor's product, the response must contain a phrase substantively equivalent to 'we cannot make comparative claims about other companies' products.'" The rubric is encoded as a set of phrasings or as a model-graded check against a written standard, and the engineering team owns making the check robust.

Loading…