The A/B Testing Trap: Why Standard Experiment Design Fails for AI Features

A team ships an improved LLM prompt. The A/B test runs for two weeks. The metric ticks up 1.2%, p=0.03. They call it a win and roll it out to everyone. Six months later, a customer audit reveals the new prompt had been producing subtly incorrect summaries all along — the kind of semantic drift that click-through rates and session lengths can't see. The A/B test didn't lie exactly. It measured the wrong thing with a methodology that was never designed for what LLMs do.

Standard A/B testing was built for deterministic systems: a button changes color, a page loads faster, a recommendation algorithm shifts a ranking. The output is stable given the same input, variance is small and well-understood, and your sample size calculation from a textbook works. None of those properties hold for LLM-powered features. When teams don't account for this, they're not running experiments — they're generating noise with statistical significance attached.

The Three Assumptions That Break

Classical experiment design rests on three assumptions that LLMs violate simultaneously.

Determinism. In a traditional feature test, showing a user variant B produces the same outcome for every identical input. LLMs don't work this way. Research on inference non-determinism found that even at temperature=0 — the setting teams reach for when they want "reliable" outputs — around 24% of GPT-4o-mini runs produced distinct outputs from the first run on the same prompt. This isn't a rounding error: continuous batching and prefix caching in inference infrastructure introduce randomness that prompt settings cannot eliminate. You're not testing a treatment; you're sampling from a distribution.

Homogeneous variance. Standard power calculations assume variance is roughly constant across the range of inputs your treatment will encounter. LLM output variance is heteroskedastic: it scales with task difficulty. Simple lookups show 5–10% output divergence across runs; complex reasoning tasks show 40–60% divergence. Your sample size estimate, built on aggregate variance, will be wildly wrong for the hardest queries — exactly the ones where getting the answer right matters most.

Independence. A/B test units are supposed to be independent: one user's experience shouldn't affect another's, and a user's treatment assignment should stay stable across their interactions. Conversational LLM systems break both. A poor response in turn one shapes what the user asks in turn two. If your assignment logic isn't deterministic across a whole session, the same user can experience both variants — a SUTVA violation that poisons your treatment effect estimate.

The Variance Problem Is Worse Than You Think

Standard A/B tests for deterministic features need large samples, but they scale predictably. For LLM features, the required sample size to achieve equivalent statistical power is typically 3–5x larger because output variance is so high. The math is unforgiving: halving the minimum detectable effect requires four times more samples. Teams running two-week experiments on LLM features are almost always underpowered unless they've explicitly recalculated required sample sizes against measured LLM variance for their specific task.

The latency confound compounds this. A new model version or restructured prompt often trades quality for speed, or vice versa. A 20% improvement in answer quality means little if it comes with a 40% latency increase that drives user abandonment before they even see the response. Teams that optimize a single outcome metric without guardrail metrics (latency, token cost, refusal rate) routinely ship "improvements" that are net negatives.

Surface Metrics Will Lie to You

The more insidious problem is that the outcomes teams measure don't capture what LLMs actually produce.

Consider click-through rate, session length, or thumbs-up feedback — the standard product metrics teams reach for in A/B tests. These metrics can stay flat or improve while the semantic quality of LLM outputs degrades significantly. A shorter, more confident-sounding response might get better engagement even if it's less accurate. A prompt change that produces more consistent outputs might lower ratings if users find less variety boring.

The reverse is equally treacherous. Research documenting a "better prompts hurt" phenomenon found that a generalized "helpful assistant" wrapper prompt degraded task-specific extraction accuracy by 10% and RAG compliance by 13% on Llama 3 — while improving instruction-following scores by 13%. Aggregate metrics would call this a wash. Slice-level analysis by task type revealed the harm. Without semantic-level evaluation broken down by intent and content type, you can't distinguish improvements from tradeoffs.

Traditional string-matching metrics make this worse. Metrics like BLEU penalize semantically correct but differently-phrased outputs. A model that says "We'll process your order today" scores differently than one that says "Your order will be handled immediately" — even though these are equivalent responses. Embedding-based semantic similarity (cosine similarity over output embeddings) is the minimum viable alternative, not a research curiosity.

What Actually Works

The methodological fixes aren't exotic, but they require changing the standard operating procedure for experiment design.

Run shadow deployments before live A/B tests. Before exposing users to a new variant, run the candidate model on production traffic silently — same inputs as the live system, no user-visible output, responses logged for comparison. Shadow testing catches obvious failures (latency spikes, systematic refusals, schema violations) at zero user impact. Only models that pass shadow validation should proceed to live A/B testing. The typical cadence is three to seven days of shadow traffic before opening the live test.

Use embedding-based evaluation as your primary quality signal. Set up an automated evaluator that computes semantic similarity between production outputs and a gold-standard reference set, or between variant A and variant B outputs on the same inputs. Cosine similarity above 0.95 indicates semantic equivalence; significant drops flag quality regression even when engagement metrics look fine. This runs as part of your CI pipeline, not just in production experiments.

Run paired analysis, not independent samples. When comparing two LLM variants, evaluate both variants on the same set of test cases rather than routing separate users to each and comparing distributions. Paired analysis leverages the correlation between responses to the same input to reduce variance by 30–50% compared to independent samples of equivalent size. This dramatically shrinks the required experiment duration.

Stratify your evaluation by task type and user segment. Aggregate metrics hide the tradeoffs that slice-level analysis surfaces. An overall quality improvement that comes at the cost of degrading a high-stakes user segment (power users, enterprise accounts, safety-critical queries) is not an improvement — but you'll only know that if you're measuring those segments separately. LLM API quality shifts show up in cohorts first; aggregate metrics catch them last.

Set sample size requirements based on measured LLM variance, not textbook formulas. Before running any live experiment on an LLM feature, measure output variance for your specific task and compute required sample sizes accordingly. A 3–5x multiplier on sample size relative to a deterministic feature is typical. If the experiment timeline doesn't allow for adequate sample size at your required minimum detectable effect, you don't have a valid experiment — you have a guess wearing statistical clothing.

Use human evaluation panels for the cases that matter most. For high-stakes features where semantic correctness is critical (legal summaries, medical information, financial advice), automated metrics are insufficient as the sole signal. Sample-efficient human evaluation using maximum discrepancy methods — evaluating only the test cases where variants diverge most — can achieve reliable rankings with a fraction of the annotations required for full corpus review. Calibrate raters with rubric alignment sessions and measure inter-rater agreement (target κ > 0.6) before treating panel results as signal.

The Testing Pipeline LLM Teams Need

The correct mental model is not "run an A/B test." It's a staged validation pipeline where each stage has explicit pass/fail criteria before advancing:

1. Offline evaluation on a fixed golden dataset (100–500 examples): confirms basic capability, semantic quality, constraint adherence, and safety. Fast, cheap, runs in CI on every change.
2. Shadow deployment on production traffic (3–7 days): catches latency regressions, API failures, and behavioral drift not visible in offline data.
3. Live A/B test with semantic evaluation, stratified sampling, guardrail metrics, and sample size calculated against measured LLM variance.
4. Continuous monitoring after rollout: LLM providers silently update models; outputs that were correct in the experiment window can degrade weeks later.

Most teams skip steps one and two and go straight to live testing, which is why their experiments are chronically underpowered, measuring the wrong things, and missing the failures that actually matter to users.

The Cost of Getting This Wrong

The A/B testing trap is subtle because it doesn't fail loudly. The experiment runs, the p-value clears the threshold, the metric moves in the right direction, and the feature ships. The degradation shows up in support tickets, churn, or a quarterly review of output quality — long after the A/B test has been declared a success and the team has moved on.

Standard experiment infrastructure was not built for systems with non-deterministic outputs, heterogeneous variance, and semantic quality that engagement metrics can't capture. Applying it uncritically to LLM features doesn't give you rigor — it gives you false confidence that's harder to detect than no experiment at all. The fix isn't to abandon experimentation; it's to build an evaluation stack that can actually see what LLM features produce, and run experiments with the sample sizes and metrics those systems require.