Your AI Feature Ramp Is Rolling Out on the Wrong Axis

A team I talked to last month ramped a new agentic feature from 1% to 50% of users over four weeks. Aggregate quality metrics held within noise. Latency stayed within SLA. They were preparing the 100% memo when the support queue caught fire — a customer with a six-tool research workflow had been getting silently corrupted outputs since the 10% step. The hard queries had been there the whole time, evenly sprinkled across every cohort, averaging into the noise floor. Nobody saw them until a single high-volume user happened to hit them at scale.

This is not a monitoring failure. It is a ramp-axis failure. Feature flag tooling — the entire LaunchDarkly / Flagsmith / Unleash / Cloudflare-Flagship category — assumes blast radius scales with the number of humans exposed. For deterministic software that is mostly true: a NullPointerException hits everyone or nobody, and showing it to 1% of users limits the user-visible blast to 1%. For AI features, blast radius does not scale on the human axis. It scales on the input axis. And the input axis is where almost no one is ramping.

The mental shift is small but consequential: stop asking "what fraction of my users see this feature" and start asking "what fraction of the hard inputs in my distribution are now flowing through the new path." The two questions sound similar. They produce wildly different rollout strategies.

Why hashing on user ID is invisible to difficulty

Percentage rollouts work by hashing a stable attribute (user ID, account ID, session ID) into a bucket and gating the feature on bucket value. The math is sound for its intended purpose: the hash is approximately uniform, so 5% of users get 5% of traffic. Crucially, the hash is independent of the query content. That independence is exactly what makes the rollout fair across users — and exactly what makes it blind to query difficulty.

Production query distributions are heavily long-tailed. Public benchmarks like Scale's enterprise tool-use leaderboard show that roughly 85% of prompts require at least three tool calls and ~20% require seven or more. Empirically the hardest 5% of queries dominate failure modes — long context, ambiguous intent, multi-hop reasoning, tools the model has never seen composed together. When you hash by user, you spread those hardest 5% evenly across every cohort. At 1% rollout, 1% of users see hard queries; at 50%, 50% of users see hard queries. The rate of hard-query failures stays constant across the ramp.

That is the trap. Every promotion gate evaluates a delta — quality at this cohort vs. quality at control — and the delta is roughly constant because the cohorts are statistically identical. The 50% step looks like the 10% step looks like the 1% step. You are not learning anything from the ramp. You are buying calendar time.

The 100% step is the first time the absolute volume of hard-query failures crosses the threshold where a single high-volume tenant or a single noisy cohort surfaces them. Or — and this is the more common pattern — a model upgrade lands during the ramp, the failure rate on hard queries doubles, and the aggregate metric still looks fine because the hard queries are 5% of traffic and a 100% relative regression on 5% of traffic shows up as a 5-percentage-point movement that gets absorbed into weekly variance.

The axes along which AI failure actually scales

There are four input-side axes where AI blast radius compounds non-linearly. Any one of them, taken seriously, is a better ramp dimension than user percentage.

Task difficulty is the most general. Difficulty here is empirical, not editorial: it is the conditional failure rate of the current system on the input. Bucketize your offline eval by historical pass rate — easy (>95% pass), medium (75–95%), hard (<75%) — and you have a difficulty stratification you can apply to live traffic via a routing classifier or an embedding-cluster lookup.

Prompt token count is a coarser proxy that is dramatically easier to instrument because it requires no classifier — just a counter. Quality regressions tend to concentrate at the long-context end where positional fragility, attention dilution, and prompt-cache eviction interact. Ramping by p50 token bucket first and p99 last forces the team to encounter the dangerous part of the distribution under controlled volume.

Query distribution by intent or topic is the slice that domain-aware teams already have in their analytics. If your support assistant handles billing, technical, and account-recovery flows, those slices have wildly different failure profiles. Rolling out billing first and account-recovery last is not a hack — it is acknowledging that account-recovery is where the model has to refuse confidently, where misfires are most visible, and where the cohort of users who file complaints is most concentrated.

Tool-call depth is the agentic axis. Failure rate compounds geometrically with depth: if each tool call has a 95% per-step success rate, a 7-step trajectory succeeds only 70% of the time. Allowing unbounded depth at any user % is rolling out the worst part of the distribution to that user fraction. Bounding depth — start at single-tool calls, then 3-step chains, then 5, then unbounded — is the only ramp that respects the actual failure curve.

These axes can compose. The strongest rollouts I have seen ramp on a primary axis (typically difficulty bucket or tool-depth) and use user-percentage as a secondary gate to bound absolute volume. The user-% becomes a rate limiter, not a quality control.

Difficulty-aware bucketing in practice

The skeptical version of the question is "but I do not have a labeled difficulty signal on production traffic." Fair. Three approximations get you most of the way:

1. Embedding-cluster lookup. Run k-means on a few thousand sampled queries from your eval set. Assign each cluster a historical pass rate from offline evals. At runtime, embed the incoming query, find the nearest cluster, and route based on its difficulty bucket. The infra cost is one embedding call and one nearest-neighbor lookup per request — well under 50ms with a hosted vector store.

2. Lightweight router model. Train a small classifier (a fine-tuned BERT-class model is overkill but plenty cheap) that predicts difficulty bucket from the prompt. The training data is your eval set with pass-rate labels. The router runs in parallel to the main feature flag check.

3. Heuristic strata. Start with the cheap proxies — token length, presence of structured input, number of conjunctions, language code, system-prompt template id. These will not be as accurate as a learned router but they cost nothing and they capture the coarse difficulty signal that matters most.

The key property: the gate fires on the input, not on the user. The same user can hit easy and hard queries within the same session, and each is routed independently. That is not a bug; it is the entire point. You are limiting blast radius on the dimension where blast radius actually lives.

Distribution-aware promotion gates

The other half of fixing the ramp is fixing the metrics that gate promotion. Aggregate quality on the cohort is the wrong metric, because the cohort is statistically identical to control. The right metric is p99 quality on tail-distribution slices.

Concretely: instrument every request with its difficulty bucket, intent slice, and tool-depth at runtime. Then evaluate the gate as "for the hard slice, is quality within X% of control? for the long-context slice, is quality within X%? for the deep-tool-call slice, is quality within X%?" A regression on any slice fails the gate, even if aggregate metrics are flat.

This is what statisticians have been calling stratified analysis since before machine learning was a thing. The reason it does not show up in feature-flag dashboards is that flag dashboards were built around uniform-cohort assumptions. Building stratified gates means the dashboard has to know about your slices. That is engineering work. It is also the engineering work that distinguishes a team that ramps AI features safely from a team that gets paged at 80%.

The variance question is unavoidable here. LLM outputs are stochastic; even on a fixed input, the same prompt at temperature > 0 produces different outputs. A common pitfall is treating a single-sample quality measurement as the gate value. The discipline that prevents this is per-input n-of-k sampling at gate-evaluation time: run each input multiple times, report the mean and the variance, and gate on the worst-case bound rather than the mean. This costs more at gate-eval but it costs less than a rollback at 80%.

Tool-depth-bounded rollout

For agentic features specifically, the depth bound is the single highest-leverage ramp primitive. Set a hard cap on max_tool_calls per trajectory and ramp the cap, not the user fraction.

The mechanics: at depth=1 the system behaves like a thin wrapper over the model's native tool calling. Errors are visible, recoverable, and bounded. At depth=3 you are exercising the planner and the recovery logic but trajectories are still short enough to debug. At depth=5+ you are in the regime where compounding failures, runaway loops, and budget-exhaustion patterns appear. The blast-radius question — what is the worst thing this feature can do to one user before the harness intervenes — is a function of depth, not of user count.

A practical ladder I have seen work: depth=1 for two days at 100% of traffic, depth=3 for three days at 100% of traffic, depth=5 for five days at 100% of traffic, then uncap. The user fraction is full the whole time. The thing being ramped is the agent's capacity to fail. This is heretical to a feature-flag team that has internalized "user percentage is the safety knob," but it matches where the actual risk lives.

What this costs and why teams resist it

The honest accounting is that ramping on the right axis costs more upfront. You need an eval set stratified by difficulty, a routing layer that knows the slice, observability that labels every request, and gate logic that evaluates per-slice rather than per-cohort. None of that comes free with LaunchDarkly.

Teams resist this because the user-percentage ramp is legible. A PM understands "we are at 50% rollout." A leadership review understands "we move to 100% next week if metrics hold." The slice-aware version is harder to communicate: "we are at 100% on easy, 50% on medium, 0% on hard, and we will lift the hard gate when p99 quality on the long-context slice is within 5% of control." It sounds like an engineering excuse. It is, in fact, what controlled rollout actually looks like for a stochastic system.

The cleanest argument for the slice-aware version is the failure mode of the alternative. The team that runs user-% to 100% and then gets paged on the hard-query tail is the team that has to roll back at 100%, lose the calendar quarter of momentum, and spend the next month explaining to the org why the metrics looked fine. That cost dwarfs the infra cost of a stratified ramp. It is the cost the team chose by picking the wrong axis.

The shift in framing

Feature-flag culture grew up around deterministic services where outcomes are bimodal: it works or it does not. AI features are not bimodal. They are quality distributions, and the variance lives in the tail. A rollout strategy designed for the bimodal world — assume uniform cohorts, gate on aggregate metrics, ramp by user fraction — masks the tail at every step.

The fix is not a new tool. It is a new question on top of the existing tools: what is the input-side distribution along which my failures concentrate, and is my ramp aware of it? In 2026 that question has at least three concrete answers — difficulty bucketing, query-distribution gates, tool-depth bounding — and the rollout primitives to implement them exist in every major flag platform if you are willing to do the slice plumbing.

The team that asks the question and builds the plumbing rolls AI features at the speed of evidence. The team that does not is calling timeline pressure "rollout" while the tail accumulates undetected. The 80% page is not a surprise. It is the audit catching up to a ramp that was never on the right axis to begin with.