Bug Bashes for AI Features: Sampling a Distribution, Not Hunting Defects

The classic bug bash is a deterministic ritual built for deterministic software. Ten engineers crowd a Slack channel for two hours, hammer a checklist of golden-path flows, and file tickets with crisp repro steps: "Click X, see Y, expected Z." It works because the system under test is reproducible — same input, same output, same bug, every time.

Run that exact ritual against an AI feature and you will produce two hundred tickets, close one hundred and eighty as "expected stochastic variation," and miss the twenty that signal a real cohort regression. The format isn't just stale; it's actively miscalibrated. A bug bash against an LLM-backed feature is not a defect-hunting session. It is a sampling exercise against a probability distribution, and the team that runs it like a deterministic test session is collecting noise and calling it signal.

This post is about how to redesign the bug bash for stochastic systems — what to change about the format, the participants, the triage rubric, and what counts as "done."

Why the Inherited Format Fails

Three assumptions are baked into the traditional bug bash, and an AI feature breaks all of them.

The first assumption is reproducibility. In classic QA, "I clicked the button and it crashed" is a complete report. In an AI feature, the same prompt can return five different answers in five consecutive runs and one of them may be dangerously wrong. A single observation is statistical noise. A bug report without a repro count is not a bug report — it's an anecdote.

The second assumption is the failure mode is structural. Buttons don't render, the form doesn't submit, the API returns 500. These are categorical failures: present or absent. AI features fail along a quality gradient. The button renders; the answer is just subtly wrong for users in a specific industry. Or the answer is correct, but its tone is off in a way that erodes trust. A binary "broken / not broken" rubric can't represent these failures, so they get filed as "minor" or not at all.

The third assumption is engineers can recognize the failure. In a deterministic system, a 500 error looks the same to everyone. In a domain-specialized AI feature — legal contract review, medical summarization, code refactoring for a specific stack — the failures that matter are domain-correctness failures. An engineer reading a clinical note summary cannot tell if "post-op day three" was misclassified as "post-op day two." A lawyer can. The failure is invisible to the people you invited.

Stack these three together and the inherited format becomes worse than useless. It produces an avalanche of unreproducible, miscategorized, surface-level reports that drown out the few that signal a real, exploitable pattern. Worse, it gives the team false confidence: "We bug-bashed it, we found stuff, we shipped it."

Reframe the Goal: From Defect Discovery to Distribution Mapping

The mental shift that has to land is this: a bug bash for an AI feature is not finding defects, it is mapping the failure distribution of a system that was always going to have failures. You are not asking "is it broken?" — you are asking "where does it fail, how often, and on which slice of users?"

That reframe changes everything downstream. You no longer want each participant pounding the same golden flow. You want them spreading out across the input space — different user cohorts, different task types, different inputs that probe different parts of the model's competence. Each session is a sampling pass; each ticket is a sample, not a verdict.

Concretely, before the bug bash starts, the organizer should pre-define the cohort grid: industry vertical, account tier, language, task type, novice vs. power user, short vs. long input. Each participant gets a cell. If you have eight people and four cohorts, two people sample each. If you have a dozen and six cohorts, two per cell with deliberate overlap so you can compare independent samples on the same slice.

This is not a small reorganization. It changes who you invite, how you brief them, what you ask them to file, and how you triage afterward.

Adversarial Inputs Beat Golden-Path Inputs

The other shift is adversarial. A traditional bug bash exercises the happy path — the flows the product manager wants to demo on Friday. For a deterministic system, that's reasonable; if the happy path works, exotic inputs probably also work, because the underlying logic is the same.

For AI, the happy path tells you almost nothing. Models are over-trained on the kind of inputs that look like product demos. The prompts that distinguish a robust feature from a fragile one are the ones the demo would never show: typos, mixed languages, half-finished sentences, deliberately misleading framings, prompt injection attempts, inputs that conflict with the system prompt, inputs at the token-limit boundary, inputs in the user's actual messy production format rather than the synthetic clean format the eval suite uses.

Red-teaming research has converged on this point. Manual adversarial testing excels at uncovering the nuanced edge cases that automated suites miss, especially multi-turn conversational attacks where the failure builds across turns. Recent work has shown attack-success rates climbing past 80% for techniques like role-play impersonation and even rhyming-stanza obfuscation that would never appear in a benchmark dataset. Your bug bash needs to channel this energy.

Practically, that means giving each participant an adversarial brief along with their cohort assignment. "You are a user in healthcare-vertical small-enterprise tier. Try to make the assistant give medical advice it shouldn't. Try to make it confidently cite a paper that doesn't exist. Try to make it leak the system prompt. Try to make it disagree with itself across two turns." The brief replaces the golden-path checklist.

Repro Counts Are the Ticket Format

If a deterministic bug ticket is "steps to reproduce," a stochastic bug ticket is "steps to reproduce plus observed frequency." Without the frequency, you cannot triage. A failure that happens once in fifty runs against the same input might be statistical noise; a failure that happens four times in five runs is a real bug. Both look identical in the moment of observation.

The triage rubric needs to enforce this. Tickets without a repro count get bounced back, not closed. The minimum bar is: "I ran this prompt N times, the failure occurred K times." Five trials is a reasonable floor for a bug bash; ten is better but slower. Tooling matters here — give participants a one-click "rerun the same prompt five times and capture all outputs" button, because asking humans to manually retry the same prompt is how you get inflated repro counts and frustrated participants.

A good rubric distinguishes three frequency bands: rare (1 in 5 or less, tag as anecdote, do not promote), occasional (2–3 in 5, promote to eval candidate, investigate before shipping), and reliable (4–5 in 5, treat as a real bug, block release until fixed). The exact thresholds are tunable, but the principle is non-negotiable: frequency is part of the data point, not metadata.

This also means you cannot triage tickets in real time during the session the way you can with a deterministic bug bash. A failure observed at minute 30 needs to be re-run, possibly by a second participant on a different cohort, before you know if it is real. Build a 30-minute synthesis block at the end where flagged tickets are re-sampled, not just discussed.

Domain Experts in the Room, Not the Loop

The participant pool is the third lever. A bug bash run only by engineers will catch engineering-shaped bugs: latency spikes, broken streaming, malformed JSON, retry storms. It will miss every domain-correctness failure that the feature actually exists to handle correctly.

For a clinical-summary feature, you need clinicians in the bash. For a contract-redline feature, you need lawyers. For a code-refactor feature targeted at a specific framework, you need engineers fluent in that framework, not generalists who can read any code. Research on domain-expert involvement in LLM evaluation consistently finds that the highest-leverage feedback is the kind only the domain expert can give: "this answer is technically grammatical but no working clinician would phrase it that way."

The hard part is the operational ask. Domain experts are expensive, fatigue quickly with repetitive tasks, and often have weak AI literacy when asked to design evaluation criteria from scratch. The bug bash format has to accommodate that. Pair each domain expert with an engineer who handles tooling, captures repro counts, and translates "this is wrong" into a structured ticket. Cap the expert's session at 60–90 minutes — fatigue past that point produces ratings that regress to the mean and stop being useful. Pre-build the input set so the expert is reviewing outputs against realistic inputs, not generating prompts under time pressure.

The synthesis pass is where the engineer-expert pair pays off. The expert flags something subtle; the engineer turns it into a deterministic eval case that can be re-run on every model bump from now on. That conversion — anecdote to eval case — is the actual deliverable of the bug bash. Tickets that get filed and then closed without becoming evals were wasted effort.

What Done Looks Like

A traditional bug bash ends when the ticket count plateaus or the time runs out. Either the bugs got fixed or they got triaged into a backlog, and the team moves on.

A stochastic bug bash ends differently, because the goal is different. The deliverables are not a ticket list; they are three artifacts:

• An expanded eval set. Every confirmed pattern from the bash gets promoted into the regression eval suite, with the failing input, the cohort it came from, and the failure description. Next model bump runs against this expanded set automatically. This is how the bash compounds in value over time instead of being a one-off ritual.
• A cohort-failure heatmap. A grid of cohorts × task types with cell-level pass rates from the bash. This tells you not just "where it fails" but "where it fails more than baseline" — which is the signal that drives roadmap, not the raw failure count.
• A list of unresolved frequencies. Tickets where the repro rate sat in the ambiguous middle (occasional, not reliable) get logged as "needs more sampling" rather than closed. They go into the next bash, or get auto-sampled in production traffic, until enough data accumulates to classify them.

If the bash ends without producing these three artifacts, it ended too early. If it produces them, the team has not just shipped a feature — it has built a piece of permanent infrastructure that catches the next regression before users see it.

The Underlying Realization

The architectural shift is that the bug bash, like every other QA ritual the industry inherited from the deterministic-software era, is a sampling instrument. For deterministic systems, the sample size of one was sufficient because every sample was the population. For stochastic systems, a sample of one is a single draw from a distribution and tells you almost nothing about the rest of the distribution.

The teams that win on AI quality are the ones that treat every QA artifact — bug bashes, dogfooding sessions, customer beta cohorts, even support tickets — as sampling instruments feeding a continuously updated map of where the system fails. The teams that lose are the ones still using a 2018 deterministic-QA playbook against a 2026 probability distribution and wondering why their post-launch incident graph looks like a heart monitor.

Run the next bug bash like a sampling pass. Bring the right cohorts. Demand repro counts. Promote patterns into evals. Close the loop. The format your QA team inherited was right for the system they used to ship. It is wrong for the system you are shipping now.