Acceptance Sampling for Agent Output: What Manufacturing QA Knows That Code Review Doesn't

Your agent fleet opened forty pull requests this week. You reviewed the six that touched payment code, skimmed a few that happened to land while you had the tab open, and merged the rest on green CI. If someone asked you what your review policy is, you would describe something like this — and it is not a policy. It is a mood.

The numbers say most teams are in the same place. A recent large-scale study of agent-authored pull requests in popular open-source repositories found that 61% receive no recorded review activity at all, and of the ones that do get reviewed, most are reviewed only by other agents. Meanwhile the volume keeps climbing: agents now produce PRs, documents, support responses, and tickets at a rate no human review process was designed for. Reviewing everything is impossible. Reviewing nothing is malpractice. So teams improvise somewhere in the middle, with no stated rule, no measured coverage, and no way to say whether the current level of scrutiny is paranoid or reckless.

Manufacturing solved this exact problem a century ago. When Western Electric was churning out telephone equipment in the 1920s, inspecting every unit was economically impossible and shipping uninspected lots was unacceptable — so Bell Labs statisticians built acceptance sampling: a mathematically grounded discipline for deciding how much of a batch to inspect, when to reject the whole batch, and when a supplier has earned lighter scrutiny. It became MIL-STD-105 during World War II, then ANSI/ASQ Z1.4 and ISO 2859-1, and it still governs how a container of goods gets accepted at a port today. The mapping onto agent fleets is almost embarrassingly direct — and almost nobody in AI engineering has picked it up.

"Review what looks important" is not a policy

Start with why the improvised middle ground fails. The typical unstated rule is salience-based: review the changes that look risky, that touch files you care about, or that happen to cross your screen. This feels reasonable and fails in three specific ways.

First, salience is correlated with the wrong thing. You review what is visible — big diffs, familiar subsystems, changes that arrive during working hours. Escaped defects concentrate in what is invisible: the small config change, the generated migration, the document nobody owns. Research on agent-authored PRs shows that review effort concentrates on structural footprint (files touched, lines changed), which is predictable before any human looks at the change — but structural footprint is a proxy for review cost, not defect risk.

Second, salience-based review has no measurable coverage. If your defect rate doubles next month, nothing in your process will notice, because you never knew what fraction of output you were inspecting or what defect rate you were observing in the sample. You cannot detect drift in a quantity you do not measure.

Third — and this is the one that matters organizationally — an unstated policy cannot be defended or budgeted. When leadership asks "are we reviewing enough of the agents' work?", the honest answer under vibes-based review is "we don't know, and we have no definition of enough." Compare that to: "we inspect at AQL 1.0 with normal inspection, our observed defect rate in samples is 0.4%, and here's the escape probability that implies." One of these survives an audit.

"Review 10% at random" — the policy teams sometimes graduate to — is genuinely better, because it is at least a stated rule with measurable coverage. But it is a blunt one: it inspects the same fraction of everything regardless of risk, batch size, or track record, and it has no answer for what happens when the sample turns up defects. Acceptance sampling is what "10% at random" grows up into.

The machinery: lots, AQL, and the OC curve

Acceptance sampling rests on a few pieces of machinery, all of which transfer.

A lot is a batch of homogeneous output judged together — in manufacturing, a production run; in an agent fleet, something like "this week's dependency-bump PRs from the upgrade workflow" or "today's 500 support-ticket summaries." Lot definition matters: mixing outputs from different workflows into one lot destroys the statistics, because defect rates are a property of a process, and different workflows are different processes.

The AQL — acceptance quality limit — is the worst defect rate you are willing to routinely tolerate from a process that is considered in control. It is a negotiated business number, not a statistical one. Cosmetic issues in generated documentation might carry an AQL of 4%; incorrect refund decisions might carry 0.1%. Declaring the AQL forces the conversation most teams never have: what defect rate are we actually willing to live with, per output type?

Given a lot size and an AQL, the standard tables (Z1.4's sample size code letters) hand you a sample size and an acceptance number: inspect n random units from the lot; if you find more than c defects, reject the whole lot. The crucial property is that sample size grows much more slowly than lot size — a lot of 500 might need a sample of 50, a lot of 3,200 needs 125, not 320. Statistical confidence is a function of absolute sample size far more than of sampling fraction, which is exactly why flat "review X%" rules over-inspect small batches and under-inspect large ones.

Behind every plan sits the operating characteristic curve, which plots the probability of accepting a lot against its true defect rate. The OC curve makes the two failure modes explicit: producer's risk (α) — the probability of rejecting a lot that was actually fine, i.e. wasted escalation — and consumer's risk (β) — the probability of accepting a lot that was actually bad, i.e. escaped defects. Every review policy that has ever existed has an OC curve; acceptance sampling is merely the discipline of knowing what yours is. "We review whatever looks important" has an OC curve too — you just can't draw it, which means you can't defend it.

And rejecting the lot means something stronger than fixing the defects you found. In manufacturing, a rejected lot goes back for 100% screening. For an agent fleet: if 3 of the 50 sampled PRs from the upgrade workflow are defective, the response is not "fix those three" — it is "a human reviews all 40 remaining PRs in that batch, and the workflow's recent merged output gets audited." The sample is not there to catch defects; it is there to judge the process that produced the lot.

Switching rules: earned trust, mechanically defined

Loading…