Debate Diversity Collapse: When Three Agents Vote 3-0 Because They Read the Same Internet

The architecture diagram says "ensemble of three frontier models, debate-and-reconcile, majority vote." The trace says all three agents converged on the same answer in round one and spent two more rounds politely paraphrasing each other. The eval says +0.4 points over a single call. The bill says 4.2x. Somewhere in there, somebody decided the panel was working.

Multi-agent debate is sold as a way to get disagreement-driven reasoning: three minds arguing toward a better answer than any one of them would reach alone. It depends on the agents actually disagreeing. Frontier LLMs trained on overlapping web corpora, instruction-tuned against overlapping preference datasets, and aligned against overlapping safety taxonomies share priors more than the architecture diagrams admit. After a round of "let's reconcile," what you observe is not three perspectives converging on truth — it is three samples from one distribution converging on the mode they were never that far from.

The pattern has a name in the recent literature: when an ensemble's vote-disagreement rate trends to zero independent of question difficulty, you have debate diversity collapse. The panel is still voting. The vote no longer carries information.

The illusion of independent verification

The mental model behind multi-agent debate borrows from human institutions: juries, peer review, scientific replication. These work because the humans involved are independent enough that their errors are uncorrelated. Replication catches mistakes precisely when the second team has a different methodology, a different lab culture, a different prior on what is plausible.

LLM ensembles fail this independence test in ways that are easy to miss because the model names are different. Three checkpoints from the same lab share an instruction-tuning recipe, a refusal training set, a system-prompt convention, and large overlapping fractions of pretraining tokens. Even cross-lab ensembles often share the same publicly scraped Common Crawl snapshots, the same Reddit and Wikipedia priors, and the same RLHF judgment patterns trained against contractor pools whose demographics overlap. When such an ensemble is asked a question whose answer is a well-known fact, of course they agree — and the agreement is informative. When asked a question whose answer is contested or out-of-distribution, they often still agree, because their distributions were never that far apart, and the agreement is no longer informative.

Recent measurement work makes this concrete. In ensembles of homogeneous agents, accuracy improves at small agent counts and then collapses to diminishing returns: marginal gains drop toward zero as the additional agents produce trajectories that are increasingly redundant copies of each other. With only two genuinely diverse agents, performance can match or exceed that of sixteen homogeneous agents. The math says the second through sixteenth homogeneous agents are paying full token cost for near-zero information.

There is a subtler failure on top of this: sycophancy. LLMs tuned to be agreeable read peer outputs and update toward them, even when their own first-round answer was correct. In published debate transcripts, conformity (changing your answer to match a peer) consistently exceeds obstinacy (sticking to your prior). The system was supposed to use disagreement to find errors. Instead, the loudest or earliest answer wins because the other agents capitulate.

Why majority vote keeps looking like it works

The reason debate diversity collapse is hard to see in dashboards is that the topline metric usually does improve over a single zero-temperature call. The improvement is real. The mechanism is not what the diagram claims.

Most of the empirical gain attributed to multi-agent debate is actually attributable to ensembling — the same mechanism that makes self-consistency work with a single model sampled multiple times at high temperature. Variance reduction across independent samples raises accuracy on questions where the model is right on most draws. You do not need three different models to get this. You need three different samples.

Once you decompose the gain this way, the cost structure looks ugly. A self-consistency baseline at temperature 0.7 with five samples from one model is often within noise of a three-agent debate using three frontier models, at a fraction of the cost and a fraction of the latency. The debate buys you one thing self-consistency cannot: the chance of catching errors that the single model makes on most draws. That chance is exactly proportional to how different the other agents' priors actually are. If the diversity is fake, the debate is fake, and you are paying a premium for what self-consistency would have given you for free.

This is the cost frame the team building the system rarely puts in writing: the second and third agents are paying tokens to agree. If they are agreeing because the answer is obvious, you didn't need them. If they are agreeing because they share priors with the first, you have an expensive variance-reduction trick dressed up as deliberation.

Measuring whether the panel is actually deliberating

You cannot fix what you cannot measure, and "did the panel deliberate" is not a default field in any tracing tool. A few measurements separate real debate from theater:

• Vote-disagreement rate over time. Track per-question, per-round disagreement at the level of intermediate plans, not just final answers. A healthy debate has high round-one disagreement on hard questions and convergence by the final round. A collapsed debate has zero disagreement from round one onward, regardless of difficulty.
• Structural diversity on intermediate reasoning. Cosine distance over reasoning-trace embeddings catches the case where the agents reach the same answer via the same argument. If two agents' chains-of-thought cluster geometrically, the agents are not contributing independent information even when their final answers happen to differ.
• Holdout adversarial cases where the right answer is unpopular. Build a small eval slice of questions where the obvious answer is wrong and the correct answer requires resisting consensus. A panel that gets these wrong by majority vote is a panel that cannot disagree productively, full stop. This is the single highest-signal eval most teams running multi-agent systems do not have.
• Identity Bias Coefficient. Measure how often each agent ends up matching the first agent's stated answer versus its own first-round answer. Conformity rates above a threshold are a flag that the debate protocol is amplifying sycophancy rather than dampening it.

The reason these metrics are not standard yet is that they require instrumenting the intermediate state, not just the final vote. The same trace data you already collect for cost attribution holds the answer; nobody is looking at it through the disagreement lens.

Architectural patterns that preserve disagreement

If you decide a real debate is what you want, the architectural choices that produce one are different from the choices most teams default to:

Cross-vendor model mix. Use checkpoints from labs with different pretraining mixes, different RLHF recipes, and different refusal training. The diversity you actually buy is in the priors, and the priors track the training data and tuning more than the parameter count. Three checkpoints from the same lab, even at different sizes, are closer to each other than any of them is to a competitor's flagship.

Role-prompted critics with adversarial system prompts. Instead of asking three identical agents the same question, fix one as the proposer and one or two as critics whose system prompts explicitly require finding flaws, demanding evidence, or arguing the opposite case. The role asymmetry is doing the work that the model diversity often fails to do. The critic prompt has to be aggressive enough to override the model's instinct to agree — "find the strongest argument against this answer" beats "review the answer for correctness."

Temperature-staggered sampling. When you must use the same model family, vary decoding parameters across the panel. One agent at low temperature gives you the modal answer. Two agents at higher temperature give you the tails of the distribution. The structural argument for this is exactly the self-consistency argument, repackaged inside a debate: variance buys you coverage of alternative completions the modal sample suppressed.

Anonymized debate transcripts. Strip identity labels from the messages each agent sees. Recent work shows that identity markers themselves create sycophancy channels — agents capitulate to peers in part because the peer's identity activates "respect this voice" patterns from training. Anonymization is cheap, requires no retraining, and measurably reduces conformity without changing the substance of the exchange.

Principal-curated panels. Choose agents for known capability differences on the task at hand, not because three is a round number. A code-debate panel with one model strong at static analysis, one strong at runtime reasoning, and one strong at finding security antipatterns will outperform three generalists at three times the cost. Capability differentiation is the load-bearing property; "different model name" is not.

The unifying principle: diversity has to be designed in. Three frontier models from three different labs is not enough on its own. The debate protocol has to actively reward disagreement and resist the pull toward consensus, or the system will collapse to "majority vote on near-identical samples" no matter how many agents you stack.

The eval discipline that distinguishes obvious from incapable

The hardest case to call from outside is the panel that agrees because the answer is, in fact, obvious. Most production traffic looks like this — the agents agree because there is one right answer, and any reasonable system would converge. You cannot distinguish a healthy panel from a collapsed panel on this traffic. They both vote 3-0. They both look great in the success metrics.

The only way to tell them apart is the adversarial holdout. You build a deliberately small eval — fifty to two hundred cases is usually enough — where the right answer requires the panel to not converge on the first plausible-sounding response. These are cases where one agent should hold out and force the protocol to take the minority view seriously. If the panel passes these, you have evidence the protocol can sustain productive disagreement when it matters. If it fails, you know what you have: a variance-reduction trick that works on easy questions and gives you confident wrong answers on hard ones, which is strictly worse than a single well-calibrated call.

This eval slice is also where you watch for regressions when you change the protocol. A common mistake is to introduce a "reconciliation" prompt that explicitly asks agents to find common ground. On easy questions the topline metric improves. On the adversarial slice it tanks, because the reconciliation prompt is the formal, written-down version of "please collapse." Without the holdout, the regression is invisible.

What the panel is actually for

The honest read is that multi-agent debate, as commonly deployed, is most useful as a forcing function for the eval discipline it requires, not as a runtime ensemble that beats single-model systems. Teams that take debate seriously end up building the diversity measurements, the adversarial holdouts, the anonymization, the role-prompted critics — and most of those investments pay off whether or not the runtime debate stays in the architecture. The instrumentation is more durable than the architecture choice.

The runtime question — debate or self-consistency or single call — should be settled by cost-per-validated-outcome on a representative eval, not by the architecture diagram's promise of deliberation. If your debate beats self-consistency by enough to justify the token premium on the questions that matter, keep it. If it doesn't, the panel was theater, and the budget is better spent on a stronger primary model or on the eval set you've been meaning to build for two quarters.

The cleanest mental model: a multi-agent panel is a public claim that disagreement is being usefully harnessed. Like any public claim, it is worth what it is auditable to. Until the team can show the vote-disagreement rate over time, the structural diversity on intermediate reasoning, and the adversarial holdout score, the claim is unverified. And unverified claims, in production, age into the kind of failure mode where the panel agrees right up until the day it confidently agrees on the wrong thing.