Dogfooding Your Agent Is Not QA

Your agent's internal metrics look great. Task completion is at 94 percent. The #agent-feedback Slack channel has been quiet for three weeks. Leadership is ready to turn it loose on customers. Then external users arrive, and within a month the numbers collapse: escalations spike, trust craters, and the postmortem question everyone asks is "how did dogfooding miss this?"

Dogfooding didn't miss it. Dogfooding hid it. Internal users are not a miniature version of your customer base — they are a population of expert operators who silently repair the agent's mistakes, learn which prompts to avoid, and share workarounds in DMs instead of filing bugs. Every repair makes the dashboard look better and the product worse-understood. The clean signal you're reading isn't quality. It's compensation.

The employee-repair effect

Sociologists who study automation in the workplace have a name for what your coworkers are doing: repair labor. Studies of automated workplaces describe "patchwork" — the invisible human effort that fills the gap between what an automated system promises and what it delivers. It comes in flavors: compensating for the system when it breaks, watching it in case it breaks, and quietly adapting inputs so it breaks less often.

Every one of those behaviors shows up the moment you dogfood an agent, and every one of them corrupts your quality signal:

• Silent correction. The agent drafts a customer reply with a wrong refund amount. Your support lead fixes the number before sending, because fixing it takes eight seconds and filing a bug takes eight minutes. The task logs as completed. No trace of the failure survives.
• Prompt avoidance. After the agent mangles two multi-account queries, your internal users simply stop asking it multi-account questions. The failure rate on that category drops to zero — because the attempt rate dropped to zero. Your traffic distribution has quietly deformed around the product's weaknesses.
• Folk knowledge. "Oh, you have to say 'as of today' or it pulls stale data" travels through Slack, lunch conversations, and onboarding buddies. New employees inherit the workarounds without ever experiencing the failures that produced them. The knowledge never reaches the eval suite.

None of this is laziness or sabotage. It's rational behavior from people whose job is to get their own work done, not to QA your product. Research on workplace AI transparency finds employees under-report AI involvement and AI problems for self-protective reasons — they don't want to look slow, incapable, or like they're complaining about the tool leadership just bet the roadmap on. The incentive gradient points away from honest failure reporting at every step.

External users invert every assumption. They don't know the magic phrasings. They ask the questions your colleagues learned to avoid. They don't fix the refund amount — they screenshot it, post it, and churn. Klarna's much-publicized AI support rollout is the canonical case: metrics that looked efficient internally translated into "lower quality," in the CEO's own words, once real customers with real problems hit the system, and the company ended up hiring humans back. The failure modes were there all along. The people who could tolerate them were not.

Why the dogfood signal reads clean

It's worth being precise about the mechanism, because "internal users are biased" undersells how structural this is.

First, your employees are the world's most forgiving user cohort. They understand the system's architecture, so they can predict where it will fail. They have organizational context, so they can spot a wrong answer instantly. And they have a social relationship with the team that built it, which converts every bug report into a small act of criticism. Each of those properties makes them better at using the agent and worse at measuring it.

Second, agents fail in a way that recruits the user into the cover-up. Traditional software fails loudly — stack traces, 500s, crashed tabs. Agents fail by producing plausible, confident, well-formatted output that is subtly wrong. Field analyses of production agent failures consistently find that the dangerous failures aren't crashes; they're fluent wrongness five steps downstream of the actual cause. An expert user catches the wrongness and repairs it in-flow, often without consciously registering it as a product failure. A non-expert either ships the wrongness or gets burned by it.

Third, "works for us" has a low bar built into it. MIT's large-scale evaluation of AI on real occupational tasks found models produce "minimally sufficient" work — a 7 out of 9, usable without edits — on roughly two-thirds of text tasks, while genuinely superior output remained a coin flip at best. Internal users, who are motivated to make the tool work and skilled at polishing its output, experience "minimally sufficient plus my quick fix" as success. Customers experience minimally sufficient as mediocre. Same output, opposite verdicts.

The result is a dashboard that measures the combined performance of the agent plus a shadow workforce of expert repairers — and attributes all of it to the agent.

Instrument for repair, not completion

The fix is not "dogfood harder." It's changing what you measure. Task completion tells you whether work got done; it cannot tell you who did it. The signal you need is the repair event: every moment a human intervened between the agent's output and the outcome.

Concretely, that means logging things most teams don't:

• Edit distance before use. When the agent drafts something — a reply, a query, a config change — diff what it produced against what the human actually sent or ran. GitHub's Copilot team learned this lesson publicly: raw acceptance rate rewarded a flood of trivial suggestions, so they moved to measuring how much accepted code survived — whether the characters were still there after the developer kept working. Retention after human contact is a repair-aware metric. Acceptance is not.
• Takeover and abandonment points. Where in a multi-step task did the human grab the wheel? A user who lets the agent do steps one through four and always does step five manually has filed a precise bug report through their behavior. Count it as one.
• Query-shape drift. Compare the distribution of internal requests against what naive users ask in early external tests. Categories that internal users have stopped exercising — or phrase in suspiciously uniform ways — are the categories where folk workarounds have replaced product quality.
• Retry-then-rephrase sequences. An internal user who asks, gets a bad answer, and immediately rephrases has experienced a failure that will never appear in your feedback channel. The transcript records it even when the human doesn't.
• Workaround archaeology. Periodically grep your own Slack for messages about the agent containing "you have to," "make sure you," "it doesn't work if," and "just do it manually." Every hit is an undocumented known-issue. This is crude and it works.

Notice what all of these have in common: they treat human behavior around the agent as the measurement, rather than trusting humans to convert their experience into reports. The KPMG and University of Melbourne finding that roughly two-thirds of workers accept AI output without verification cuts both ways — the ones who don't verify won't catch failures, and the ones who do verify will fix them silently. Neither group will tell you. Your telemetry has to.

"It works" versus "our people learned to work around it"

Once you're logging repair events, a handful of derived metrics separate genuine quality from learned compensation:

• Naive-user success rate, measured separately. Run a standing cohort of users with zero exposure to internal folklore — new hires in their first week, contractors, or a rotating "cold user" panel — and never mix their numbers with the veterans'. The gap between cold-user success and veteran success is your folklore debt, and it is the single best predictor of what external launch will feel like.
• Repair rate per completed task. Of tasks marked successful, what fraction involved a human edit, takeover, or rephrase? If completion is 94 percent but 40 percent of completions carry a repair event, your autonomous success rate is not 94.
• Time-to-workaround. When a new failure mode appears, how long until internal traffic routes around it instead of reporting it? Fast workaround propagation is a sign your feedback channel has lost to your Slack culture.
• Failure-report half-life. Track whether bug reports per active internal user decay over time while repair events per user stay flat. Reports declining while repairs persist is the signature of the employee-repair effect: people haven't stopped experiencing failures, they've stopped telling you about them. That decay curve looks identical to "the product got better" on a chart that only plots reports.

These metrics also change behavior upstream. Once "repair rate" is on the dashboard next to "completion rate," the eight-second silent fix stops being invisible, and the team building the agent starts seeing the workload their users have been absorbing — the same "new layers of human work" that every automation wave since ERP has generated and then failed to account for.

Dogfooding still matters — as a different instrument

None of this means you should skip dogfooding. It means you should stop asking it a question it cannot answer. Internal usage is excellent at what expert users are good at: catching architectural gaps, stress-testing integrations, surfacing the failures so severe that even a forgiving user can't repair around them. Rigorous eval practice makes the same point from the other direction — manual testing and intuition carry you surprisingly far in early development precisely because expert judgment is dense signal, and they break down exactly when the user population stops resembling the builders.

So treat your rollout as three separate instruments, each with its own question. Dogfooding answers "is this architecturally sound and survivable by experts?" Repair instrumentation answers "how much human labor is propping up the success metrics?" Cold-user cohorts answer "what will strangers actually experience?" Ship when all three agree — not when the first one goes quiet.

The uncomfortable discipline is refusing to celebrate a quiet feedback channel. Silence from internal users is the most ambiguous signal in your entire telemetry stack: it means either the agent stopped failing or the humans stopped telling you. The difference between those two states is the difference between a launch and an incident, and the only way to know which one you're looking at is to instrument the repairs your people are too busy, too kind, or too adapted to report.