The Hidden Ops Org Behind Your Evals: Annotator Economics

Your eval suite reports a number. Behind that number is a label. Behind that label is a person — usually one you have never met, working a two-week gig managed over WhatsApp, paid through a mobile money app, ranking model outputs they were given fifteen seconds to read. The eval score you ship to your VP, the regression gate that blocks your deploy, the leaderboard rank you put in the launch blog — all of it inherits the quality of that person's attention on that afternoon.

We talk about evals as if they were instruments: calibrated, repeatable, objective. They are not. An eval is a measurement device whose sensor is a human labor pipeline, and most teams budget for the harness while treating the people as a free input. That accounting error is why eval scores drift, why your "ground truth" disagrees with itself, and why the most expensive bottleneck in frontier AI is no longer compute.

The numbers make this concrete. Producing 600 high-quality RLHF annotations can run around $60,000 — on the order of 167 times the compute cost of the equivalent training step. The data annotation market is projected near $2.26 billion in 2025, growing roughly 32% a year, and a single vendor like Scale AI expected to roughly double sales toward $2 billion. None of that shows up in the architecture diagram. It shows up as a line item someone in procurement signs, disconnected from the engineers who depend on it.

The Eval Is Only As Good As the Labor That Built It

Every benchmark you trust started as a labeling project. GPQA, the graduate-level question set, reportedly cost over $120,000 to assemble. The HLE benchmark allocated around $500,000 to collect high-quality questions. These are not data-cleaning expenses; they are payroll. When a benchmark becomes the thing the whole field optimizes against, its validity rests entirely on whether the humans who built it understood the task, agreed with each other, and stayed careful through the thousandth example.

That last condition is the one nobody instruments. RLHF and preference work are not object tagging. Ranking two model responses for helpfulness, safety, or tone requires a judgment that is genuinely cognitive — and cognitively fatiguing. A standard simplified protocol yields roughly 250 samples per eight-hour shift. Do the arithmetic: that is under two minutes per comparison, sustained across a full day, often seven days a week on a short project. The label you treat as immovable ground truth was produced under exactly the conditions that degrade human judgment.

So when your model's score moves by two points between eval runs, the honest question is not "did the model change?" It is "did the labelers change, did their guidelines change, did their fatigue change, or did the vendor swap in a fresh cohort who interpret 'harmful' differently than the last one?" If you cannot answer that, your eval is not measuring the model. It is measuring an entangled system of model and workforce, and attributing all the variance to the model.

Quality Decays Under Throughput Pressure — Predictably

The dangerous part is that label quality does not fail randomly. It fails in the direction of whatever the incentives point at, and annotation incentives almost always point at throughput. Vendors are paid per task or per project. Annotators are paid per task or per hour with quotas. When you push for more labels faster — because your release is slipping and you need the eval refreshed — you are not buying more of the same quality. You are buying a different, lower quality, and the price tag looks identical.

There are well-understood mechanisms for catching this, and most teams either don't use them or don't watch them:

• Gold questions: pre-labeled items with verified answers, seeded invisibly into the work stream. An annotator who drifts from the gold answers gets flagged. Without a gold set, you have no continuous signal on whether quality is holding.
• Attention checks: deliberately perturbed examples that a careful worker should score one way and a coasting worker another. Repeated failures pull a worker off the task and trigger re-annotation of their prior work.
• Inter-annotator agreement: metrics like Cohen's Kappa and Krippendorff's Alpha that quantify how consistently independent annotators apply the same labels. Tracked over time, a falling agreement number is your early-warning system for calibration drift.

Here is the operational point most engineering teams miss: these are not one-time acceptance tests you run when the dataset is delivered. They are monitoring. Inter-annotator agreement is a production metric for your eval pipeline in exactly the way p99 latency is a production metric for your service. If you only measure agreement once, at the end, you learn that quality decayed after you have already trained on or shipped against the bad labels. You want the chart, updated continuously, with an alert when it drops.

The People Are Invisible, Which Is Exactly the Problem

The reason quality decay goes unmanaged is the same reason the cost goes unbudgeted: the workforce is deliberately abstracted away. You buy "annotations" from a vendor the way you buy compute from a cloud. But compute does not get tired, does not reinterpret ambiguous guidelines, and does not quit mid-project because the content is traumatizing.

The actual conditions are worth stating plainly, because they directly shape the data you build on. Reporting on the industry has documented annotators in Kenya earning as little as $2 an hour for work that U.S. tech firms pay outsourcing intermediaries up to $12.50 an hour to source — labeling content that ranges from pornography to graphic violence, more than eight hours a day. Onboarding can be a Google Form, coordination a WhatsApp group, payment a mobile-money transfer, with no formal contract and two-week project horizons. In 2025, Kenyan data workers formed a Data Labelers Association, drawing hundreds of members in its first week, precisely because the people underwriting these eval scores had no other channel to push back on conditions.

You do not have to frame this purely as an ethics question to take it seriously as an engineering one. A workforce that is underpaid, churned every two weeks, onboarded through a form, and exposed to disturbing material without support is a workforce that produces high-variance, drift-prone labels. The labor conditions are not adjacent to data quality. They are the data quality. A two-week churn cycle means you are constantly re-incurring the cost of teaching new people your guidelines, and constantly re-paying the calibration tax of a fresh cohort. Stability in the workforce shows up downstream as stability in your metrics.

"Just Use an LLM Judge" Moves the Labor, It Doesn't Delete It

The obvious 2026 reflex is to replace the humans with an LLM-as-judge. The economics are seductive: per-annotation cost under a third of a cent, roughly twenty times cheaper than a crowd platform, available instantly at any scale. For triage, for cheap continuous regression checks, for generating draft labels a human then reviews, this genuinely works and you should use it.

But it does not eliminate the human dependency — it relocates it, and adds new failure modes that are harder to see. Three matter most:

• Preference leakage. When the model generating your training data and the model judging your eval share lineage, the judge systematically favors outputs related to its own family. This contamination is harder to detect than the biases the field already names, and it quietly inflates exactly the scores you care about.
• Benchmark contamination. If the eval set leaked into pretraining, the judge and the candidate are both compromised, and the number is fiction. No amount of cheap judging fixes a contaminated benchmark.
• Anchoring to a human baseline you stopped maintaining. An LLM judge is only trustworthy where it has been validated against human labels. Reported agreement is wildly uneven — substantial on some datasets (Cohen's κ above 0.8), poor on others. The judge's reliability is itself a measurement that requires fresh human gold data to establish. So you still need the annotators; you just need fewer of them, doing higher-leverage calibration work instead of bulk labeling.

The right architecture is a hierarchy, not a replacement: LLM judges handle volume, humans own the gold set and the calibration audits that keep the judges honest. That makes your human spend smaller and far more valuable per dollar — but only if you actually fund and instrument that human layer instead of pretending the LLM made it disappear.

Budget the Org, Not Just the Harness

The practical takeaway is a reframe. Stop treating eval data as a commodity you procure and start treating it as a production system you operate, with the workforce as a first-class component. Concretely:

• Put inter-annotator agreement on a dashboard next to your model metrics, and alert on its decline. A score with no agreement chart behind it is a number without error bars.
• Track workforce stability as a quality leading indicator. Annotator tenure, cohort churn, and guideline-version changes predict your eval variance. If your vendor rotated the team, expect your numbers to move for reasons that have nothing to do with your model.
• Pay for calibration, not just throughput. The highest-leverage human work in an LLM-judge world is maintaining gold sets and auditing the judge — skilled, stable, well-compensated work, not the cheapest bulk labeling you can source.
• Make the cost visible to the engineers who depend on it. When the team that ships against an eval cannot see what the labels cost or who produced them, they will over-trust a number they would scrutinize if they understood its provenance.

The leaderboard tells a clean story: a model, a benchmark, a score. The real story has an ops org in it — annotators, guidelines, gold sets, agreement metrics, and a labor market with its own economics and ethics. You are already paying for that org whether you manage it or not. The only choice is whether you budget for it, instrument it, and treat the people in it as the load-bearing infrastructure they are — or keep shipping numbers you have quietly decided not to understand.