The Ten-Thousand-File Codemod: Running an Agent Fleet Over a Mechanical Migration

Every framework migration has the same shape. You write a codemod, run it across the repository, and it cleanly converts 80% of the files — the ones that follow the patterns the codemod's author anticipated. Then you hit the long tail: the test file where someone monkey-patched the renderer, the component that reaches into framework internals, the module written in 2017 by an engineer who has since left, using an idiom nobody else ever adopted. The codemod parses these files fine. It just has no rule that applies. So the last 20% of the migration consumes 80% of the calendar, done by hand, file by file, by engineers who would rather be doing anything else.

Coding agents invert this economics. The hand-written weirdness that defeats a deterministic AST transform is exactly what a model handles well — it reads the file, understands intent, and rewrites it the way a human would, without needing an explicit rule for every variant. Airbnb proved the point at scale: nearly 3,500 Enzyme test files migrated to React Testing Library in six weeks, against an original estimate of 1.5 years of manual work. But here's what gets lost in the headline: the hard part wasn't the prompting. Once you point a fleet of agents at ten thousand files, the engineering problem stops being an AI problem and becomes a batch-operations problem — sharding, verification, quarantine, and merge strategy. The right mental model is a MapReduce job whose mapper is stochastic.

Codemods Win the Head, Agents Win the Tail

Deterministic codemods — jscodeshift, OpenRewrite, comby — remain the right tool for the head of the distribution. They're fast, free to run, and their output is provably consistent: the same input always produces the same output, so reviewing one transformation reviews them all. If a pattern appears five thousand times in identical form, you want a rule, not five thousand model invocations.

The failure mode is well documented: codemods stall on variation. Import aliases, unconventional coding styles, dynamic constructs the AST matcher can't see through — each edge case either gets skipped or requires extending the codemod, and extending a codemod for a pattern that appears four times in the repo is a losing trade. This is why organizations like Google built human-in-the-loop tooling (Rosie) to shepherd large-scale changes: the automation generates the commits, but the tail always leaked back to humans.

Agents change the cost curve on that tail. Google's twelve-month study of LLM-assisted migrations found that across 39 distinct migrations and 93,574 edits, roughly 69% of edits were generated by the model, and developers estimated a 50% reduction in total migration time. Airbnb's result is more dramatic because they leaned harder on automation: 75% of files migrated in the first four hours, 97% after four days of iterative tuning, and only about 100 files left for manual work.

The practical takeaway is a hybrid pipeline, not a religious choice:

• Run the deterministic codemod first. It handles the head cheaply and gives you a clean baseline diff.
• Send codemod failures and skips to agents. Per-file, with rich context: the file itself, its imports, a few already-migrated sibling files as exemplars.
• Reserve humans for what survives both. If neither the rule nor the model can convert a file after several attempts, it's genuinely weird and deserves human eyes.

The Mapper Is Stochastic — Design for Retries, Not Correctness

A codemod either works or it doesn't; running it twice tells you nothing new. An agent is a different kind of worker. The same file with the same prompt can succeed on attempt three after failing attempts one and two. This single property reshapes the whole pipeline design.

Airbnb's most counterintuitive lesson was that brute-force retries beat prompt sophistication. Instead of polishing a master prompt, they built a per-file state machine — Enzyme refactor, Jest fixes, lint, TypeScript validation, done — where any failed check loops the file back to the model with the error output injected into the prompt. Most files of low-to-medium complexity converted within ten attempts. What moved the needle for hard files wasn't cleverer instructions but more context: prompts grew to 40,000–100,000 tokens, pulling in up to 50 related files, sibling tests, and team-specific patterns.

If you've operated MapReduce or any large batch system, this should feel familiar. You don't debug an individual flaky mapper; you design the job so that retries are cheap and safe:

• Idempotence per file. Every attempt starts from the original source, not from a previous attempt's partial output. A half-migrated file is corrupted state, not progress.
• Failure output as input. The typecheck error from attempt N is the most valuable token in attempt N+1's prompt. This is the agent-fleet equivalent of speculative execution — cheap, mechanical, and it recovers a huge fraction of failures.
• Tune on samples, then sweep. Airbnb's loop for the long tail: bucket the remaining failures by pattern, pick 5–10 representative files, adjust the pipeline until those pass, then re-run against everything left. Each sweep is cheap because the fleet is parallel; each tuning round is informed because the sample is small enough to actually read.

The mental shift is from "make the agent correct" to "make incorrectness cheap." You will never get a model to one-shot ten thousand files. You don't need to.

Verification Gates Are the Review — Because Nobody Reads Ten Thousand Diffs

Here is the uncomfortable arithmetic: at two minutes per diff — an absurdly optimistic pace for a meaningful review — ten thousand diffs is over 300 hours of senior-engineer attention. Nobody does this. What actually happens on under-engineered migration projects is worse than skipping review: reviewers rubber-stamp the first two hundred diffs, and approval becomes theater that launders unverified changes into the codebase.

The honest move is to admit that per-diff human review doesn't scale, and to replace it with a machine gate that every file must pass:

• It compiles and typechecks. Table stakes, and the cheapest signal you have.
• The tests pass — and still test something. For a test migration, assertion count and code coverage before and after are your proxy for preserved intent. An agent that "fixes" a failing test by deleting its assertions passes the naive gate; coverage-delta catches it.
• The diff stays in bounds. A semantic diff check, or even a crude one — the change touches only the expected APIs, doesn't add dependencies, doesn't modify files outside its shard. Agents occasionally take creative detours; the gate is where creativity goes to die, deliberately.

The gate has to be adversarial to the worker, not cooperative. If the agent can see the gate's implementation, the gate is part of the prompt, and models are alarmingly good at satisfying the letter of a check while violating its purpose. Keep verification logic out of the agent's context; feed it only the failure output.

Human review still exists, but it moves up a level: humans review the pipeline — the codemod rules, the prompts, the gate definitions, and a random sample of passed files from each pattern bucket. You review the factory, not each unit of output.

Sharding, Quarantine, and Landing the Plane

Three operational problems remain once conversion works, and each one can sink the project independently.

Sharding. Parallel agents must not touch overlapping files, or you spend your gains resolving merge conflicts between robots. File-level sharding is the natural default for a migration — each file is an independent work item — but the shard boundary should also respect ownership boundaries (codeowners, directory trees), because that determines who gets paged when a landed change misbehaves and keeps each landed batch reviewable by a single team. Give each worker an isolated worktree; shared checkouts are how you get two agents editing the same lockfile.

Quarantine. Some files will fail attempt after attempt. Set a retry budget — Airbnb's data suggests ten attempts covers the routine cases — and after that, quarantine the file: pull it out of the job, log its failure pattern, and move on. The quarantine list is not a failure of the project; it is the project's output for the human phase. A hundred quarantined files with categorized failure reasons is a two-week task. Ten thousand un-triaged files was the eighteen-month task you started with. The worst thing you can do is let the fleet grind on quarantine-worthy files, burning tokens and wall-clock on the 3% while the 97% waits to land.

Landing. A migration that converts everything but can't merge is a very expensive branch. Thousands of commits contending with normal feature traffic will starve in the merge queue or, worse, force a repo freeze that turns the whole org against the project. The playbook borrowed from Bors-style merge queues and Google's TAP: batch landed changes by shard, let the queue test batches optimistically in parallel, and on failure bisect the batch rather than ejecting everything. Because every change already passed the verification gate, queue failures should be rare and almost always mean environmental flake or a genuine cross-file interaction — both of which you want surfaced anyway. Land continuously from day one; a long-lived migration branch is a merge conflict with the entire company.

The Dashboard Is the Deliverable

You can tell whether a team understands fleet-scale migration by asking what they look at while it runs. If the answer is "the agent transcripts," they're operating a demo. The operating view of a ten-thousand-file codemod looks like a batch job dashboard, because that's what it is:

• Funnel by stage: how many files are converted by the deterministic codemod, passed by the agent on attempt one, passed after retries, quarantined. The shape of this funnel tells you where to invest — a fat retry band means your prompts need better context; a fat quarantine band means you have an unmodeled pattern bucket.
• Retry histogram: if median attempts creep from two to six between sweeps, something regressed — a prompt change, a model version, a gate tightening. This is your error-budget signal.
• Cost per landed file: tokens plus compute divided by files that actually merged. Airbnb's total — six weeks of engineering plus API costs — was a rounding error against 1.5 years of manual effort, but only because they watched the tail and stopped paying for hopeless retries.
• Quarantine aging: the human backlog, bucketed by failure pattern, oldest first.

None of this is exotic. It's the same operational discipline data engineering settled on twenty years ago, applied to a worker that happens to be a language model. The teams that struggle with agent fleets are usually strong at prompting and weak at batch operations; the teams that succeed treat the model as the least interesting component of the system — a mapper with a known failure rate, wrapped in retries, gated by verification, sharded for isolation, and landed through a queue.

The framework migrations, language upgrades, and API deprecations sitting in your backlog rated "too expensive to ever do" were priced under the old economics, where the long tail was hand labor. That price is stale. Repricing it isn't a prompting exercise — it's a distributed-systems exercise, and your team already knows how to do those.