How to Migrate Thousands of Legacy Test Files With LLMs, Not Rewrites

Somewhere in most mature codebases sits a migration nobody wants to own: thousands of files written against a framework, library, or pattern the team has since moved past, sitting there because the manual rewrite cost is prohibitive and the risk of breaking coverage during a bulk edit is real. Here’s a concrete engineering pattern for turning that category of problem from “a year of dedicated headcount” into “a few weeks of pipeline work” — using LLMs not as autocomplete, but as one step in a validated, retryable state machine.

Why this class of migration resists find-and-replace

The concrete case: thousands of test files written against one testing library needed to move to a different one that tests components from a fundamentally different angle — inspecting internal component state versus testing what a user actually sees and interacts with. That difference means tests can’t be translated line-by-line. Each file needs real structural understanding: what is this test actually verifying, and how do I express that same intent in the new framework’s vocabulary?

Two risks make this worse than a normal refactor:

• Coverage loss. Deleting the old tests without a faithful migration silently removes regression protection, especially on older components nobody actively works in anymore.
• Prohibitive manual cost. Early estimates for hand-migrating a codebase this size ran well past a year of engineering time — too expensive to justify against other roadmap work, which is exactly why these migrations tend to get postponed indefinitely.

The pipeline: treat each file as a state machine, not a batch job

1. Break the migration into discrete, validated steps

   Instead of “migrate the file” as one big fuzzy task, split it into narrow stages: refactor framework-specific API calls, fix assertion syntax, resolve lint/type errors, run final validation. Each file only advances to the next stage if the current one actually passes — real validation, not an LLM’s self-assessment of its own output.

2. On failure, retry with the actual error, not a better initial prompt

   When a step fails, feed the model the current file state plus the specific validation error from that attempt, and let it retry. This beat investing more effort in a single “perfect” upfront prompt — most files needed just one or two retries once the model had a concrete, specific reason its last attempt didn’t pass.

3. Escalate to rich context only for files that need it

   Simple retries handle typo-level and structural-pattern-level fixes. Files with deep indirection, custom test utilities, or non-standard setups need real project context to get right — so for those, expand the prompt to include the component’s own source, sibling test files from the same directory, curated high-quality migration examples, and relevant imports. More tokens only helped when they carried information the model couldn’t otherwise infer.

4. Instrument the long tail as its own problem

   A first bulk pass can clear the majority of files quickly, leaving a stubborn remainder that’s too inconsistent for generic fixes. Stamp every file with a machine-readable status of exactly where it failed, build a CLI to rerun just one failing step against just a matching subset of files, then run a tight loop: sample a handful of similar failures, tune the fix, sweep it across that category, move to the next failure pattern.

Why retries beat prompt perfectionism

There’s a real temptation, when reaching for an LLM in an automation pipeline, to spend disproportionate effort crafting the ideal upfront prompt — more examples, more instructions, more edge-case handling written into the prompt text itself. This pattern argues for the opposite default: build the retry loop first, and let the model see its own concrete failures.

The reasoning holds up mechanically. A single upfront prompt has to anticipate every failure mode across thousands of structurally different files, which is a losing bet — you will always miss cases. A retry loop instead gives the model a specific, narrow problem on each attempt (“this exact validation error, on this exact file”), which is a much easier target to hit than “write correct code on the first try, blind.” The feedback loop runs automatically and cheaply at scale, and most files simply didn’t need many attempts once given real feedback to work from.

Rich context is a scalpel, not a default

It’s tempting to conclude “just always give the model more context” — more source code, more examples, more surrounding files. That’s not what worked here. Rich context (in this case, prompts running well into the tens of thousands of tokens) was reserved for files that had already failed simpler attempts, because the insight that mattered was which context to include, not how much: the component’s own source, sibling tests that reflect the team’s actual conventions, and curated examples of already-correct migrations — not an undirected dump of the surrounding codebase.

Takeaway

The result here — a migration that would have cost a year-plus of manual engineering time finishing in six weeks with six engineers — didn’t come from a smarter model or a cleverer prompt. It came from pipeline design: breaking a fuzzy, judgment-heavy task into small validated steps, treating LLM failure as data to feed back rather than a dead end, escalating context deliberately instead of by default, and instrumenting the stubborn remainder as its own separate problem instead of re-running the whole thing and hoping. None of that is specific to test migrations — it’s the shape of the answer for any large, repetitive, structurally-similar code transformation sitting on a team’s backlog because “someone should really automate that.”