Large-Scale Test Framework Migration Using LLMs

airbnb · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale code migration and test modernization

Airbnb needed to migrate nearly 3.5K React component test files from Enzyme to React Testing Library without losing original test intent or code coverage. The migration could not be a simple find-and-replace because the frameworks encourage different testing styles, and the original manual estimate was 1.5 years of engineering time.

Constraints:

• nearly 3.5K React component test files had to move from Enzyme to RTL
• Enzyme and RTL differ enough that a simple swap was not feasible
• deleting Enzyme files would create significant code-coverage gaps
• each file had to pass validation stages such as Jest, lint, and TypeScript checks
• complex files required large amounts of project-local context
• automation needed progress tracking and targeted reruns for long-tail failures
• the final semantic goal was preserving test intent and coverage, not just passing syntax checks

Architecture

Airbnb built a state-machine migration pipeline where each test file moved through Enzyme-to-RTL refactor, Jest fixes, lint/TypeScript fixes, and completion states. Failed validation triggered LLM repair attempts with updated error context and the latest file version. Prompts grew to 40K-100K tokens with component source, target tests, sibling tests, imports, RTL examples, and guidelines. The team used status comments, step-filtered reruns, and a sample-tune-sweep loop to push success from 75% to 97%, then manually finished the remaining 3% using LLM-generated baselines.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  corpus["Target Enzyme test corpus"] --> state["Migration state machine"]
  state --> validate["Validation runner"]
  validate --> loop["LLM repair loop"]
  context["Context assembler"] --> loop
  selector["Related-file selector"] --> context
  loop --> output["Migrated RTL test file"]
  output --> validate
  state --> status["Migration status tracker"]
  status --> rerun["Targeted rerun tool"]
  rerun --> sweep["Sample tune sweep loop"]
  sweep --> context
  sweep --> manual["Human migration finish"]
  manual --> output

Components

Data flow

Each Enzyme test file enters a per-file state machine. The pipeline validates a migration step; if validation fails, it builds a dynamic prompt with the current file, validation errors, component source, related files, examples, and guidelines, then asks the LLM to repair the file. Status comments record step progress and test counts. Engineers inspect aggregate failures, sample 5-10 representative files, update prompts or scripts, rerun the sample, and sweep the remaining failing files. Files that hit the automation ceiling are handed to engineers with generated baselines.

Why this architecture

Airbnb used a state machine because code migration could be decomposed into discrete, validated stages with clear pass/fail signals. Retry loops were useful because validation errors and the current file gave the LLM concrete feedback. Rich context was needed for complex test setups and team-specific conventions. The breadth-first sample-tune-sweep loop prevented the team from overfitting to one hard file while steadily reducing common failure modes across the remaining corpus.

Models used: frontier models · LLM-driven code migration

Scale

• The migration covered nearly 3.5K React component test files.
  “updating nearly 3.5K React component test files from Enzyme to use React Testing Library (RTL) instead.”article, stated
• The original manual estimate was 1.5 years of engineering time.
  “We’d originally estimated this would take 1.5 years of engineering time to do by hand”article, stated
• The migration finished in six weeks.
  “using a combination of frontier models and robust automation — we finished the entire migration in just 6 weeks.”article, stated
• Prompts expanded to 40,000-100,000 tokens and as many as 50 related files.
  “our prompts had expanded to anywhere between 40,000 to 100,000 tokens, pulling in as many as 50 related files”article, stated
• The first bulk run migrated 75% of target files in four hours.
  “we successfully migrated 75% of our target files in just four hours”article, stated
• After four days of sample-tune-sweep work, completion reached 97%.
  “After running this “sample, tune, sweep” loop for 4 days, we had pushed our completed files from 75% to 97% of the total files”article, stated
• The long tail included files retried 50-100 times.
  “we had retried many of these long tail files anywhere between 50 to 100 times”article, stated

Alternatives considered (roads not taken)

Manual migration

Rejected because: Airbnb estimated manual work would take 1.5 years, whereas the LLM-assisted pipeline finished in six weeks.

Delete legacy Enzyme files

Rejected because: Analysis showed deletion would create significant code-coverage gaps.

Simple framework swap or find-and-replace

Rejected because: The frameworks differ fundamentally, requiring preservation of test intent and coverage rather than syntax substitution.

Keep tuning automation after the 97% mark

Rejected because: The remaining files had hit a practical automation ceiling, so engineers manually completed them from generated baselines.

Patterns

tool-use-orchestration (agentic-code)

The system repeatedly invoked an LLM as a repair agent only when concrete validation steps failed, passing errors and current code as tool feedback.

durable-execution (workflow)

The migration was modeled as a resumable state machine with per-file status, step tracking, and targeted reruns.

context-engineering (proposed-context)

Airbnb found that selecting the right related files, examples, imports, and component context mattered more than prompt wording for hard migrations.

eval-driven-development (quality)

Automated Jest, lint, TypeScript, and migration-step validations drove each LLM retry and improvement loop.

human-in-the-loop-gate (workflow)

Humans sampled failure modes, tuned prompts/scripts, and manually completed the final 3% once automation reached diminishing returns.

observability-tracing (operations)

Generated migration-status comments exposed per-step pass/fail counts and helped engineers find common failure patterns.

Tradeoffs

Migration workflow structure

Chose state-machine pipeline with step validation over single-pass LLM conversion of all files.

Rationale: A step-based state machine allowed parallelism, progress tracking, targeted retries, and per-step failure-rate improvement.

Retry strategy

Chose brute-force retry loops with dynamic validation feedback over more elaborate prompt engineering alone.

Rationale: The team found repeated attempts with fresh error context were the most effective way to improve per-file outcomes.

Long-tail improvement strategy

Chose sample-tune-sweep across common failures over manual deep-dives into each failed file.

Rationale: Sampling representative failures helped improve prompts/scripts in ways that affected many remaining files at once.

Automation cutoff

Chose manual intervention for the final 3% over indefinite prompt/script tuning.

Rationale: After 50-100 retries on long-tail cases, manual completion from generated baselines was more efficient.

Gotchas (what bit them)

Passing automated checks is necessary but may not prove semantic test equivalence.

Root cause: The migration validates Jest, lint, TypeScript, and completion, but preserving test intent is subtler than syntax and pass/fail status.

Resolution: Keep human review for the long tail and explicitly target preservation of original intent and coverage as an outcome.

Prompt structure was less important than selecting the right context.

Root cause: Complex tests depended on team-specific patterns and related files that the model could not infer from the target file alone.

Resolution: Pull in component source, sibling tests, examples, imports, and relevant nearby files, with context up to 40K-100K tokens.

Long-tail files can consume huge numbers of retries without resolving.

Root cause: The hardest migration cases remained stuck even after many dynamic retry attempts.

Resolution: Stop after the automation ceiling and manually finish from LLM-generated baselines.

Optimizing one hard case at a time would stall progress.

Root cause: The remaining failures contained common patterns plus difficult outliers, so depth-first debugging could overfocus on worst cases.

Resolution: Use a breadth-first sample-tune-sweep loop to address common failures across all remaining files.

Evaluation

Evaluation relied on staged automated validation for each file, status comments with test counts and step state, bulk success-rate tracking, sample-based prompt/script validation, and final human completion to preserve intent and coverage.

Metrics: Jest tests passed/failed/skipped · lint status · TypeScript compilation status · per-step migration status · bulk file completion rate · manual remaining-file count · overall test intent and code coverage preservation

Outcomes

• Airbnb completed the migration in six weeks instead of the estimated 1.5 years
• the first bulk run migrated 75% of target files in four hours
• sample-tune-sweep work raised completion to 97% after four days
• the remaining 3% were finished manually from LLM-generated baselines
• Airbnb reports preserving original test intent and overall code coverage

The migration covered nearly 3.5K files, prompts of 40K-100K tokens, hundreds of concurrent files, and long-tail retries of 50-100 attempts.