AI-Powered Hybrid Approach for Large-Scale Test Migration from Enzyme to React Testing Library

slack · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale frontend test migration

Slack needed to migrate thousands of Enzyme React tests to React Testing Library so the company could upgrade to React 18 without losing test coverage or spending more than 10,000 engineering hours on manual migration.

Constraints:

• the legacy codebase contained many years of JavaScript and TypeScript variations
• Enzyme and React Testing Library follow different testing philosophies
• existing deterministic codemods handled only part of the migration
• LLM API calls were expensive and token-limited at the time
• generated code had to pass linting, tests, and human review before merge
• the migration had to be distributed across more than 150 frontend developers without blocking feature work

Architecture

Slack built a hybrid AST-plus-LLM conversion pipeline for Enzyme-to-React-Testing-Library migration. The system collects the source file, rendered DOM trees for test cases, and a partial deterministic AST conversion with annotations. It packages that context for Claude 2.1, parses the response, runs linters and autofix, executes the converted tests, and performs one feedback iteration with runtime logs when the first attempt fails. Successful or partially successful outputs are bucketed by pass rate for developers to review and merge manually.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  migration-work-allocator["migration-work-allocator"] --> developer-review["developer-review"]
  source-test-file["source-test-file"] --> context-collector["context-collector"]
  source-test-file --> ast-codemod["ast-codemod"]
  ast-codemod --> context-collector
  dom-tree-adapter["dom-tree-adapter"] --> context-collector
  context-collector --> claude-2-1-api["claude-2-1-api"]
  claude-2-1-api --> response-parser["response-parser"]
  response-parser --> lint-and-autofix["lint-and-autofix"]
  lint-and-autofix --> test-runner["test-runner"]
  test-runner --> feedback-loop["feedback-loop"]
  feedback-loop --> claude-2-1-api
  test-runner --> success-bucketing["success-bucketing"]
  success-bucketing --> developer-review

Components

Data flow

A test file enters the migration pipeline. Context collection gathers the file code, DOM trees for each test case, and codemod output for patterns Slack can convert deterministically. The codemod performs 100%-accurate AST conversions where possible and annotates unknown conversion sites. The packaged context is sent to Claude 2.1; the response is parsed, linted, autofixed, and executed. If execution fails, logs are added to a second prompt for one feedback loop. The resulting file is bucketed by pass rate, then a developer reviews and signs off before any merge.

Why this architecture

Slack used a hybrid architecture because neither deterministic codemods nor the LLM alone captured enough context. AST rules were reliable for known transformations, DOM trees supplied user-visible context required by React Testing Library, annotations constrained hallucination, runtime execution verified generated code, and human review protected the main branch. The LLM became valuable only as one tool inside a larger preprocessing, validation, feedback, and review pipeline.

Models used: Anthropic Claude 2.1

Scale

• Slack started with 15,500 tests and estimated more than 10,000 engineering hours of work.
  “Initially we had 15,500 tests to convert where one test takes about 30 to 45 minutes, and overall, we estimated it would take more than 10,000 engineering hours.”article, stated
• The tests were owned across more than 50 teams and more than 150 frontend developers.
  “we were tasked to first lead the migration of tests owned by more than 50 teams with more than 150 frontend developers.”article, stated
• Manual migration reduced the remaining tests from 15,500 to 13,000 between August and October 2023.
  “At the end of October, we had 13,000 tests, and we converted 2,500 test cases.”article, stated
• The AST phase reduced remaining tests from 13,000 to 8,000 between November and January.
  “At the beginning of November, we had 13,000 tests, at the end of January we had 8,000 tests.”article, stated
• The hybrid AST/LLM phase ran from 8,000 tests in February to zero in May 2024.
  “Next quarter, February to May 2024, we still had 8,000 tests in February. In April, we had 1,000 tests. May, we had zero.”article, stated
• The pipeline achieved about 80% code conversion correctness on selected test files.
  “Eighty percent on average of selected test files is automatically converted correctly, and the rest, 20%, you have to perform yourself.”article, stated
• The measured impact covered 338 files and more than 2,000 test cases.
  “We were able to run 338 files with more than 2,000 test cases, and we looked at how many test files passed.”article, stated
• Slack measured 16% fully converted files, 24% partially converted files, and 22% test cases passing immediately.
  “In this case, it was 16% of fully converted files. That's great. This was a pretty decent result. Then, 24% were partially converted files”article, stated

  “Twenty-two percent of test cases passed. We calculated that 45 minutes per each test case was saved.”article, stated
• One file conversion took roughly two to four minutes and was run sequentially.
  “One conversion for one file maybe takes from two to four minutes, depending on the complexity.”article, stated
• Some legacy files produced 8,000-line LLM requests, causing rate-limit or response-quality issues.
  “some of the components, they yield a query with 8,000 lines of code, and we were just rate limited or did not get a proper response.”article, stated

Alternatives considered (roads not taken)

Manual migration across developers

Rejected because: It was necessary initially but too slow for the full 15,500-test workload and carried a more-than-10,000-hour estimate.

Pure AST codemod

Rejected because: JavaScript/TypeScript variation, 65 Enzyme methods, and RTL's different testing philosophy made rule coverage insufficient.

Pure LLM conversion from Enzyme code and prompt

Rejected because: Claude 2.1 alone had high variability and only 40-60% success, so Slack did not release that tool to developers.

Multiple feedback loops on failures

Rejected because: An extra loop improved conversion about 10-20%, but further steps were not cost-effective under API/resource constraints.

Parallelize conversions for speed

Rejected because: The team avoided overloading shared API endpoints and ran sequential conversions in CI, often at night.

Patterns

tool-use-orchestration (architecture)

Slack chained deterministic AST conversion, DOM rendering, LLM conversion, parsing, linting, test execution, feedback prompts, bucketing, and human review into one migration tool.

prompt-chaining (workflow)

The pipeline uses an initial conversion prompt and one failure-feedback prompt augmented with runtime logs when the generated tests fail.

structured-output-contract (control)

Slack annotated every conversion instance and gave the model constrained suggestions so it would edit only intended regions and avoid hallucinated transformations.

eval-driven-development (evaluation)

The team evaluated conversions against human-converted references, ran linters and tests, and counted only executed passing tests toward impact claims.

human-in-the-loop-gate (control)

Generated migration output was never merged without developer inspection and sign-off.

fallback-degradation (operations)

When token limits or oversized DOM context broke the first request, the retry dropped DOM-tree information to fit the model call.

cost-aware-routing (operations)

API costs and endpoint capacity constrained the system to one feedback step, sequential processing, and off-peak CI runs.

Tradeoffs

Use AI only where deterministic tooling was insufficient

Chose hybrid AST plus LLM conversion over LLM-first automation.

Rationale: The speaker argues that deterministic tools are preferable when available, and LLMs shine where high variability and dissimilar context exceed rule-based systems.

Limit failure feedback to one loop

Chose one log-augmented retry over longer iterative self-repair loops.

Rationale: Additional loops increased API cost and gave diminishing returns after the first retry.

Run sequentially and off-peak

Chose one-by-one CI conversions at night over parallelizing all file conversions.

Rationale: Slack prioritized endpoint stability and cost/resource limits over maximum conversion throughput.

Distribute migration by individual developer responsibility

Chose flat direct assignments to frontend developers over strict organizational hierarchy.

Rationale: The code ownership data was imperfect, and even workload distribution reduced bottlenecks while still keeping developers close to code they had touched.

Gotchas (what bit them)

Rule-based AST codemods hit a combinatorial wall.

Root cause: JavaScript and TypeScript allow many ways to express the same testing logic, and Enzyme had dozens of methods with many callback and abstraction variants.

Resolution: Slack kept deterministic AST conversions for patterns it could handle perfectly and used annotations plus LLM reasoning for the unresolved cases.

Pure LLM conversion lacked the same context human developers use.

Root cause: The model initially saw Enzyme code and a prompt, while humans rely on rendered DOM, React code, partial AST transformations, and frontend experience.

Resolution: The team modeled context collection after human migration behavior and supplied DOM trees, prompts, partial conversion output, and annotations.

Code-generation claims needed conservative measurement to overcome skepticism.

Root cause: A syntactically broken file may still contain mostly converted code, but counting it would overstate actual developer time saved.

Resolution: Slack counted only test cases whose generated code executed and passed, even when that understated broader partial conversion value.

Large legacy components exceeded practical token and API limits.

Root cause: Some files generated huge requests, including combinations of source code and per-test DOM trees.

Resolution: The second attempt dropped DOM context when the first call failed due to very long requests.

Evaluation

Slack evaluated conversion quality by bucketed test sets, comparison against human-converted references, linting, autofix, executing generated tests, tracking file/test pass rates, measuring developer time saved, and adoption of the codemod versus manual conversion.

Metrics: AST-only success rate · LLM-only success rate · automatically converted code share · fully converted file rate · partially converted file rate · executed test-case pass rate · developer minutes saved per passing test · tool adoption rate · feedback-loop improvement · runtime per file

Outcomes

• Slack completed the Enzyme-to-RTL migration by May 2024.
• The hybrid pipeline reached 80% average code conversion correctness on selected files.
• The measured AI codemod impact saved 22% of developer time using conservative executed-test accounting.
• Developers used the codemod in 64 cases and reviewed generated code before merge.
• Slack open-sourced the tool on npm after interest from other companies.

The source reports 15,500 initial tests, more than 10,000 estimated engineering hours, more than 150 developers across 50 teams, 338 files and more than 2,000 test cases in the measured run, 16% fully converted files, 24% partially converted files, 22% executed test cases passing, and two-to-four-minute per-file runtime.