AI-Powered Regression Testing with Natural Language Test Case Generation

duolingo · Education · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Mobile app quality assurance

Duolingo needed regression testing that could keep up with weekly releases, extensive A/B variants, and fast product iteration without consuming large amounts of QA team time or requiring brittle code-heavy automation.

Constraints:

• weekly releases introduced many regressions to monitor
• tests had to be maintainable by QA team members without deep coding knowledge
• A/B tests and feature variants made the next screen hard to predict
• automation had to tolerate frequent UI and flow changes
• LLM-driven runs still required review because bugs could be worked around rather than exposed

Architecture

Duolingo adopted MobileBoost's GPT Driver as a natural-language regression testing layer. QA authors describe target test goals in plain language, GPT Driver interprets each mobile screen on a virtual device, decides the next action needed to reach the goal, and records the live run for human review. The strongest design shift was moving from brittle step-by-step button instructions to goal-oriented prompts such as progressing until a success screen appears, which lets the LLM adapt to variant flows while keeping QA reviewers in the loop through stored recordings.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  natural-language-test-authoring["natural-language-test-authoring"] --> gpt-driver-screen-interpreter["gpt-driver-screen-interpreter"]
  gpt-driver-screen-interpreter --> virtual-device-runner["virtual-device-runner"]
  gpt-driver-screen-interpreter --> software-layer-checks["software-layer-checks"]
  virtual-device-runner --> run-recording-store["run-recording-store"]
  run-recording-store --> qa-review-workflow["qa-review-workflow"]

Components

Data flow

A QA team member writes a natural-language test case, runs it in GPT Driver, and watches execution on a virtual device. GPT Driver uses GPT-based interpretation of each screen to choose the next mobile action toward the stated success criteria. The run is recorded and stored, then QA reviews the recording to confirm that the workflow completed as expected and to spot bugs that did not strictly block progress.

Why this architecture

Duolingo chose goal-directed natural language execution because its product variants made fixed scripts brittle. Letting GPT reason from the end goal made tests more reliable across uncertain flows, while recordings and QA review compensated for the risk that the LLM could work around non-blocking bugs. MobileBoost's software-layer checks added a deterministic escape hatch for cases where GPT interpretation was unreliable.

Models used: GPT · GPT Driver

Scale

• The QA team monitored and tested weekly releases.
  “the QA team spends a substantial portion of its bandwidth monitoring and testing our weekly releases”article, stated
• Initial test authoring covered key scenarios within a few hours.
  “Within a few hours, we were able to generate tests for key scenarios like session progression, onboarding, and social features.”article, stated
• The prior manual regression workflow took several hours for multiple QA team members every week.
  “a process that previously took several hours for numerous QA Team members every week”article, stated
• Reviewing recordings reduced the workflow from hours to minutes.
  “our team can now quickly scrub through recordings to ensure workflows are completed as expected, a process of minutes.”article, stated
• The process reduced manual regression testing workflows by as much as 70%.
  “This process has reduced manual regression testing workflows by as much as 70%.”article, stated

Alternatives considered (roads not taken)

manual regression testing as the primary workflow

Rejected because: It consumed weekly QA bandwidth that Duolingo wanted to redirect toward bug fixes and new feature testing.

rigid step-by-step UI automation

Rejected because: Specific button-by-button scripts broke down as screens and variants changed.

enumerating every UI eventuality in prompts

Rejected because: The test cases ballooned after weeks of trying to handle every variant explicitly.

Patterns

tool-use-orchestration (agent-architecture)

GPT Driver turns a natural-language testing goal into UI actions on a virtual mobile device, effectively using the app interface as the tool surface.

human-in-the-loop-gate (reliability)

Human QA reviewers remain responsible for auditing recorded test runs because successful completion does not guarantee that no bug occurred.

fallback-degradation (reliability)

MobileBoost mitigates unreliable GPT interpretation by adding checks or bypassing the model where a deterministic next step is clear.

Tradeoffs

Use goal-oriented prompts instead of exact step scripts.

Chose broader success-criteria prompts over button-by-button UI instructions.

Rationale: Goal prompts were more robust to uncertain screen sequences but required recording review because the LLM might work around bugs.

Keep human review in the loop rather than fully trusting pass/fail automation.

Chose scrubbing stored recordings over treating automated completion as sufficient.

Rationale: The review step preserved visibility into non-blocking bugs while still reducing execution time from hours to minutes.

Gotchas (what bit them)

Specific step prompts made tests brittle across A/B variants.

Root cause: The app's weekly changes and variant flows made it hard to know which screen a test user would see next.

Resolution: Rewrite tests around broader end goals instead of enumerated button sequences.

LLM-driven tests can hide bugs by navigating around them.

Root cause: A goal-directed agent may complete the task even when intermediate defects occurred.

Resolution: Store recordings and make review of the run itself a core QA step.

GPT interpretation sometimes behaved unpredictably.

Root cause: Prompt directions can be misinterpreted, and the model can go rogue in ways that are hard to troubleshoot.

Resolution: Add software-layer checks or bypass GPT for clear next-step actions.

Some content types remained hard for GPT-based testing.

Root cause: Complex challenge types involved nuanced translations or timed interactions.

Resolution: Treat those cases as limitations while waiting for LLM speed and accuracy to improve.

Evaluation

Duolingo evaluated the system through operational regression-testing results: how quickly QA could create and review tests, whether runs completed expected workflows, and how much manual workflow time was reduced.

Metrics: tests for key scenarios generated within a few hours · recording review reduced a several-hour workflow to minutes · manual regression testing workflows reduced by up to 70%

Outcomes

• regression testing became accessible to QA team members regardless of coding depth
• manual workflow time dropped substantially through recorded-run review
• the QA team could focus more on bug fixes and new feature testing

The article reports weekly release testing, several-hour manual workflows for numerous QA team members, and up to 70% reduction in manual regression testing workflows.