Automated Unit Test Generation Pipeline for iOS Using LLMs

duolingo · Education · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Mobile software engineering automation

Duolingo needed to expand iOS unit test coverage fast enough to keep pace with weekly releases, monthly code growth, and increasing LLM-generated production code.

Constraints:

• iOS releases go out weekly
• the codebase grows by tens of thousands of lines each month
• generated tests must move through PR review and CI rather than pile up
• Linux Temporal workers could not compile or lint iOS code locally
• humans still needed to approve generated PRs

Architecture

Duolingo built a set of Temporal workflows around Claude Code, GitHub, CI data, and a shared S3 state store. Developers can trigger tests from a PR label, while a backfill workflow repeatedly scores untested files from coverage data and launches parallel child workflows. Each child starts a remote Claude Code session to generate test files, mocks, and fakes. A lifecycle workflow manages generated-test PRs by assigning reviewers, enabling auto-merge, closing stale or conflicted PRs, retrying CI, and invoking an auto-fix workflow for failing builds.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  coverage-data-store[Coverage data store] --> backfill-workflow[Backfill workflow]
  s3-state-store[S3 state store] --> backfill-workflow
  backfill-workflow --> test-generation-child-workflow[Test generation child workflow]
  test-generation-child-workflow --> claude-code-agent[Claude Code agent]
  claude-code-agent --> github-api[GitHub API]
  test-generation-child-workflow --> s3-state-store
  s3-state-store --> pr-lifecycle-workflow[PR lifecycle workflow]
  github-api --> pr-lifecycle-workflow
  pr-lifecycle-workflow --> ci-auto-fix-workflow[CI auto-fix workflow]
  ci-auto-fix-workflow --> github-api

Components

Data flow

CI uploads coverage to S3. The backfill workflow scores Swift files, filters out active or cooled-down files, and starts child test-generation workflows. Claude Code reads rule files and source code, creates generated-test PRs, and writes PR state to S3. The lifecycle workflow reads that state and GitHub/CI status to assign reviewers, auto-heal failures, close stale PRs, or set cooldowns.

Why this architecture

Durable workflows matched the problem because test generation is not a single model call: file selection, PR creation, CI repair, reviewer assignment, retries, cooldowns, and stale-PR cleanup all need stateful coordination. S3 was chosen over DynamoDB for lightweight shared records. Human review stayed in the path because generated tests still needed owner context and approval.

Models used: Claude Code

Scale

• Duolingo ships iOS releases weekly.
  “iOS releases go out weekly, our codebase grows by tens of thousands of lines a month, and a meaningful portion of new code is now LLM-generated.”article, stated
• The iOS codebase grows by tens of thousands of lines monthly.
  “iOS releases go out weekly, our codebase grows by tens of thousands of lines a month, and a meaningful portion of new code is now LLM-generated.”article, stated
• In about 17 weeks, the pipeline merged 250 PRs and added about 85,000 lines of test code.
  “In ~17 weeks, the pipeline merged 250 PRs and added ~85,000 lines of test code, almost entirely autonomously. ## How we approached it”article, stated
• The pipeline added 4,460 test functions across 233 classes.
  “Over 17 weeks, we generated and merged 250 test PRs, adding ~85K lines of test code and 4,460 test functions across 233 unique classes.”article, stated
• The fully automated pipeline generates about 20 test PRs per day.
  “The pipeline went from zero to fully automated in under two months and now runs autonomously, generating ~20 test PRs per day.”article, stated
• The lifecycle workflow allows up to five CI-repair retries before closing a PR.
  “CI red + max retries (5) exceeded? Close the PR, label it as a failure, and set a 30-day cooldown. 7. **CI red?”article, stated

Alternatives considered (roads not taken)

Manual local Claude Code runs

Rejected because: Local validation was useful for tuning, but production scale required automated selection, PR generation, and lifecycle management.

DynamoDB for workflow state

Rejected because: The team chose a lightweight file-record model in S3 for PR and file state.

Open PRs without lifecycle automation

Rejected because: Unmanaged PRs would create stale work, merge conflicts, and reviewer noise.

Linux-only workers for final validation

Rejected because: Linux workers could not compile or lint iOS code, so Duolingo planned Mac workers for local validation.

Patterns

durable-execution (orchestration)

Temporal workflows coordinate long-running test generation, backfill scheduling, lifecycle management, CI repair, retries, and cooldowns.

tool-use-orchestration (agent-architecture)

Claude Code operates through GitHub, CI logs, rule files, repository cloning, and PR comments rather than isolated generation.

human-in-the-loop-gate (workflow)

Generated PRs require human engineering review before merging.

eval-driven-development (evaluation)

Early CI outcomes and failure categories drove prompt tuning, selection heuristics, and platform fixes.

observability-tracing (operations)

S3 records track PR history, retry counts, action history, and cooldown timestamps for each generated PR/source file.

Tradeoffs

Use autonomous backfill plus PR lifecycle automation

Chose scheduled generation and lifecycle workflows over manual one-off Claude Code runs.

Rationale: Manual experimentation informed the system, but ongoing coverage improvement required repeatable selection, generated PRs, and automatic queue management.

Use S3 as shared state

Chose JSON records in S3 over DynamoDB.

Rationale: The pipeline needed lightweight shared records more than database complexity.

Accept human review as the merge gate

Chose generated PRs reviewed by human engineers over fully automatic merge of model-written tests.

Rationale: Human review preserves ownership and context while the automation handles generation and lifecycle overhead.

Gotchas (what bit them)

Generated PRs create noise if their lifecycle is unmanaged.

Root cause: PRs can sit in queues, accumulate conflicts, or miss review assignment.

Resolution: Duolingo added an hourly lifecycle workflow that routes PRs by CI, staleness, review, and retry state.

Linux execution prevented local iOS validation before opening PRs.

Root cause: Temporal workers ran on Linux hosts while iOS lint/compile validation requires Mac capabilities.

Resolution: Duolingo planned Mac instances for SwiftLint and local validation.

Auto-fix agents can edit outside their intended boundary.

Root cause: A CI auto-fix agent modified production source files inside a test PR.

Resolution: The team is adding stricter file-change guardrails.

Review throughput became the next bottleneck after code generation.

Root cause: The pipeline can create PRs faster than reviewers can absorb them.

Resolution: Future work focuses on generated PR quality and a second reviewer agent before human review.

Evaluation

Duolingo evaluated the pipeline through early CI outcome batches, production PR merge metrics, coverage change, first-attempt CI pass rate, and failure categorization.

Metrics: merged PRs · lines of test code · test functions · classes tested · coverage lift · first-attempt CI pass rate · failure rate · generated PRs per day

Outcomes

• The pipeline merged 250 generated test PRs over roughly 17 weeks.
• It added about 85,000 lines of test code and 4,460 test functions.
• Core MVVM coverage increased from 9% to 30%.
• The pipeline reached about 20 autonomous test PRs per day.
• 76% of merged PRs passed CI on the first attempt.

The article reports a production system running every few hours and generating roughly 20 reviewed PRs per day.