Automated Unit Test Improvement Using LLMs for Android Applications

meta · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Android test improvement

Meta needed to improve unit-test coverage for large Android Kotlin applications by generating additional tests that cover neglected edge cases while filtering out unreliable or non-buildable LLM output.

Constraints:

• generated tests must build
• generated tests must be reliable and resistant to flakiness
• recommendations must add novel coverage rather than duplicate existing tests
• the tool had to work in industrial Meta codebases such as Instagram and Facebook
• engineers needed recommendations suitable for production acceptance

Architecture

Meta built TestGen-LLM around an Assured Offline LLM-Based Software Engineering workflow. The tool generates additional Kotlin unit tests for existing Android test suites, then subjects candidate tests to a filtration process that checks buildability, reliability, flakiness resistance, and novel coverage contribution before recommending them to engineers. The system was evaluated and deployed in test-a-thons for Instagram and Facebook.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  existing-test-suite[Existing test suite] --> testgen-llm-generator[TestGen LLM generator]
  testgen-llm-generator --> assured-filtration-pipeline[Assured filtration pipeline]
  assured-filtration-pipeline --> engineer-recommendation-surface[Engineer recommendation surface]

Components

Data flow

Existing Android unit tests and target classes feed TestGen-LLM. The LLM proposes additional edge-case tests, the assured offline pipeline filters candidates through buildability, reliability, flakiness, and coverage checks, and only candidates that strengthen the test suite become engineer-facing recommendations.

Why this architecture

The architecture treats generation as only the first step. The key design is the assured offline filter: generated tests are useful only if they compile, behave reliably, avoid flakiness, and add coverage. This allows LLM output to be used in production engineering workflows without asking engineers to manually triage every raw candidate.

Models used: large language model for test generation · ensemble learning mentioned as an innovative aspect

Scale

• The deployment targeted Instagram and Facebook.
  “The paper describes TestGen-LLM's deployment at Meta, highlighting its application in test-a-thons for platforms such as Instagram and Facebook.”article, stated
• The tool enhanced 10% of the classes it was applied to.
  “Quantitative results from the study indicate that TestGen-LLM was able to enhance 10% of the classes it was applied to, with most recommendations being positively accepted by Meta's software engineers for production.”article, stated

Alternatives considered (roads not taken)

Manual test coverage improvement

Rejected because: Manual coverage improvement requires significant developer effort across large codebases.

Recommend raw LLM-generated tests

Rejected because: Meta filtered generated tests for buildability, reliability, flakiness resistance, and novel coverage before recommendation.

Use generation only for evaluation

Rejected because: The system was designed for both evaluation and deployment.

Patterns

eval-driven-development (evaluation)

Generated tests are accepted only after objective checks for buildability, reliability, flakiness resistance, and coverage improvement.

guardrails-io-filtering (reliability)

The filtration layer discards candidate test cases that fail quality criteria before they reach engineers.

batch-offline-processing (operations)

The workflow is explicitly framed as assured offline LLM-based software engineering rather than an interactive assistant.

human-in-the-loop-gate (workflow)

Filtered test recommendations are accepted by Meta engineers before production use.

Tradeoffs

Use an assured offline workflow

Chose filtered, validated recommendations over online raw generation.

Rationale: Offline assurance gives the organization time to validate generated tests before exposing them as production recommendations.

Target improvement of existing tests

Chose additional test cases for existing unit tests over generating entire test suites from scratch.

Rationale: The source frames the tool around neglected edge cases in existing suites, which narrows validation and engineer review.

Gotchas (what bit them)

LLM-generated tests can fail build or reliability requirements.

Root cause: Raw generated tests may not satisfy industrial codebase constraints.

Resolution: Meta discards candidates that fail buildability, reliability, flakiness, or coverage checks.

Improving coverage is not enough if tests duplicate existing coverage.

Root cause: Generated tests need to contribute novel code coverage rather than simply adding more lines.

Resolution: The filtration process requires contribution to novel code coverage.

Evaluation

Meta evaluated TestGen-LLM through deployment in test-a-thons and quantitative measurement of classes enhanced and engineer acceptance.

Metrics: classes enhanced · coverage improvement · engineer recommendation acceptance · buildability · reliability · flakiness resistance

Outcomes

• TestGen-LLM enhanced 10% of the classes it was applied to.
• Most recommendations were accepted by Meta engineers for production.
• The system demonstrated utility on Instagram and Facebook test-a-thons.

The available article gives percentage and deployment context, but not absolute project or test counts.