Automating Test Generation with LLMs at Scale

assembled · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Engineering productivity and test generation

Assembled wanted engineers to create more comprehensive unit, frontend, and integration tests without spending hours manually enumerating scenarios and writing boilerplate.

Constraints:

• generated tests must match the codebase's style and conventions
• engineers must review, run, and refine generated tests before production merge
• model quality matters more than latency for offline test generation
• examples are needed to steer output format and idioms
• poorly structured code can make useful test generation harder

Architecture

Assembled uses LLMs as an internal developer workflow for test creation. Engineers provide a structured prompt containing the function under test, relevant type definitions, and an example test suite from the codebase. A strong code model such as OpenAI o1-preview or Claude 3.5 Sonnet generates candidate tests, engineers review and refine the output for compilation, edge cases, and style, and only then copy the tests into the codebase.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  example-test-suite[Example test suite] --> engineer-prompt[Engineer prompt]
  engineer-prompt --> code-generation-llm[Code generation LLM]
  code-generation-llm --> generated-test-suite[Generated test suite]
  generated-test-suite --> engineer-review-gate[Engineer review gate]
  engineer-review-gate --> codebase[Codebase]

Components

Data flow

An engineer assembles a prompt with the exact function, relevant struct definitions, and example tests. The prompt is submitted to a high-quality code-generation model. The model returns a test suite, which the engineer checks for compilation, missed edge cases, and style before pasting the acceptable tests back into the repository.

Why this architecture

The workflow is deliberately manual and example-driven because tests must compile, assert the right behavior, and match local conventions. Assembled prioritizes the smartest available models because latency is not a major concern for this offline developer workflow, while human review keeps incorrect or poorly formatted tests from reaching production.

Models used: OpenAI o1-preview · Anthropic Claude 3.5 Sonnet · ChatGPT o1-preview

Scale

• The team reports saving hundreds of engineering hours.
  “Collectively, our engineers have saved hundreds of hours, reallocating that time to developing new features and refining existing ones.”article, stated
• A sample test suite was generated in 48 seconds.
  “in just 48 seconds, it generated a comprehensive test suite that was ready to use straight out of the box.”article, stated
• The approach was extended beyond Go unit tests to TypeScript, React component tests, and mocked-service integration tests.
  3 supporting quotes

  “Different programming languages. It’s relatively straightforward to adjust the prompt for other languages and tailor the results to specific testing frameworks.”article, stated

  “Frontend component testing. You can also extend this to test React components with user interactions and state changes”article, stated

  “Integration testing with mocked services. By changing the test case examples, you can test functions that interact with external APIs by mocking HTTP clients.”article, stated

Alternatives considered (roads not taken)

Use faster or cheaper models for test generation

Rejected because: The team says latency is not a major concern and recommends the smartest available models for better results.

Merge LLM-generated tests without human verification

Rejected because: LLMs can get test logic wrong, so Assembled requires engineers to read and run generated tests before production.

Use generic prompts without examples

Rejected because: Assembled found examples are the strongest way to drive the model toward local style and conventions.

Patterns

prompt-versioning (prompting)

The workflow depends on a reusable structured prompt recipe with fixed sections for code, types, examples, and output expectations.

human-in-the-loop-gate (quality)

Engineers must read, run, and refine generated tests before merging them.

model-routing (modeling)

Assembled selects high-capability code models for quality because the task is offline and latency-insensitive.

eval-driven-development (quality)

Generated tests are treated as candidates that must compile, cover edge cases, and fit codebase conventions before acceptance.

Tradeoffs

Choose strongest models for offline developer tooling

Chose o1-preview or Claude 3.5 Sonnet over lower-latency or cheaper models.

Rationale: Test generation is not interactive production serving, so model quality is prioritized over latency.

Use LLMs as a drafting accelerator rather than an autonomous merge system

Chose human review and iteration before copying tests into the codebase over automatically accepting generated tests.

Rationale: LLMs accelerate the tedious draft but engineers still own correctness, compilation, style, and production readiness.

Do not over-test simple code

Chose focus test-generation effort where tests provide meaningful confidence over using LLMs to generate tests for every simple glue or UI component.

Rationale: The team warns that not all code requires testing, so the automation should not turn into indiscriminate test expansion.

Gotchas (what bit them)

LLM-generated tests can be wrong or non-compiling.

Root cause: The model may miss edge cases, generate incorrect logic, or misunderstand formatting requirements.

Resolution: Engineers review, refine, read, and run the tests before production merge.

Weak examples reduce output quality.

Root cause: The model relies heavily on examples to learn desired test style, libraries, and conventions.

Resolution: Use comprehensive, idiomatic examples from the existing codebase in the prompt.

Difficult test generation can reveal code-design problems.

Root cause: Poorly structured or overly complex inputs/outputs make tests harder for the LLM to construct.

Resolution: Refactor the code or ask the LLM to break it up using the same prompting principles.

Evaluation

The article evaluates the workflow through concrete generated-test examples, engineer review requirements, and anecdotal productivity outcomes rather than a formal benchmark.

Metrics: 48 seconds to generate one comprehensive test suite · hundreds of engineering hours reportedly saved · generated tests checked for compilation issues, missed edge cases, and style · extension to Go, TypeScript, React component tests, and integration tests

Outcomes

• Engineers generated useful test suites much faster than manual authoring.
• The technique expanded from unit tests to frontend and mocked-service integration tests.
• Assembled reports hundreds of engineering hours saved.

The article gives one 48-second example and aggregate time-saved claims, but it does not disclose team size, number of generated tests, or acceptance rates.