Augmented Unit Test Generation Using LLMs

adyen · Finance · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Unit test generation

Adyen explored whether LLMs could help developers generate unit tests for large production codebases, where useful tests require the right surrounding code context rather than generic text generation.

Constraints:

• financial software requires high correctness and test coverage
• LLMs struggle with large, interrelated code frameworks
• unit tests depend on dependencies, interactions, mocks, and invisible abstract-method logic
• fine-tuning is resource-intensive and hard to keep current with daily code changes
• the tool is intended to augment developers, not replace human judgment

Architecture

Adyen’s approach frames LLM unit-test generation as a context-selection problem. A code-context retrieval layer identifies relevant snippets around the method under test, including dependencies and invisible logic behind abstractions, and feeds those snippets to an LLM so it can propose tests that complement existing code. The article contrasts this with ordinary RAG over text and with fine-tuning: RAG can retrieve relevant information, but code requires richer structural context; fine-tuning can help specialized generation but is expensive and hard to refresh in fast-moving codebases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  developer["developer"] --> unit-test-generation-workflow["unit-test-generation-workflow"]
  unit-test-generation-workflow["unit-test-generation-workflow"] --> codebase-context-retriever["codebase-context-retriever"]
  large-existing-codebase["large-existing-codebase"] --> codebase-context-retriever["codebase-context-retriever"]
  codebase-context-retriever["codebase-context-retriever"] --> test-generation-llm["test-generation-llm"]
  test-generation-llm["test-generation-llm"] --> unit-test-generation-workflow["unit-test-generation-workflow"]
  unit-test-generation-workflow["unit-test-generation-workflow"] --> test-suite["test-suite"]
  test-suite["test-suite"] --> developer["developer"]

Components

Data flow

A developer or tool selects code that needs unit tests. The context-retrieval step finds the relevant class/method sections, dependencies, interactions, and implementation details needed to understand the method’s role. Those snippets are provided to the LLM, which generates candidate unit tests that implement assertions around existing behavior. Developers review and integrate the generated tests into the normal code-quality workflow.

Why this architecture

The article argues that unit-test generation cannot be treated like ordinary text RAG because code behavior depends on relationships across methods, abstractions, and dependencies. The architecture therefore emphasizes finding the right code snippets before generation. Fine-tuning is acknowledged as useful for specialized behavior, but codebases change daily, making retrieval and per-task context selection more practical for developer assistance.

Models used: LLMs · GitHub Copilot · RAG · vector embeddings · fine-tuning

Scale

• The target environment includes large, pre-existing code frameworks.
  “integrating or modifying code within large, pre-existing code frameworks”article, stated
• Software codebases can update daily, which affects fine-tuning practicality.
  “rapid evolution common in software development environments, where codebases frequently update daily.”article, stated

Alternatives considered (roads not taken)

Conventional text RAG over a knowledge base

Rejected because: Text-style retrieval does not automatically capture code dependencies, abstractions, and interrelated behavior needed for unit tests.

Fine-tune a model for the codebase

Rejected because: Fine-tuning can help specialized domains but is resource-intensive and hard to keep current with daily code changes.

Ask the LLM to generate tests without targeted code snippets

Rejected because: The article says correct snippets are necessary for the LLM to understand invisible code logic and assist test generation.

Patterns

rag-retrieval (context)

The system retrieves code snippets relevant to the unit under test before asking the LLM to generate tests.

context-compaction (context-management)

Rather than sending an entire codebase, the workflow selects the snippets needed for the LLM to understand the code under test.

fine-tuning-adaptation (model-adaptation)

Fine-tuning is discussed as an option for specialized contexts, but not the default for fast-changing code.

human-in-the-loop-gate (developer-workflow)

The LLM assists engineers in generating tests; the framing is augmented developer assistance rather than autonomous test acceptance.

Tradeoffs

Use retrieval/context selection for code understanding

Chose task-specific code snippets over blind generation from the target method alone.

Rationale: Relevant snippets help the LLM reason about dependencies and hidden logic before generating tests.

Keep humans in the test-generation loop

Chose augmented unit test generation over fully automated test authoring and acceptance.

Rationale: The article frames LLMs as assistants that help engineers generate tests, preserving developer responsibility for correctness.

Gotchas (what bit them)

Code context is structurally different from prose context.

Root cause: A method’s test behavior depends on dependencies, abstract implementations, and interactions with the wider system.

Resolution: Retrieve and feed the relevant code segments that expose the method’s role and invisible logic.

A single LLM integration strategy does not fit every development task.

Root cause: Some tasks need up-to-date retrieval, while others need exact specialized generation.

Resolution: Select among RAG, fine-tuning, or specialized approaches based on the task’s requirements.

Evaluation

The article is exploratory and does not report a formal benchmark. Its evaluation lens is whether selected snippets let the LLM understand code enough to generate unit tests that complement and verify existing behavior.

Metrics: relevance of retrieved code snippets · developer usefulness · unit tests generated · coverage/complementarity to existing code · fit to fast-changing codebase

Outcomes

• Adyen identified code-context retrieval as central to LLM-assisted unit test generation
• the article clarifies when RAG or fine-tuning may be appropriate for LLM integration
• the approach positions LLM-generated tests as developer assistance rather than autonomous replacement

The source does not report deployment counts or measured productivity gains. It grounds scale in large pre-existing frameworks and codebases that update daily.