Practical Implementation of LLMs for Automated Test Case Generation

cesar · Research & Academia · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM-assisted software test-case generation

Cesar wanted to determine whether GPT-3.5 Turbo and LangChain could generate practical test cases for a real production web platform, while keeping outputs structured enough for QA use and evaluating whether AI-generated cases matched human-written test documentation quality.

Constraints:

• The target application, Da.tes, is a real production web platform rather than a toy example.
• GPT-3.5 Turbo struggled when multiple software features were described in a single template.
• Integration tests require explicit cross-feature dependencies, which increases prompt size and can degrade quality.
• The workflow depends on humans providing well-formed feature descriptions.
• The team lacked access to deeper production metrics such as code coverage, execution history, and defect data.
• Generated tests needed to be evaluated by QA engineers against human-written test cases.
• The generated artifacts had to include structured fields such as title, preconditions, steps, expected results, test data, and classification.

Architecture

The implementation uses GPT-3.5 Turbo through the OpenAI API, orchestrated with LangChain, to run a three-stage prompt chain. A structured feature template is supplied feature-by-feature. The first prompt asks the model, in a QA Engineer role, to generate requirements; the second converts requirements into JSON test conditions grouped by functional or non-functional category; and the third combines the previous outputs into markdown test cases. QA engineers then blindly score AI-generated and human-written cases on correctness, consistency, and completeness and perform side-by-side A/B preferences.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Da_tes_application["Da.tes application"] --> Feature_input_template["Feature input template"]
  Feature_input_template --> LangChain_orchestration["LangChain orchestration"]
  LangChain_orchestration --> GPT_3_5_Turbo["GPT-3.5 Turbo"]
  LangChain_orchestration --> Requirements_prompt["Requirements prompt"]
  GPT_3_5_Turbo --> Requirements_prompt
  Requirements_prompt --> Test_conditions_prompt["Test-conditions prompt"]
  GPT_3_5_Turbo --> Test_conditions_prompt
  Test_conditions_prompt --> Test_cases_prompt["Test-cases prompt"]
  GPT_3_5_Turbo --> Test_cases_prompt
  Test_cases_prompt --> AI_generated_test_cases["AI-generated test cases"]
  AI_generated_test_cases --> QA_evaluation_forms["QA evaluation forms"]
  Human_written_test_cases["Human-written test cases"] --> QA_evaluation_forms
  QA_evaluation_forms --> Evaluation_analysis["Evaluation analysis"]

Components

Data flow

A human fills a structured template for one software feature of Da.tes. LangChain inserts that context into a QA-engineer role prompt and sends it to GPT-3.5 Turbo to produce requirements. The requirements and original template feed a second prompt that creates JSON test conditions. The template, requirements, and conditions feed a final prompt that emits markdown test cases with test steps, preconditions, expected results, data, and classification. Human QA volunteers then compare AI and human test cases through quality-factor scoring and A/B preference forms.

Why this architecture

The team decomposed test-case generation into intermediate prompt stages because direct generation from broad application context was too hard for GPT-3.5 Turbo. Role prompting and structured templates gave the model a QA-specific frame; JSON and markdown outputs made intermediate and final artifacts machine- and human-usable; feature-by-feature inputs reduced context overload; and blind QA scoring made the result empirical rather than anecdotal.

Models used: GPT-3.5 Turbo

Scale

• The study used GPT-3.5 Turbo as the underlying language model.
  “using OpenAI's GPT-3.5 Turbo model combined with the LangChain framework”summary, stated
• The prompt pipeline has three main stages.
  “The workflow consists of three main stages:”summary, stated
• The input template limited the main purpose description to a maximum of three lines.
  “Main purpose (brief description, maximum 3 lines)”summary, stated
• Ten QA engineer volunteers were expected to score the test cases.
  “10 QA engineer volunteers assessed test cases without knowing whether they were AI-generated or human-written”summary, stated
• Only seven of the ten expected responses were collected.
  “While only 7 of 10 expected responses were collected”summary, stated
• AI-generated cases averaged 4.31 versus 4.18 for human-written cases.
  “Overall average score: AI 4.31 vs Human 4.18”summary, stated
• A/B evaluators preferred AI-generated test cases in 58.6% of responses.
  “In the A/B comparison, 58.6% of responses (41 of 70) preferred AI-generated test cases.”summary, stated

Alternatives considered (roads not taken)

Generate test cases directly from a broad software description

Rejected because: The team instead decomposed the work into requirements, conditions, and cases so the model could reason through intermediate artifacts.

Describe all features in one template

Rejected because: GPT-3.5 Turbo produced worse results when asked to handle multiple features in one context, so the template had to be completed separately per feature.

Evaluate with code coverage, execution history, and defect data

Rejected because: Those metrics were unavailable, so the researchers focused on documentation quality factors from testing literature.

Fully automate without human input

Rejected because: The workflow still depends on human-authored feature descriptions, and output quality depends heavily on those templates.

Use one large template for integration tests

Rejected because: Cross-feature dependencies require bigger prompts, but larger templates correlated with degraded results.

Patterns

prompt-chaining (architecture)

The workflow chains requirements generation, JSON test-condition generation, and final markdown test-case generation.

structured-output-contract (quality)

The test-condition stage uses JSON and the final stage uses a structured markdown test-case format with required fields.

eval-driven-development (quality)

The study uses QA-engineer blind review and A/B comparison to evaluate generated artifacts against human-written test cases.

human-in-the-loop-gate (operations)

Humans provide feature templates and QA volunteers judge generated test cases, so the system augments QA rather than removing human oversight.

context-engineering (architecture)

The workflow controls model context with feature-level templates and staged intermediate outputs so GPT-3.5 Turbo sees a focused slice of the application.

prompt-versioning (operations)

The research depended on empirical prompt/template iteration to find a usable input structure and staged prompting method.

Tradeoffs

Break generation into three prompt stages

Chose requirements, conditions, then cases over one-shot test-case generation.

Rationale: Intermediate stages gave GPT-3.5 Turbo a structured reasoning path and reusable outputs.

Use feature-level templates

Chose separate template per feature over one template for the whole application.

Rationale: Feature isolation improved relevance and reduced context overload, but increased operational work.

Use QA documentation factors for evaluation

Chose correctness, completeness, and consistency over code-coverage or defect-history metrics.

Rationale: Those deeper operational metrics were unavailable for Da.tes, so documentation quality became the measurable proxy.

Use human blind evaluation

Chose QA engineer volunteer scoring and A/B comparison over automatic self-scoring alone.

Rationale: Human QA review made the comparison credible for documentation usefulness and preference.

Gotchas (what bit them)

Multiple features in one prompt degraded generation quality.

Root cause: GPT-3.5 Turbo struggled to handle several software features in one context window and task frame.

Resolution: Complete the structured template separately for each feature and process the application feature-by-feature.

Integration tests are hard to generate without bloating the prompt.

Root cause: Cross-feature dependencies require explicit descriptions, but larger templates degraded results.

Resolution: Treat integration coverage as a separate design problem and avoid hiding cross-feature dependency complexity inside one giant prompt.

The system was not fully automated.

Root cause: Correctly formatted feature descriptions still had to be supplied by people.

Resolution: Position the workflow as QA augmentation and template discipline, not autonomous test design.

Evaluation was limited by unavailable operational quality data.

Root cause: The researchers could not use code coverage, execution history, or defect data from the production platform.

Resolution: Use literature-backed documentation quality factors and blind QA review as a practical evaluation proxy.

Performance varied by feature type.

Root cause: Common features such as login and signup matched familiar model priors better than unusual domain-specific functionality.

Resolution: Expect context-dependent performance variation and evaluate across feature classes, not only common workflows.

Evaluation

The study used a blind QA evaluation with two forms. In the first, volunteers scored 10 AI-generated and 10 human-written test cases on a 1-5 scale for correctness, completeness, and consistency. In the second, similar AI and human test cases were placed side by side for A/B preference and justification. The study also reports limitations from the incomplete response count.

Metrics: Correctness score · Completeness score · Consistency score · Overall average quality score · A/B preference percentage · Number of QA volunteer responses collected

Outcomes

• AI-generated test cases achieved a 4.31 overall average score versus 4.18 for human-written cases.
• A/B comparison preferred AI-generated test cases in 58.6% of responses.
• AI-generated test cases were particularly strong on writing quality, simplicity, directiveness, clarity, and completeness.
• The study found feature-by-feature decomposition necessary for GPT-3.5 Turbo.
• Integration tests and unusual functionality remained practical limitations.

The study evaluated a real production web platform using a three-stage prompt workflow. Ten QA engineers were expected, seven responses were collected, evaluators scored 20 test cases in the first form, and A/B comparisons produced 70 preference responses.