Multi-Agent AI System for Automated Test Case Generation in Payment Systems

amazon_amet_payments · Finance · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Payment-system QA and test-case generation

Amazon AMET Payments needed to automate comprehensive test-case generation for payment features across diverse countries and regulatory environments without losing the specificity, risk awareness, and institutional knowledge of experienced QA engineers.

Constraints:

• The system supports payments across five countries and multiple payment methods, so generated tests must capture regional, regulatory, customer-segment, and transaction-flow variations.
• Single-agent prompting created generic and sometimes irrelevant test cases, making hallucination and lack of specificity production blockers.
• Outputs must integrate with existing test-management processes, so generated test cases need structured, validated, consistently formatted data.

Architecture

SAARAM is a multi-agent QA lifecycle app built with Amazon Bedrock, Claude Sonnet, and Strands Agents. The final workflow routes multiple input types through a file-type decision agent, parallel data-extraction subagents, a visualizer that creates multiple Mermaid diagrams, a data condenser that preserves acceptance criteria and edge cases while reducing context, and a test generator pipeline that produces, reformats, deduplicates, and organizes structured test suites. Pydantic schemas validate LLM outputs, while Strands workflow orchestration handles dependencies, parallel execution, retries, state persistence, and audit logging.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Inputs["Documentation and Design Inputs"] --> Router["File Type Decision Agent"]
  Router --> Extractor["Data Extractor Agent"]
  Router --> Visualizer["Visualizer Agent"]
  Extractor --> Visualizer
  Extractor --> Condenser["Data Condenser Agent"]
  Visualizer --> Condenser
  Condenser --> Generator["Test Generator Agent"]
  Bedrock["Amazon Bedrock Claude Sonnet"] --> Extractor
  Bedrock --> Visualizer
  Bedrock --> Condenser
  Bedrock --> Generator
  Workflow["Strands Agents Workflow"] --> Router
  Workflow --> Extractor
  Workflow --> Visualizer
  Workflow --> Condenser
  Workflow --> Generator
  Generator --> Validation["Structured Output Validation"]
  Validation --> Output["Test Management Output"]

Components

Data flow

Business requirements, documentation files, Figma designs, and code repositories enter a file-type decision agent that routes inputs to downstream agents. A data extractor uses six specialized subagents to pull domain-specific requirements, a visualizer creates six Mermaid views, and a data condenser merges text and diagrams into context-preserving addendums. The test generator then uses journey analysis, scenario identification, data-flow mapping, test-case generation, and test-suite organization subagents to produce deduplicated test suites. Claude Sonnet runs through Amazon Bedrock and Strands Agents, with Pydantic structured-output validation before results enter existing test-management workflows.

Why this architecture

Amazon moved from a single-agent prompt to a workflow of specialized agents because experienced testers do not read requirements holistically; they move through phases such as acceptance criteria extraction, journey analysis, scenario identification, data-flow mapping, and prioritization. Strands workflow orchestration gave the team explicit dependencies and parallelism, which was better suited to consistent test generation than dynamic supervisor routing. Structured Pydantic outputs kept LLM results usable in production test-management systems.

Models used: Claude Sonnet by Anthropic on Amazon Bedrock · Strands Agents SDK workflows

Scale

• Amazon AMET Payments serves approximately 10 million customers monthly across five countries.
  “we serve approximately 10 million customers monthly across five countries in the Middle East and North Africa region”article, stated
• The team publishes about five new payment features each month.
  “publishing on average five new features monthly. Each feature requires comprehensive test case generation”article, stated
• Before SAARAM, each feature consumed about one week of manual test-case generation and one full-time engineer annually.
  “traditionally consumed 1 week of manual effort per project. Our quality assurance (QA) engineers spent this time analyzing business requirement documents”article, stated

  “a process that required one full-time engineer annually merely for test case creation.”article, stated
• SAARAM reduced test-case generation from one week to hours, lowered QA validation effort from 1.0 FTE to 0.2 FTE, and found 40% more edge cases.
  “Resource optimization: QA effort decreased from 1.0 full-time employee (FTE) to 0.2 FTE for validation”article, stated

  “Coverage improvement: 40% more edge cases identified compared to manual process”article, stated

Alternatives considered (roads not taken)

Single AI agent fed with entire business requirement documents

Rejected because: It generated generic outputs, hit context-length limits, lacked specialized processing phases, and hallucinated irrelevant scenarios.

Custom-built looping, serial, parallel, orchestration, and workflow logic

Rejected because: Building the orchestration substrate manually demanded considerable effort, so the team migrated to Strands Agents SDK.

Supervisor-agent style dynamic routing for test generation

Rejected because: For a consistency-sensitive testing workflow, explicit workflow and graph patterns were more reproducible than dynamic routing.

Broad first-iteration agent set limited to Word documents and basic journey/segment parsing

Rejected because: The first design had context isolation, irrelevant data generation, narrow parsing, single-source input, and rigid architecture limitations.

"we identified five crucial limitations in our initial approach:

• Context and hallucination challenges"

Patterns

multi-agent-orchestration (architecture)

SAARAM decomposes QA analysis and test generation into specialized agents and subagents for routing, extraction, visualization, condensation, and test-suite generation.

context-engineering (architecture)

A dedicated Data Condenser agent preserves acceptance criteria, business rules, customer segments, and edge cases while reducing token load for downstream agents.

structured-output-validation (quality)

Pydantic schemas validate generated test-case objects before they propagate into test-management workflows.

durable-execution (operations)

The Strands workflow uses explicit dependencies, retries, state persistence, and audit logging for long-running test-generation workflows.

human-in-the-loop-gate (safety)

SAARAM reduces generation effort but still leaves QA engineers validating outputs, making the system an augmentation of expert testing rather than an unattended replacement.

eval-driven-development (quality)

The team built the workflow iteratively, tested each decomposed step with Amazon Q Developer CLI, and uses standards adherence and edge-case coverage as quality signals.

rag-retrieval (architecture)

A planned Bedrock Knowledge Bases integration would retrieve comparable historical test cases to ground future generation.

Tradeoffs

Mirror human QA cognition instead of asking a single model to infer the testing process.

Chose Specialized sequential agents based on expert tester phases over A single BRD-to-test-case agent.

Rationale: The human-centric decomposition produced more specific, risk-aware, actionable test cases than holistic document prompting.

Use Strands workflow orchestration rather than homegrown workflow logic.

Chose Strands Agents SDK workflow orchestration over Custom looping, serial, parallel, orchestration, and workflow graphs.

Rationale: The SDK supplied multi-agent coordination, clear execution paths, better performance, and less development effort.

Prefer workflow/graph patterns over supervisor-style dynamic routing for test generation.

Chose Explicit workflow dependencies and parallel execution over Supervisor agents making dynamic routing decisions.

Rationale: Test generation needs reproducibility and consistency more than maximum runtime flexibility.

Keep human validation while automating generation.

Chose Human-amplifying QA automation with 0.2 FTE validation effort over Replacing QA testers with unattended generation.

Rationale: The system codifies expert patterns and reduces repetitive work while preserving human accountability for high-stakes payment QA.

Gotchas (what bit them)

Single-agent document prompting produced generic, unusable test cases.

Root cause: The model compressed complex payment business logic without the phased reasoning process used by experienced testers.

Resolution: The team studied QA cognition and decomposed testing into specialized sequential thinking actions and agents.

Segregated first-iteration agents created hallucinations and irrelevant data.

Root cause: Agents independently collected data and built visual representations with limited context sharing.

Resolution: Iteration 2 introduced modular routing, context-aware extraction, visualization, and a condenser that preserves key context for downstream generation.

The first workflow could not ingest the full range of artifacts used in modern QA work.

Root cause: It was constrained to Word documents and narrow parsing for customer segments, journey mapping, and basic requirements.

Resolution: The second workflow accepts documentation files, Figma designs, code repositories, product requirements, UX specifications, acceptance criteria, and workstream extraction.

Unstructured LLM text could not be trusted as production test-management data.

Root cause: LLMs can produce human-like prose while failing to consistently follow required data formats.

Resolution: The system uses Pydantic models and validation errors to enforce typed, required fields and prevent malformed data propagation.

SAARAM still lacks examples of what high-quality test cases look like in practice.

Root cause: The current generation pipeline does not yet retrieve comparable historical test cases as examples before generating new ones.

Resolution: The team plans to add Bedrock Knowledge Bases backed by curated historical test cases.

Evaluation

Amazon reports before/after operational metrics for generation time, QA effort, edge-case coverage, and standards adherence, plus reliability controls such as Pydantic validation, retries, state persistence, and audit logging.

Metrics: Test case generation reduced from one week to hours. · QA effort reduced from 1.0 FTE to 0.2 FTE for validation. · 40% more edge cases identified compared to manual process. · 100% adherence to test case standards and formats.

Outcomes

• SAARAM reduced test-case generation from one week to hours.
• QA validation effort dropped from 1.0 FTE to 0.2 FTE.
• The team reports 40% more edge cases and 100% adherence to standards and formats.
• The system codifies institutional QA knowledge and reduces onboarding time for new QA team members.
• The architecture is positioned for expansion across AMET QA and later International Emerging Stores and Payments.

The system supports Amazon AMET Payments across five countries, about 10 million monthly customers, and an average of five new features per month.

• Historical test case learnings are automatically applied to new features"