AI Agent for Automated Quality Assurance Testing in Cryptocurrency Platform

coinbase · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Crypto platform QA

Coinbase wanted to scale product testing while reducing manual effort and flaky coded end-to-end tests.

Constraints:

• financial trust
• browser automation
• valid bug detection
• cost reduction

Architecture

Coinbase’s qa-ai-agent turns natural-language QA requests into browser-driven test runs. The service exposes gRPC and WebSocket entry points, uses the browser-use LLM browser agent to control remote BrowserStack sessions against Coinbase.com, persists executions, session history, and issues in MongoDB, and relies on LLM reasoning over visual/textual browser state to assert test outcomes. A second LLM-as-judge reviews screenshots and issue descriptions to filter false positives, while Slack and JIRA integrate results into developer workflows.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  natural-language-test-request[Natural language test request] --> qa-agent-service[QA agent service]
  qa-agent-service --> browser-use-agent[Browser-use agent]
  browser-use-agent --> browserstack-browser[BrowserStack browser]
  browser-use-agent --> llm-reasoning-layer[LLM reasoning layer]
  llm-reasoning-layer --> llm-judge[LLM judge]
  qa-agent-service --> mongodb-run-store[MongoDB run store]
  llm-reasoning-layer --> mongodb-run-store
  mongodb-run-store --> developer-workflow-integrations[Developer workflow integrations]

Components

Data flow

A tester describes a scenario in natural language. Coinbase starts a test run through service endpoints, launches a BrowserStack browser controlled by browser-use, and lets the agent navigate Coinbase.com using visual and textual page state instead of generated test code. The agent reasons about whether the feature works, records the execution and issue artifacts in MongoDB, sends accepted issues into Slack/JIRA, and has a judge LLM score whether reported bugs are genuine.

Why this architecture

Coinbase chose a no-code browser agent because traditional end-to-end tests are brittle under minor UI changes and expensive to maintain. Visual/textual browser control lets tests pass when the feature still works, while natural-language test authoring makes new coverage much faster to add. The judge layer compensates for the agent’s false-positive risk by scoring bug artifacts before issues are accepted.

Models used: LLM browser agent · LLM judge

Scale

• Coinbase started with an ambition to 10x testing effort at one-tenth the cost.
  “We started with an idea to 10x our testing effort at 1/10 the cost.”article, stated
• In comparative testing, qa-ai-agent scored 75% accuracy versus 80% for human testers and found 300% more bugs in the same timeframe.

  "Accuracy: 75% (AI) vs. 80% (Manual)

• Efficiency: qa-ai-agent detected 300% more bugs in the same timeframe"

— article, stated

• New tests can be added in 15 minutes to 1.5 hours and token costs showed an 86% reduction versus manual testing.
  “Cost Savings: Our token cost analysis shows an 86% reduction compared to traditional manual testing expenses”article, stated
• The agent now executes 40 scenarios, finds about 10 issues weekly, and is aimed at replacing at least 75% of manual testing.
  “Today qa-ai-agent executes 40 test scenarios, encompassing localization, UI/UX, compliance, and functional aspects of the Coinbase product experience.”article, stated

Alternatives considered (roads not taken)

Coded end-to-end integration tests

Rejected because: Minor layout changes made them flaky and expensive to debug, while natural-language tests are faster to create and maintain.

Text-to-code intermediary test generation

Rejected because: Coinbase has the agent act directly in the browser from visual and textual state instead of generating code first.

Accept all agent-reported bugs

Rejected because: A judge LLM filters potential false positives using screenshots and issue descriptions.

Patterns

tool-use-orchestration (testing)

The agent directly operates a browser from natural-language testing requests instead of generating conventional test code.

eval-driven-development (quality)

Coinbase treated agent QA performance evaluation as first-class and benchmarked against human testers.

Tradeoffs

How to create new test automation.

Chose natural-language agent tests over coded test automation and human training.

Rationale: Natural-language descriptions are faster to add and maintain but require rigorous agent evaluation.

Test authoring interface.

Chose natural-language browser-agent tests over coded test automation.

Rationale: Natural-language tests are faster to add and less brittle when UI layout changes do not affect functionality.

Bug acceptance path.

Chose LLM judge confidence filtering over trusting every issue found by the QA agent.

Rationale: The judge adds a quality gate for false positives before developer-facing issue flow.

Gotchas (what bit them)

Traditional integration tests were brittle under UI changes.

Root cause: Coded test selectors and layouts broke for reasons unrelated to product functionality.

Resolution: Use a visual browser agent whose test passes when the underlying feature works.

Evaluation

Coinbase compared productivity, correctness, scalability, cost, and issue validity against human testers.

Metrics: 75% AI accuracy · 80% human accuracy · 300% more bugs · 86% cost reduction

Outcomes

• The system reportedly runs 40 scenarios and finds about 10 issues weekly.

One of two Coinbase records in the batch; this one uses the Netherlands source URL.