AI Agents for Automated Product Quality Testing and Bug Detection

coinbase · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Product QA and bug detection

Coinbase needed to increase QA coverage for a high-trust crypto product while avoiding brittle manual and coded tests.

Constraints:

visual UI state
valid bug acceptance
cost
test creation speed

Architecture

Coinbase’s qa-ai-agent is a browser-driven QA service. Users describe tests in natural language; gRPC endpoints and WebSocket connections start runs; browser-use lets an LLM control a browser session, with BrowserStack-style device/browser execution and MongoDB persistence for test executions and session history. The agent reasons from visual/textual artifacts, identifies issues, and another LLM judges bug quality from screenshots and descriptions to filter false positives. Slack and JIRA close the loop into developer workflow.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  natural-language-test-request["natural-language-test-request"]
  qa-ai-agent-service["qa-ai-agent-service"]
  browser-use-agent["browser-use-agent"]
  browser-session["browser-session"]
  test-artifacts["test-artifacts"]
  llm-bug-judge["llm-bug-judge"]
  mongodb-store["mongodb-store"]
  developer-workflow["developer-workflow"]
  natural-language-test-request --> qa-ai-agent-service
  qa-ai-agent-service --> browser-use-agent
  browser-use-agent --> browser-session
  browser-session --> test-artifacts
  test-artifacts --> llm-bug-judge
  test-artifacts --> mongodb-store
  qa-ai-agent-service --> mongodb-store
  llm-bug-judge --> developer-workflow
  qa-ai-agent-service --> developer-workflow

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`natural-language-test-request`	Human-authored test description that defines the scenario without code.	`qa-ai-agent-service`
`qa-ai-agent-service`	Orchestrates browser-driven test runs via gRPC/WebSocket endpoints.	`natural-language-test-request`, `browser-use-agent`, `mongodb-store`, `developer-workflow`
`browser-use-agent`	Open-source LLM browser agent that controls the browser session and executes the test.	`qa-ai-agent-service`, `browser-session`
`browser-session`	Live product/browser environment where visual and textual test behavior is observed.	`browser-use-agent`, `test-artifacts`
`test-artifacts`	Screenshots, issue descriptions, histories, and other outputs used for judging bugs.	`browser-session`, `llm-bug-judge`, `mongodb-store`
`llm-bug-judge`	Evaluates whether an identified issue is genuine or a false positive and produces a confidence score.	`test-artifacts`, `developer-workflow`
`mongodb-store`	Storage backbone for test executions, session history, and artifacts.	`qa-ai-agent-service`, `test-artifacts`
`developer-workflow`	Slack/JIRA-integrated workflow where issues and test runs are surfaced to developers.	`qa-ai-agent-service`, `llm-bug-judge`

Data flow

A tester or developer supplies a natural-language test request. The service starts a run through gRPC or WebSocket, browser-use drives the browser session, and execution artifacts such as screenshots and issue descriptions are persisted. The QA agent identifies candidate bugs; a separate LLM-as-judge scores whether each issue is genuine or a false positive; accepted issues flow into developer workflow through Slack/JIRA integrations.

4 supporting quotes

“It processes testing requests in natural language; for example, a prompt such as "log into coinbase test account in Brazil, and buy 10 BRL worth of BTC" is sufficient to initiate”article, stated

“At its core, we use an open-source LLM browser agent, browser-use, to enable AI to control a browser session.”article, stated

“Our service includes both a set of gRPC endpoints and WebSocket-based connections to start a test run.”article, stated

“Based on the artifacts (screenshot, issue description, etc.), we ask another LLM to evaluate if the issue is genuine or potentially a false positive.”article, stated

Why this architecture

Coinbase chose no-code browser automation because existing end-to-end tests were brittle: minor layout changes caused failures and hours of debugging. Natural-language tests plus browser-use aim to preserve functional intent rather than DOM-level implementation detail. The extra judge layer exists because AI-generated bug reports still need correctness filtering before developer consumption.

3 supporting quotes

“these tests are prone to flakiness; minor layout adjustments can cause failures that require hours of debugging. The qa-ai-agent eliminates this flakiness thanks to its no-code nature”article, stated

“creating new test automation simply involves describing it in natural language, which is significantly faster and easier to maintain than code.”article, stated

“An innovation we introduced to improve correctness was leveraging LLMs as a judge to evaluate the quality of identified bugs.”article, stated

Models used: LLM browser agent · LLM-as-a-Judge

Scale

The strategic target was 10x testing effort at one-tenth the cost.
“We started with an idea to 10x our testing effort at 1/10 the cost.”article, stated
The system currently executes 40 scenarios across localization, UI/UX, compliance, and functional aspects.
“Today qa-ai-agent executes 40 test scenarios, encompassing localization, UI/UX, compliance, and functional aspects of the Coinbase product experience.”article, stated
In reported comparisons, AI accuracy was 75% versus 80% for manual testing, while detecting 300% more bugs in the same timeframe.

"Accuracy: 75% (AI) vs. 80% (Manual)

Efficiency: qa-ai-agent detected 300% more bugs in the same timeframe"

— article, stated

New tests can be integrated in 15 minutes if the prompt is already tested, or about 1.5 hours when prompt testing is needed.
“Scalability: New tests can be integrated within 15 minutes if the prompt has already been tested, or approximately 1.5 hour if prompt testing is needed”article, stated
The token cost analysis reports an 86% reduction compared with traditional manual testing expenses.
“Cost Savings: Our token cost analysis shows an 86% reduction compared to traditional manual testing expenses”article, stated

Alternatives considered (roads not taken)

Traditional code-based end-to-end test automation

Rejected because: It was flaky under layout changes and required hours of debugging, while no-code natural-language testing preserved functional intent.

“traditional end-to-end integration tests. However, these tests are prone to flakiness; minor layout adjustments can cause failures that require hours of debugging.”article, stated

Manual QA as the primary scaling path

Rejected because: Manual testing remained more accurate in some areas, but the AI agent found more bugs in the same timeframe and was cheaper.

"Accuracy: 75% (AI) vs. 80% (Manual)

Efficiency: qa-ai-agent detected 300% more bugs in the same timeframe
Scalability: New tests can be integrated within 15 minutes"

— article, stated

Patterns

`tool-use-orchestration` (testing)

The system lets an LLM control browser sessions directly for product test execution.

“At its core, we use an open-source LLM browser agent, browser-use, to enable AI to control a browser session.”article, stated

`eval-driven-development` (quality)

Coinbase benchmarked QA-agent performance against human QA testers on productivity, correctness, and scalability.

“To evaluate its performance, we benchmarked qa-ai-agent against human testers using a defined set of key success metrics.”article, stated

Tradeoffs

Manual QA versus AI QA coverage.

Chose AI agent for broad repetitive test execution over human testers for all scenarios.

Rationale: The agent had slightly lower accuracy but found more bugs at lower cost in the same timeframe.

“* Accuracy: 75% (AI) vs. 80% (Manual)”article, implied

How to handle lower AI accuracy than human QA.

Chose Use AI QA for scale and speed while keeping humans for flows automation struggles with. over Expecting the AI agent to fully replace every human tester immediately..

Rationale: The results show AI slightly behind manual accuracy but ahead on bug volume, speed, and cost; Coinbase explicitly notes humans retain an advantage in onboarding/liveness flows.

"Accuracy: 75% (AI) vs. 80% (Manual)

Efficiency: qa-ai-agent detected 300% more bugs in the same timeframe"

“human testers retain an advantage in areas challenging for test automation systems, such as the user onboarding flow, which requires a real human ID and liveness tests”article, stated

Gotchas (what bit them)

Preexisting end-to-end tests were flaky and expensive to maintain.

Root cause: Minor layout changes could fail tests even when the feature worked.

Resolution: Use natural-language, visual browser-agent tests and validate accepted issues.

“Additionally, existing end-to-end integration tests suffered from flakiness—minor layout adjustments would cause failures requiring hours of debugging.”summary, stated

Evaluation

Coinbase compared agent and human QA over issue productivity, correctness, scalability, and cost.

Metrics: 300% more bugs · 75% accuracy · 86% cost reduction · 15-minute test creation

“80% (Manual) * Efficiency: qa-ai-agent detected 300% more bugs in the same timeframe”article, stated

Outcomes

Coinbase targets replacing 75% of manual testing with AI-driven automation.

Duplicate Coinbase case family; this Italian source includes more navigation noise from Jina fallback.

“We anticipate that at least 75% of current manual testing will eventually be replaced by AI agents, a goal we are rapidly approaching.”article, stated