AI Agents for Data Labeling and Infrastructure Maintenance at Scale

plaid · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Fintech data labeling and integration reliability

Plaid needed higher labeling capacity for transaction ML and faster repair of bank integrations across thousands of institutions.

Constraints:

• financial data quality
• manual labeling limits
• integration downtime
• human spot checks

Architecture

Plaid describes two internal operational AI agents. AI Annotator is an internally hosted LLM labeling platform that generates labels for anonymized transaction data, routes humans to golden-data/edge-case validation, and centralizes annotation/review for multiple teams. Fix My Connection monitors financial-institution integrations, detects quality issues before broad user disruption, uses agents to analyze breaking login changes and generate repair scripts, and runs across thousands of integrations in parallel.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  anonymized-transaction-data["anonymized-transaction-data"]
  ai-annotator["ai-annotator"]
  human-validation["human-validation"]
  centralized-annotation-infrastructure["centralized-annotation-infrastructure"]
  connection-quality-monitoring["connection-quality-monitoring"]
  fix-my-connection["fix-my-connection"]
  bank-integration-analysis["bank-integration-analysis"]
  repair-script-generator["repair-script-generator"]
  anonymized-transaction-data --> ai-annotator
  ai-annotator --> human-validation
  ai-annotator --> centralized-annotation-infrastructure
  human-validation --> centralized-annotation-infrastructure
  connection-quality-monitoring --> fix-my-connection
  fix-my-connection --> bank-integration-analysis
  bank-integration-analysis --> repair-script-generator
  repair-script-generator --> fix-my-connection

Components

Data flow

For labeling, anonymized transaction data enters AI Annotator, an LLM proposes likely labels, humans create benchmark golden data or spot-check edge cases, and centralized infrastructure lets internal teams review and scale the resulting datasets. For connectivity, monitoring detects integration degradation, Fix My Connection analyzes the affected bank integration, generates a repair script, and applies automated repairs so user-permissioned logins recover faster.

Why this architecture

Plaid applies LLMs where human-only operations did not scale: transaction labeling volume outpaced manual annotation, and maintaining thousands of bank integrations manually was costly and slow when login experiences changed. The design preserves human validation where benchmark quality or edge cases matter, while using agents for high-volume label generation and repetitive integration repair.

Models used: LLMs

Scale

• Plaid processes millions of transactions daily, creating a labeling scale problem.
  “Although we process millions of transactions daily, the scalability of manual labeling was a persistent challenge.”article, stated
• AI Annotator reports greater than 95% human alignment in early use.
  “our AI Annotator is able to produce high quality labels that have greater than 95% human alignment at a fraction of cost and time”article, stated
• Fix My Connection operates across thousands of small banks and credit unions/integrations.
  “Reliable access to thousands of smaller banks and credit unions is critical for our customers, they depend on seamless connectivity to reach every user”article, stated
• Automated repairs enabled more than two million successful user-permissioned logins and reduced average degradation repair time by 90%.
  “Our automated repairs have already enabled over 2 million successful user-permissioned logins, and reduced the average time to fix a degradation by 90%.”article, stated

Alternatives considered (roads not taken)

Bulk manual transaction labeling

Rejected because: Manual labeling could not keep pace with transaction volume, so humans were moved to benchmark and edge-case validation.

Manual repair flows for bank integrations

Rejected because: Manual maintenance at thousands-of-institution scale was costly and slow when banks changed login experiences.

Patterns

data-quality-loop (mlops)

AI Annotator expands labeled-data capacity and feeds categorization model improvement.

human-in-the-loop-gate (quality)

Humans focus on golden datasets, edge cases, and quality spot checks rather than bulk labeling.

Tradeoffs

Where to put human labeling effort.

Chose golden data and spot checks over bulk annotation.

Rationale: The agent scales routine labels while humans preserve evaluation quality and handle edges.

How to spend scarce human review on transaction labeling.

Chose Use humans for golden datasets, edge cases, and quality spot checks. over Keeping humans in the loop for bulk labeling of every transaction..

Rationale: Plaid keeps human oversight where it improves benchmark and edge-case quality while shifting the bulk of label generation to AI Annotator.

Gotchas (what bit them)

Manual labeling could not keep pace with transaction volume.

Root cause: Millions of daily transactions require nuanced financial categorization for multiple products.

Resolution: Use an internal LLM annotation platform plus human oversight for benchmark and edge-case quality.

Evaluation

Plaid reports alignment and operational repair metrics.

Metrics: >95% human alignment · 2M successful logins · 90% repair-time reduction

Outcomes

• Plaid reports improved categorization, faster connection repairs, and more resilient user experiences.

Promotional product engineering blog; metrics are company-reported.