Scaling Email Content Extraction Using LLMs in Production

yahoo · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: consumer email content extraction

Yahoo Mail needed to break through a coverage ceiling in receipt and package extraction across billions of daily emails while preserving privacy and production reliability.

Constraints:

• Traditional sequence-to-sequence models covered top senders but struggled with the long tail.
• Yahoo Mail receives billions of messages across hundreds of millions of mailboxes every day.
• The production system had to bridge on-prem extraction pipelines with Vertex AI while meeting legal and privacy requirements.

Architecture

Yahoo deployed LLM-based extraction on Google Cloud Vertex AI with a hybrid on-prem/cloud architecture. Their Yahoo private-cloud pipelines sample live traffic, compare legacy and LLM models, copy test/training data to GCS, fine-tune and serve models through Vertex AI endpoints, connect Vertex endpoints back into the on-prem extraction pipeline, and use error analysis plus active learning for model improvement.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Yahoo Mail messages"] --> B["On-prem extraction pipeline"]
  B --> C["Legacy sequence model"]
  C --> D["Live traffic comparison"]
  B --> H["Vertex AI endpoint"]
  H --> D
  D --> K["Go/no-go release gate"]
  F["Yahoo private-cloud data"] --> E["GCS training store"]
  E --> G["Vertex AI notebook"]
  G --> H
  B --> I["Error analysis tool"]
  I --> J["Active learning loop"]
  J --> G

Components

Data flow

Live Yahoo Mail extraction traffic is sampled and run through both the existing production model and the new LLM model for go/no-go comparison. Training/test data moves from Yahoo private cloud to GCS, Vertex AI notebooks and endpoints support model fine-tuning and serving, the on-prem extraction pipeline calls the Vertex endpoint, and error analysis plus user feedback feeds an active-learning loop.

Why this architecture

Yahoo chose a hybrid Vertex AI/on-prem architecture because the existing mail extraction stack and user data remained in Yahoo infrastructure, while Vertex AI provided GPU access, managed endpoints, and cloud tooling needed for LLM training and inference at scale.

Models used: Google PL5 250M parameter model · legacy sequence-to-sequence extraction model · Vertex AI fine-tuned LLM

Scale

• Yahoo Mail receives billions of messages daily across hundreds of millions of mailboxes.
  “for Yahoo mail every day we receive billions of messages or hundreds of million of mailboxes”transcript, stated
• The team planned to scale to 100% of US daily active users for shopping season.
  “to scale up to 100% of our us daily active users”transcript, stated
• The deployed system produced almost 3.5 million new purchase receipt extractions per day.
  “almost 3 and a half million new purchase reset extraction on daily basis”transcript, stated
• The package extractor produced 5.5 million new package extractions and covered almost 99% of tail domains.
  “we were able to extract 5.5 million of new packages with covering almost 99% of the tail domain”transcript, stated
• The chosen model was a 250-million-parameter PL5 open-source model.
  “the model we chose for our use case was pl5 open source model from Google with 250 million parameters”transcript, stated

Alternatives considered (roads not taken)

Continue legacy sequence-to-sequence models only

Rejected because: They hit an 80/20 coverage ceiling and failed to cover long-tail sender diversity.

Fine-tune LLMs only on Yahoo infrastructure

Rejected because: The existing infrastructure lacked enough GPUs for LLM fine-tuning.

Blind rollout without live-traffic comparison

Rejected because: The hybrid migration needed a sampled quality comparison before go/no-go decisions.

Patterns

canary-rollout (deployment)

Yahoo sampled live traffic and compared legacy and LLM extractors before go/no-go rollout decisions.

fine-tuning-adaptation (model-development)

The team used Vertex AI notebooks and GPUs to fine-tune an LLM-based extractor for Yahoo Mail extraction.

data-quality-loop (operations)

Yahoo connected error analysis and user feedback into an active-learning loop for model improvement.

privacy-preserving-processing (governance)

The extraction architecture prioritized secure, legal, and privacy-compliant handling of mailbox data.

Tradeoffs

Model modernization

Chose LLM extraction through Vertex AI over legacy sequence-to-sequence extractors alone.

Rationale: LLMs addressed the long-tail coverage ceiling and freshness requirements.

Release validation

Chose sampled live-traffic comparison over offline-only launch confidence.

Rationale: Production comparison caught a November-release issue before the successful December release.

Serving deployment

Chose multiple Vertex AI endpoints across US regions over single endpoint deployment.

Rationale: Endpoint placement was aligned to Yahoo's traffic pattern for scale.

Gotchas (what bit them)

Top-sender coverage hid long-tail user pain.

Root cause: The 80/20 sender distribution made aggregate coverage look acceptable while missing individual receipts or packages still broke user trust.

Resolution: Use LLM extraction to expand coverage beyond the top-sender ceiling.

Six-month retraining was too slow for email commerce data.

Root cause: New receipts, packages, and sender formats arrive continuously.

Resolution: Move to a faster GenAI/Vertex experimentation and release process.

Hybrid on-prem/cloud integration was more complex than expected.

Root cause: Yahoo was mid-cloud transformation, with systems partly on-prem and new solutions in Vertex AI.

Resolution: Build network configuration, validation, and privacy controls between the on-prem extraction pipeline and Vertex endpoints.

Evaluation

Yahoo evaluated model changes by comparing legacy and LLM extractors on sampled live traffic, post-validation before release, error analysis, extraction richness, domain coverage, and tracking API errors.

Metrics: go/no-go sampled quality · standard-domain coverage · tail-domain coverage · daily receipt extractions · daily package extractions · extraction richness · tracking API error reduction

Outcomes

• Yahoo achieved almost 3.5 million new daily purchase receipt extractions with 94% standard-domain coverage.
• Yahoo achieved 5.5 million package extractions with almost 99% tail-domain coverage.
• Extraction richness increased by 51%.
• Tracking API errors fell by 16%.
• The product reached 100% of US daily active users after a delayed but successful release.

The system runs over consumer email at Yahoo Mail scale, which the source describes as billions of daily messages and hundreds of millions of mailboxes.