Email Classification System Using Foundation Models and Prompt Engineering

travelers_insurance · Insurance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance service-email classification

Travelers needed to classify millions of incoming agent and customer service-request emails into policy-service categories quickly enough for production automation without building and maintaining a traditional supervised classifier.

Constraints:

Emails can include ambiguous insurance requests and PDF attachments.
The system must classify into predefined service request categories.
Production use requires high accuracy and acceptable latency.
Manual processing consumes large operational effort.
Fine-tuning would add cost and was not necessary if prompt engineering reached sufficient accuracy.

Architecture

The Travelers/AWS solution is a serverless email-classification pipeline using Anthropic Claude models on Amazon Bedrock. Raw email bodies are extracted, PDF attachments are split into pages and processed by Amazon Textract for OCR, entities, and tables, and the email body plus attachment text are cleaned and combined into one LLM prompt. Claude classifies the content into one of 13 categories and returns the class for downstream automation and performance analysis.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  email-ingestion[Email ingestion] --> email-parser[Email parser]
  email-ingestion --> pdf-processor[PDF processor]
  pdf-processor --> amazon-textract[Amazon Textract]
  email-parser --> prompt-builder[Prompt builder]
  amazon-textract --> prompt-builder
  prompt-builder --> claude-on-bedrock[Claude on Bedrock]
  claude-on-bedrock --> automation-system[Automation system]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Email ingestion`	Receives raw Outlook .msg or .eml service request emails.	`Email parser`, `PDF processor`
`Email parser`	Extracts and cleans body text from email files.	`Prompt builder`
`PDF processor`	Splits PDF attachments into page images for OCR.	`Amazon Textract`
`Amazon Textract`	Extracts text, entities, and table data from PDF page images.	`Prompt builder`
`Prompt builder`	Combines email and attachment text with persona, examples, class definitions, and output instructions.	`Claude on Bedrock`
`Claude on Bedrock`	Foundation model classifier returning one of 13 categories.	`Automation system`
`Automation system`	Uses predicted classes to route or automate service processing.	`Claude on Bedrock`

Data flow

A raw Outlook or EML email enters the pipeline. The body text is parsed and cleaned; PDF attachments are split into images and processed with Amazon Textract for OCR, entity extraction, and tables. The email body and attachment text are combined into a prompt, and Claude on Bedrock returns one of 13 service categories.

“The workflow started with an email, then, given the email’s text and any PDF attachments, the email was given a classification by the model.”article, stated

“The text from the email body and PDF attachment are combined into a single prompt for the large language model (LLM).”article, stated

Why this architecture

The team chose a prompt-engineered foundation-model classifier because it was faster to develop, easier to switch between models, explainable, and extensible to related classification tasks. PDF Textract preprocessing was needed because attachments contained predictive signals, and prompt structure reduced output variance without fine-tuning.

“there are benefits to building an FM-based classifier using an API service such as Amazon Bedrock, such as the speed to develop the system, the ability to switch between models”article, stated

“By providing the FM with examples and other prompting techniques, we were able to significantly reduce the variance in the structure and content of the FM output”article, stated

Models used: Anthropic Claude v2 · Anthropic Claude Instant · Amazon Bedrock · Amazon Textract

Scale

Travelers receives millions of service request emails per year.
“Travelers receives millions of emails a year with agent or customer requests to service policies.”article, stated
The classifier predicts one of 13 defined categories.
“Anthropic’s Claude classifies this content into one of 13 defined categories and then returns that class.”article, stated
The system can save tens of thousands of hours of manual processing.
“can save tens of thousands of hours of manual processing and redirect that time toward more complex tasks.”article, stated
The ground truth dataset had over 4,000 labeled examples.
“The ground truth dataset contained over 4,000 labeled email examples.”article, stated
About 25% of emails had PDF attachments.
“Approximately 25% of the emails had PDF attachments, of which most were ACORD insurance forms.”article, stated

Alternatives considered (roads not taken)

Train a traditional supervised classifier.

Rejected because: Prompt engineering with a pre-trained FM avoided training resources and still met production accuracy.

“instead of using supervised learning, which normally involves training resources, we used prompt engineering with few-shot prompting”article, stated

Fine-tune a foundation model.

Rejected because: Accuracy was already high through prompt engineering, so fine-tuning would need to justify extra cost.

“given the accuracy was already high by just using prompt engineering, the accuracy after fine-tuning would have to justify the cost.”article, stated

Ignore attachments or non-PDF attachments.

Rejected because: PDF forms carried additional signal, though the implementation limited scope to PDFs and ignored other attachments.

“The PDF forms included additional details that provided a signal for the classifier. Only PDF attachments were processed to limit the scope; other attachments were ignored.”article, stated

Patterns

`prompt-versioning` (prompting)

Travelers improved classification through structured prompt sections: persona, instruction, few-shot examples, class definitions, input, and output instruction.

“The structure of the prompt was as follows: * Persona definition * Overall instruction * Few-shot examples * Detailed definitions for each class”article, stated

`structured-output-contract` (data)

Claude returns one of 13 defined categories for each combined email/attachment prompt.

“Anthropic’s Claude classifies this content into one of 13 defined categories and then returns that class.”article, stated

`batch-offline-processing` (operations)

The serverless pipeline processes email bodies and PDF attachments before classification for automation.

“This solution is serverless, which has many benefits for the organization.”article, stated

Tradeoffs

Classifier approach

Chose Prompt-engineered pre-trained FM on Bedrock. over Supervised training or fine-tuning first..

Rationale: The FM path enabled faster development, model switching, rationale output, extensibility, and high enough accuracy.

“This allowed us to use a pre-trained FM without having to incur the costs of training.”article, stated

Attachment scope

Chose Process PDF attachments only. over Processing every possible attachment type..

Rationale: PDF forms provided useful signal while limiting implementation scope.

“Only PDF attachments were processed to limit the scope; other attachments were ignored.”article, stated

Infrastructure model

Chose Serverless AWS managed services. over Self-managed classifier infrastructure..

Rationale: Serverless reduced ownership cost and maintenance complexity.

“With a serverless solution, AWS provides a managed solution, facilitating lower cost of ownership and reduced complexity of maintenance.”article, stated

Gotchas (what bit them)

Attachments contained classifier signal.

Root cause: PDF forms included details such as names, policy numbers, dates, and table data.

Resolution: The pipeline uses Amazon Textract to extract text, entities, and tables from PDF page images.

“Additional entity extraction and table data detection was included to identify names, policy numbers, dates, and more.”article, stated

Initial prompt-free or weak-prompt classification was not production accurate.

Root cause: Initial testing without prompt engineering achieved only 68% accuracy.

Resolution: The team refined categories, document processing, instructions, and prompt engineering to reach 91%.

“Initial testing without prompt engineering yielded 68% accuracy.”article, stated

Fine-tuning cost had to be justified against prompt-engineering gains.

Root cause: Prompt engineering already achieved high accuracy and Bedrock fine-tuning availability was limited during the engagement.

Resolution: The team shipped the prompt-engineered classifier rather than fine-tuning.

“Although at the time of the engagement, Anthropic’s Claude models weren’t available for fine-tuning on Amazon Bedrock”article, stated

Evaluation

Travelers evaluated classification accuracy on a labeled ground-truth dataset and iterated prompt engineering, category condensation, document processing, and instructions.

Metrics: Initial accuracy 68% · Claude v2 accuracy 91% after improvements · Claude Instant accuracy 90% · Over 4,000 labeled email examples

“After using a variety of techniques with Anthropic’s Claude v2, such as prompt engineering, condensing categories, adjusting document processing process, and improving instructions, accuracy increased to 91%.”article, stated

“Anthropic’s Claude Instant on Amazon Bedrock also performed well, with 90% accuracy”article, stated

Outcomes

Reached 91% classification accuracy through prompt engineering.
Powered an automation system that can save tens of thousands of manual-processing hours.
Handled complex, ambiguous service request emails and PDF attachment text.
Demonstrated FM classification as a production alternative to supervised models.

Travelers receives millions of emails yearly; the dataset included over 4,000 labeled emails, about 25% with PDFs, and the classifier predicts 13 categories.

“FMs can reliably automate the classification of insurance service emails through prompt engineering.”article, stated