Intelligent Document Processing for Mortgage Servicing Using Amazon Bedrock and Multimodal AI

onity_group · Finance · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Mortgage-servicing document processing

Onity Group needed to process millions of pages across hundreds of mortgage document types while extracting buried legal facts, handwritten values, notarization details, seals, appraisal comparisons, and credit-report data. Traditional OCR and custom AI/ML could digitize text, but struggled with semantic context, visual cues, untrained document types, and inconsistent formats.

Constraints:

• documents include verbose legal forms such as deeds of trust where critical data is buried in dense text
• handwritten elements vary in quality, style, legibility, abbreviation, and numeric format
• notarization and legal seal checks require visual plus textual interpretation
• traditional OCR lacks semantic contextual understanding
• mortgage documents contain sensitive financial data requiring encryption, IAM controls, and secure processing
• the company processes millions of pages across hundreds of document types annually
• routing must balance accuracy, performance, cost, and speed to market
• different document types and fields require different extraction rules

Architecture

Onity built an intelligent document processing system that combines Amazon S3 ingestion, preprocessing, Amazon Textract, custom classification models, Amazon Bedrock foundation models, and persistence in operational databases and S3. Documents enter through S3 and preprocessing, Textract extracts initial content, a custom AI model classifies known document types when confidence is high, and unrecognized or complex cases route to Anthropic Claude Sonnet or other Bedrock text/vision models. Extraction rules are retrieved from an internal database, and the extraction application dynamically chooses Textract or Bedrock based on each document type and data element.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Amazon_S3_document_intake["Amazon S3 document intake"] --> Preprocessing_workflow["Preprocessing workflow"]
  Preprocessing_workflow --> Amazon_Textract["Amazon Textract"]
  Amazon_Textract --> Document_classification_application["Document classification application"]
  Document_classification_application --> Custom_AI_classification_model["Custom AI classification model"]
  Custom_AI_classification_model --> Amazon_Bedrock_classification_fallback["Amazon Bedrock classification fallback"]
  Document_classification_application --> Amazon_Bedrock_classification_fallback
  Amazon_Textract --> Document_extraction_application["Document extraction application"]
  Internal_extraction_rules_database["Internal extraction rules database"] --> Document_extraction_application
  Document_extraction_application --> Amazon_Bedrock_text_and_vision_models["Amazon Bedrock text and vision models"]
  Document_extraction_application --> Operational_databases["Operational databases"]
  Document_extraction_application --> Semi_structured_S3_output["Semi-structured S3 output"]
  Amazon_S3_document_intake --> Semi_structured_S3_output
  AWS_security_controls["AWS security controls"] --> Amazon_S3_document_intake
  AWS_security_controls --> Amazon_Bedrock_text_and_vision_models
  AWS_security_controls --> Operational_databases

Components

Data flow

Documents are uploaded to Amazon S3, which triggers preprocessing and automated processing workflows. Textract extracts contents for classification. Onity's document classification application sends the extracted content to a custom AI model; if confidence clears the threshold, classification completes, otherwise the application routes the document to Anthropic Claude Sonnet in Amazon Bedrock. For extraction, the application queries an internal database for document-type and field-specific rules, then dynamically routes each extraction task between Textract and Bedrock foundation models. Extracted information is stored in structured operational databases and semi-structured S3 output.

Why this architecture

Onity chose a hybrid OCR/custom-model/foundation-model architecture because mortgage documents vary widely in complexity. Textract and the custom AI model handle known or simpler work cost-effectively, while Bedrock text and vision models handle unrecognized document types, context-heavy legal language, handwriting, seals, and notarization. Bedrock's model catalog keeps the system flexible as cost and accuracy tradeoffs change by document type.

Models used: Amazon Textract · Onity custom AI model · Anthropic Claude Sonnet · Amazon Nova Pro · Amazon Nova Lite · Amazon Bedrock foundation models · text and vision foundation models

Scale

• Onity was founded in 1988 and operates through PHH Mortgage Corporation and Liberty Reverse Mortgage.
  “Onity Group, founded in 1988, is headquartered in West Palm Beach, Florida.”article, stated
• The document-processing workload spans millions of pages each year.
  “Onity processes millions of pages across hundreds of document types annually, including legal documents such as deeds of trust where critical information is often contained within dense text.”article, stated
• The corpus spans hundreds of document types annually.
  “Onity processes millions of pages across hundreds of document types annually, including legal documents such as deeds of trust where critical information is often contained within dense text.”article, stated
• The deed-of-trust use case reports a 50% extraction-cost reduction.
  “The intelligent extraction solution has reduced data extraction costs by 50% while improving overall accuracy by 20% compared to the previous OCR and AI/ML solution.”article, stated
• The appraisal-review use case reports a 65% accuracy improvement over manual review.
  “The automated solution now validates property comparisons and identifies potential discrepancies, significantly reducing review times while improving accuracy by 65% over the manual process.”article, stated
• The credit-report use case reports approximately 85% accuracy.
  “The solution automatically extracts and standardizes credit scores and scoring models across different report formats, achieving approximately 85% accuracy.”article, stated

Alternatives considered (roads not taken)

Use the previous OCR and AI/ML solution for mortgage document extraction

Rejected because: The prior approach struggled with dense legal text, handwriting, seals, and contextual interpretation, while the new solution reports lower costs and higher accuracy.

Use only the custom classification model

Rejected because: The classification application falls back to Claude Sonnet when the custom model was not trained on a document type.

Use Textract or OCR alone for all extraction tasks

Rejected because: Complex tasks such as notarization require deeper visual and textual analysis from Bedrock text and vision models.

Commit to a single foundation model for all document tasks

Rejected because: The Bedrock API lets Onity choose different models for the best balance of cost and accuracy by document type.

Patterns

model-routing (architecture)

Onity routes classification and extraction work across a custom AI model, Textract, Claude Sonnet, and other Bedrock text/vision models based on document recognition and task complexity.

confidence-thresholding (control)

The classification workflow accepts the custom model's classification only when confidence exceeds a predetermined threshold; otherwise it escalates to Bedrock.

cost-aware-routing (operations)

The system reserves Bedrock foundation models for complex cases while using Textract or custom models where sufficient, balancing cost, performance, and accuracy.

structured-output-contract (interface)

Classification and extraction prompts produce JSON outputs such as page type/page number, notarization fields, rider lists, appraisal explanations, and credit bureau score objects.

tool-use-orchestration (architecture)

The pipeline coordinates S3 triggers, preprocessing, Textract, custom models, Bedrock, internal extraction rules, operational databases, and S3 storage into one document-processing workflow.

safety-policy-enforcement (governance)

Because mortgage documents contain sensitive financial information, the architecture uses KMS encryption, TLS, IAM, Bedrock security capabilities, and AWS FSI/security best-practice guidance.

Tradeoffs

Route between custom AI and Bedrock instead of sending every classification to a foundation model.

Chose custom model first with Bedrock fallback for unrecognized document types over Bedrock-only classification for all documents.

Rationale: The custom model handles recognized document types, while Bedrock provides flexibility and speed when the model was not trained for a type.

Use dynamic extraction routing instead of one extractor for every data element.

Chose task-level routing between Textract and Bedrock FMs over a uniform OCR-only or FM-only extraction path.

Rationale: Simple fields can stay on cheaper extraction paths, while visually or semantically complex tasks can use Bedrock's text and vision models.

Build on Bedrock's model API instead of hard-coding one model provider.

Chose a managed API with multiple foundation-model providers over a single fixed model integration.

Rationale: Model choice can change by document type and as accuracy, performance, and cost profiles evolve.

Ask the model for explanations and structured checks rather than only field extraction.

Chose prompted analysis with JSON fields and explanations over plain OCR field extraction without reasoning output.

Rationale: Appraisal and notarization use cases require justification, rule checks, and handling of modified fields, not just raw text capture.

Gotchas (what bit them)

Traditional OCR digitized text but did not understand mortgage-document semantics.

Root cause: OCR could read text without interpreting the context needed to identify embedded legal facts or document meaning.

Resolution: Onity added foundation models through Bedrock for contextual text and vision analysis on complex cases.

Verbose legal documents hide critical fields inside unstructured paragraphs.

Root cause: Mortgage documents such as deeds of trust contain dense legalese where key data lacks obvious labels or structure.

Resolution: The extraction layer uses Bedrock for complex contextual analysis and document-specific extraction rules from an internal database.

Handwriting and format variation can break brittle extractors.

Root cause: The same value can appear in inconsistent handwriting, abbreviation, or numeric formats.

Resolution: The system routes complex handwritten and multimodal extraction tasks to Bedrock text and vision models rather than relying on OCR alone.

Notarization and seal checks require visual-textual reasoning, not just text extraction.

Root cause: Notary seals, court stamps, expiration dates, and modified handwritten fields combine layout, visual marks, and text.

Resolution: For notarization, the application passes the image to a prompt that extracts fields and handles manually modified values.

Model choice cannot be frozen when document types and cost/accuracy needs vary.

Root cause: Different mortgage document tasks demand different balances of performance, accuracy, vision capability, and cost.

Resolution: The solution uses Bedrock's model catalog and evaluation workflow to choose models by use case.

Sensitive financial document pipelines need security controls built into the architecture.

Root cause: Mortgage servicing documents contain financial and personal information that cannot be treated like generic text-processing data.

Resolution: Onity uses KMS, TLS, IAM policies, and AWS FSI/security guidance around the document-processing workflow.

Evaluation

The article evaluates Onity's solution through reported before/after business metrics against the previous OCR/AI/ML and manual processes, plus task-specific examples for deeds of trust, appraisal review, and credit reports. It reports cost reduction, overall accuracy improvement, appraisal-review accuracy improvement, and credit-report accuracy, but the metrics are vendor-published and not independently audited in the article.

Metrics: 50% reduction in document extraction costs · 20% improvement in overall accuracy · 65% improvement in appraisal-review accuracy over manual process · approximately 85% credit-report accuracy · cost · performance · accuracy · robustness

Outcomes

• Onity reports a 50% reduction in document extraction costs and 20% improvement in overall accuracy compared with the previous OCR/AI/ML solution.
• The deed-of-trust extraction path handles notarization fields, notary seals, crossed-out or modified fields, and rider information.
• The appraisal-review path validates room counts and square-footage variance and returns an explanation.
• Credit-report processing standardizes credit scores and scoring models across bureau formats with approximately 85% accuracy.
• The architecture gives Onity a flexible path to evolve model selection over time through Bedrock.

Onity processes millions of pages across hundreds of document types annually. Reported outcomes include 50% extraction-cost reduction, 20% overall accuracy improvement, 65% appraisal-review accuracy improvement over manual review, and approximately 85% credit-report accuracy. These are article-reported figures from an AWS customer story.