Insurance Policy Review Automation Using Retrieval-Augmented Generation and Prompt Engineering

verisk · Insurance · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance policy document comparison and review automation

Verisk needed to help insurance professionals compare complex legal policy forms and understand material changes without spending days or weeks manually reviewing document versions. The system had to preserve domain accuracy, ignore non-meaningful textual noise, operate under security and compliance governance, and produce structured summaries that policy experts could trust.

Constraints:

• policy forms are complex legal documents with coverage and exclusion language
• reviewers need material-change summaries rather than raw text diffs
• document metadata such as type, jurisdiction, version number, and effective dates matters for retrieval relevance
• foundation-model output had to meet accuracy, consistency, adherence-to-context, speed, and cost criteria
• security, compliance, data use, IP protection, and provider data-retention rules were governance constraints
• change summaries, excerpts, locations, and redline-style tracked changes required different technical approaches
• customers needed review time reduced from days or weeks to minutes

Architecture

Verisk built the Mozart companion as a RAG-backed document comparison service on AWS. Policy documents live in S3; a periodic AWS Batch ingestion job chunks them, creates Amazon Titan embeddings through Bedrock, stores vectors in OpenSearch, and records document metadata through an internal Metadata API. At request time, a user selects two documents, Lambda retrieves relevant embeddings and metadata, Claude 3 Sonnet through Bedrock generates structured change summaries, and an API returns JSON results to the UI alongside excerpts and tracked-change formatting. Verisk kept non-FM logic for redlines and section-difference detection, while domain experts drove prompt iterations and human evaluation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Amazon_S3_policy_document_store["Amazon S3 policy document store"] --> AWS_Batch_ingestion_job["AWS Batch ingestion job"]
  AWS_Batch_ingestion_job --> Amazon_Titan_Text_Embeddings["Amazon Titan Text Embeddings"]
  AWS_Batch_ingestion_job --> Internal_Metadata_API["Internal Metadata API"]
  Amazon_Titan_Text_Embeddings --> OpenSearch_vector_database["OpenSearch vector database"]
  Mozart_document_selection_UI["Mozart document-selection UI"] --> AWS_Lambda_comparison_function["AWS Lambda comparison function"]
  OpenSearch_vector_database --> AWS_Lambda_comparison_function
  Internal_Metadata_API --> AWS_Lambda_comparison_function
  AWS_Lambda_comparison_function --> Claude_3_Sonnet_on_Amazon_Bedrock["Claude 3 Sonnet on Amazon Bedrock"]
  Prompt_and_definition_library["Prompt and definition library"] --> Claude_3_Sonnet_on_Amazon_Bedrock
  AWS_Lambda_comparison_function --> Non_FM_difference_engine["Non-FM difference engine"]
  AWS_Lambda_comparison_function --> API_result_service["API result service"]
  Non_FM_difference_engine --> API_result_service
  API_result_service --> Mozart_document_selection_UI
  Domain_expert_evaluation_loop["Domain expert evaluation loop"] --> Prompt_and_definition_library
  Domain_expert_evaluation_loop --> Governance_review["Governance review"]

Components

Data flow

Policy forms are stored in Amazon S3. A periodic AWS Batch job reads the documents, chunks text into smaller slices, embeds those chunks with Amazon Titan Text Embeddings through Bedrock, stores them in an OpenSearch vector database, and records metadata such as document type, jurisdiction, version number, and effective dates through an internal Metadata API. When a user chooses two documents to compare, a Lambda function retrieves document embeddings, sends the relevant context to Claude 3 Sonnet through Bedrock, stores the generated JSON result, and serves it through the API to a UI that displays summaries, excerpts, locations, and tracked changes.

Why this architecture

The architecture separates retrieval, generation, deterministic document-diff work, and expert review because the product needed more than a single FM call. RAG and metadata filtering make the comparison context relevant, prompt and splitting iterations teach the model to focus on material changes rather than noise, non-FM logic handles redline construction and section-difference detection more cheaply, and human expert evaluation keeps insurance-domain quality visible.

Models used: Anthropic Claude 3 Sonnet · Amazon Titan Text Embeddings · other foundation models evaluated during development · Amazon Titan Embeddings V2 planned for future use

Scale

• Mozart reduces insurance policy change adoption time from days or weeks to minutes.
  “cutting the change adoption time from days or weeks to minutes.”article, stated
• Human expert testing produced over 90% good or acceptable summaries.
  “they concluded that the results generated over 90% good or acceptable summaries.”article, stated
• Document splitting used 500-character chunks with 15% overlap.
  “a recursive character text splitter with a chunk size of 500 characters with 15% overlap provided the best results.”article, stated
• Domain experts manually graded output quality on a 1-10 scale.
  “in-house domain experts would grade accuracy, consistency, and adherence to context on a manual grading scale of 1–10.”article, stated

Alternatives considered (roads not taken)

Amazon Bedrock Knowledge Bases for embedding storage

Rejected because: Verisk considered it because it is purpose-built for embeddings in OpenSearch Serverless, but the described implementation used a custom AWS Batch, Bedrock embedding, OpenSearch, and Metadata API pipeline.

Foundation-model-only solution

Rejected because: A single-FM approach was insufficient for the target accuracy, consistency, and efficiency, so Verisk combined prompts, RAG, system design optimizations, and non-FM components.

Foundation model for tracked redline construction

Rejected because: Verisk moved redline construction to deterministic non-FM logic, reducing FM load and improving cost/accuracy for that subtask.

NLTK splitter

Rejected because: It was considered during splitting experiments, but the recursive character text splitter with 500-character chunks and 15% overlap performed best for this use case.

Static model choice

Rejected because: Verisk kept evaluating FM options as new lower-cost or better-performing models became available.

Patterns

rag-retrieval (retrieval)

Verisk embedded policy document chunks, stored them in OpenSearch, and retrieved relevant context for Claude 3 Sonnet when users compared two documents.

eval-driven-development (quality)

Multiple rounds of expert human grading drove prompt, splitting, retrieval, model, and system-design improvements.

human-in-the-loop-gate (governance)

Insurance domain experts reviewed generated summaries, graded output quality, and supplied the feedback used to tune the system.

cost-aware-routing (operations)

Verisk reduced FM usage by routing only summary generation to the model where possible, using non-FM logic for redlines and evaluating lower-cost model options over time.

safety-policy-enforcement (governance)

A governance council and legal review controlled whether generative AI solutions met Verisk's security, compliance, data-use, and IP requirements.

structured-output-contract (interface)

The comparison output was stored as JSON and exposed through an API to the end-user interface.

prompt-versioning (prompting)

Verisk iteratively added domain instructions, terms, exclusions, and model-specific prompt adjustments based on expert findings.

Tradeoffs

Use a hybrid FM and non-FM architecture.

Chose FM for material-change summarization plus non-FM logic for changed-section detection and redlines over using the foundation model for every comparison subtask.

Rationale: This reduced FM load and improved accuracy/cost for deterministic formatting tasks while keeping generative reasoning where it mattered.

Use recursive character splitting with overlap.

Chose 500-character recursive chunks with 15% overlap over other tested splitting strategies such as NLTK splitting.

Rationale: For this policy-document use case, the recursive splitter gave the best processable chunks for downstream summarization quality.

Keep prompt designs model-specific.

Chose adjust prompts based on the selected foundation model over one prompt expected to work identically across FMs.

Rationale: Model behavior differed enough that model-specific prompts produced better responses.

Evaluate and switch foundation models over time.

Chose regular FM option evaluation for lower cost and better performance over locking the solution to one model regardless of market changes.

Rationale: The model landscape changed quickly enough that ongoing price-performance review was a production optimization lever.

Build reusable APIs and common components.

Chose decoupled API, common prompts, common definitions, retrieval service, embedding creation, and persistence service over a one-off UI-coupled comparison feature.

Rationale: Reusable architecture let Verisk expose the capability independently of the frontend and reuse components across generative AI workflows.

Gotchas (what bit them)

Raw text diffs did not satisfy the business need.

Root cause: Policy reviewers needed concise material-change explanations, not just highlighted textual differences.

Resolution: Verisk tuned prompts and domain instructions so the model described material changes and ignored non-meaningful noise.

Initial FM outputs were not accurate and consistent enough.

Root cause: A plain model invocation was insufficient for the insurance-domain quality target.

Resolution: Verisk iterated across redesigns, multiple FM calls, FM comparisons, prompt engineering, RAG, and system optimizations.

Prompt behavior varied by foundation model.

Root cause: Different FMs responded differently to the same instructions.

Resolution: Verisk adjusted prompts for each FM, including role framing, common definitions, and exclusions.

Retrieval quality depended on selecting the right documents and metadata.

Root cause: Broad or poorly filtered retrieval could provide less relevant context to the FM.

Resolution: Verisk used document metadata to include or exclude documents and narrow search results.

FM usage was too expensive or unnecessary for every subtask.

Root cause: The product had several outputs, and not all of them required generative reasoning.

Resolution: Verisk found changed sections before calling the FM, used non-FM logic for redlines, and evaluated lower-cost FMs as they became available.

Generative AI introduced security, compliance, and data-use obligations.

Root cause: Policy documents and insurance data are sensitive, and model providers must not retain or train on Verisk data.

Resolution: Verisk used governance council review, legal review, secure transmission requirements, and Bedrock/provider selection criteria.

Evaluation

Verisk evaluated generated policy-change summaries through multiple rounds of in-house insurance-domain expert review. Experts graded accuracy, consistency, and adherence to context on a 1-10 scale, while the team also tracked latency, speed, and cost. Evaluation findings fed prompt changes, document splitting strategy, FM selection, RAG design, and system optimizations.

Metrics: accuracy · consistency · adherence to context · speed · cost · latency · 1-10 manual grading scale · over 90% good or acceptable summaries

"To assess the quality of the results produced by generative AI, Verisk evaluated based on the following criteria:

• Accuracy
• Consistency
• Adherence to context
• Speed and cost"

Outcomes

• Mozart companion compares policy documents and summarizes changes in minutes.
• Change adoption time fell from days or weeks to minutes.
• Domain-expert evaluations concluded that generated results were over 90% good or acceptable.
• The system reduced FM load by passing only changed sections for summary generation and using non-FM redline logic.
• The architecture became reusable through API, retrieval, embedding, prompt, definition, and persistence components.

The article reports customer review time reduced from days or weeks to minutes and over 90% good-or-acceptable summaries in expert testing. It does not disclose production volume, number of customers using the companion, throughput, or cost figures.