Building a RAG-Based Premium Audit Assistant for Insurance Workflows

verisk · Insurance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance premium-audit support and advisory search

Verisk needed to help premium auditors and underwriters get accurate answers from a very large PAAS document corpus without hours of manual review, while preserving data governance, reducing hallucinations, and supporting contextual follow-up questions.

Constraints:

• The PAAS corpus spans classification guides, advisory content, bulletins, training material, and product expertise
• Users have different access permissions, so retrieval and responses must respect data governance boundaries
• The platform evolves continuously, making static fine-tuning a poor fit for current answers
• Standard vector search did not retrieve all relevant context for premium-audit questions
• Responses needed custom evaluation for relevancy, context relevancy, and faithfulness

Architecture

Verisk built PAAS AI as a RAG conversational assistant inside the PAAS platform. User questions flow through a governed RAG pipeline that retrieves PAAS documentation from OpenSearch using hybrid BM25 plus dense vector search, stores chat history in ElastiCache, uses Claude models through Amazon Bedrock for response generation and conversation summarization, applies Bedrock and prompt-level guardrails, and evaluates answer quality with a custom evaluation API and customer-feedback loop.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  paas-user[PAAS User] --> paas-ai[PAAS AI]
  paas-ai[PAAS AI] --> rag-pipeline[RAG Pipeline]
  rag-pipeline[RAG Pipeline] --> opensearch[OpenSearch]
  opensearch[OpenSearch] --> paas-knowledge-base[PAAS Knowledge Base]
  rag-pipeline[RAG Pipeline] --> elasticache[ElastiCache]
  rag-pipeline[RAG Pipeline] --> amazon-bedrock[Amazon Bedrock]
  amazon-bedrock[Amazon Bedrock] --> claude-models[Claude Models]
  amazon-bedrock[Amazon Bedrock] --> guardrails-layer[Guardrails Layer]
  paas-ai[PAAS AI] --> evaluation-api[Evaluation API]
  evaluation-api[Evaluation API] --> customer-feedback[Customer Feedback]

Components

Data flow

A premium-audit user asks a PAAS question. The system retrieves authorized document chunks from OpenSearch using hybrid search over section- and length-based chunks, brings recent conversational context from ElastiCache, summarizes follow-up context with Claude when needed, sends retrieved excerpts to Claude through Bedrock, screens the interaction with Bedrock guardrails and specialized prompts, and feeds responses plus customer feedback into Verisk's evaluation loop.

Why this architecture

Verisk chose RAG over fine-tuning because PAAS data changes, authorized access matters, and transparency into retrieved context is needed to reduce hallucinations and improve documents. OpenSearch hybrid retrieval fixed vector-only misses, Claude handled domain language and conversation summarization, and a custom evaluator replaced generic tools that did not fit the insurance use case.

Models used: Anthropic Claude · Anthropic Claude Haiku · Anthropic Claude Sonnet

Scale

• PAAS contains more than 40,000 classification guides and more than 500 bulletins.
  “The Verisk PAAS platform houses a vast array of documents—including class guides, advisory content, and bulletins”article, implied

  “PAAS offers a wide range of essential services, including more than 40,000 classification guides and more than 500 bulletins.”article, stated
• The first rollout targeted one beta customer before broader expansion.
  “Verisk initially rolled out the PAAS AI to one beta customer to demonstrate real-world performance and impact.”article, stated
• Verisk plans to roll the assistant out toward approximately 15,000 users.
  “As Verisk collects additional information, this data will help uncover what will be needed to improve results and prepare to roll out to a wider customer base of approximately 15,000 users.”article, stated
• The beta reported a 96-98% reduction in processing time per specialist.
  “What previously took hours of manual review can now be accomplished in minutes, resulting in an extraordinary 96–98% reduction in processing time per specialist.”article, stated

Alternatives considered (roads not taken)

Fine-tune a model on PAAS content

Rejected because: PAAS changes continuously, so Verisk needed current data access without retraining and transparent retrieved context.

Use standard vector search alone

Rejected because: Vector search alone missed relevant contexts, so Verisk combined sparse BM25 with dense vector retrieval.

Use off-the-shelf evaluation tools unchanged

Rejected because: Generic tools had limitations for the PAAS use case, so the team built a custom evaluation API.

Patterns

rag-retrieval (retrieval)

PAAS AI grounds answers in retrieved PAAS documents rather than model memory alone.

hybrid-search (retrieval)

Verisk combines BM25 and dense vector search after vector-only retrieval missed contexts.

agent-memory (context)

The assistant stores chat history and uses Claude summaries so follow-up questions retain context.

safety-policy-enforcement (safety)

Bedrock guardrails and prompt-level checks screen inputs, outputs, unrelated questions, and prompt attacks.

eval-driven-development (evaluation)

A custom evaluation API and ground truth agreements track relevancy, context relevancy, and faithfulness.

Tradeoffs

Use RAG instead of fine-tuning for PAAS AI

Chose RAG over model fine-tuning over A fine-tuned model as the main knowledge strategy.

Rationale: RAG kept answers current, transparent, and configurable while avoiding frequent retraining.

Route different Claude models by task characteristics

Chose Claude Haiku for latency-sensitive scenarios and Claude Sonnet for detailed question answering over One model for every task.

Rationale: The team balanced latency, performance, and cost by matching model capability to task needs.

Use hybrid retrieval rather than vector search alone

Chose Sparse BM25 plus dense vector search over Standard vector search.

Rationale: Hybrid retrieval returned relevant contexts that vector-only search missed.

Gotchas (what bit them)

Vector-only search missed relevant insurance contexts.

Root cause: PAAS questions needed both lexical matches and semantic similarity across specialized insurance documents.

Resolution: Verisk combined sparse BM25 search with dense vector search.

Generic LLM evaluation tools did not fit the PAAS use case.

Root cause: The team needed domain-specific answer relevancy, context relevancy, and faithfulness checks.

Resolution: They built a proprietary evaluation API and kept ground truth agreements updated.

Open-ended generative AI could be misused or answer unrelated questions.

Root cause: The assistant is exposed to user prompts and model invocations in a regulated insurance support context.

Resolution: Verisk layered Bedrock guardrails with specialized prompts for unrelated questions and prompt attack attempts.

Evaluation

Verisk uses a custom evaluation API with answer relevancy, context relevancy, faithfulness, customer feedback analysis, QA test cases, and ground truth agreements.

Metrics: Answer relevancy score · Context relevancy score · Faithfulness score · Ground truth agreement reviews · Customer feedback analysis

Outcomes

• Beta usage reduced processing time per specialist by 96-98%
• Manual review that previously took hours can be completed in minutes
• Verisk identified additional business-value areas for broader rollout

The early rollout started with one beta customer and is preparing for a wider customer base of approximately 15,000 users.