AI Agent Development and Evaluation Platform for Insurance Underwriting

snorkel · Insurance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance underwriting agent evaluation

Snorkel needed realistic enterprise benchmarks for underwriting agents that use tools, ask users for missing information, and follow proprietary guidelines.

Constraints:

• domain expertise
• multi-turn interaction
• tool correctness
• hallucination risk

Architecture

Snorkel built an underwriting-agent benchmark rather than a production underwriting app: CPCU experts designed realistic small-business insurance scenarios, a fictional All National Insurance environment exposed applicant information, databases, and long-form underwriting guidelines, and each benchmarked model was wrapped as a LangGraph/MCP ReAct agent. The agent interacts with an underwriter, uses tools and resources in sequence, and is evaluated for task accuracy, tool correctness, efficiency, hallucination, and domain-specific business-rule failures.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  cpcu-expert-network[CPCU expert network] --> fictional-underwriting-dataset[Fictional underwriting dataset]
  cpcu-expert-network --> evaluation-harness[Evaluation harness]
  fictional-underwriting-dataset --> react-model-agent[ReAct model agent]
  langgraph-mcp-runtime[LangGraph MCP runtime] --> react-model-agent
  react-model-agent --> applicant-database[Applicant database]
  react-model-agent --> underwriting-guidelines[Underwriting guidelines]
  react-model-agent --> underwriter-interaction[Underwriter interaction]
  react-model-agent --> evaluation-harness
  fictional-underwriting-dataset --> evaluation-harness

Components

Data flow

A benchmark scenario starts with a fictional small-business applicant and incomplete information held by a junior underwriter. The LangGraph/MCP runtime wraps a model as a ReAct agent. The agent asks the underwriter clarifying questions, reads underwriting guidelines, queries database tables, and composes an answer for tasks such as appetite checks, classification, limits, deductibles, or product recommendations. The evaluation harness scores accuracy plus tool-use correctness and efficiency, using traces to identify failures.

Why this architecture

The benchmark is domain-specific because generic reasoning scores did not predict enterprise agent behavior: underwriting requires proprietary rules, tool sequencing, and interaction with incomplete user information. LangGraph/MCP plus ReAct matched common practitioner prototypes, while CPCU-authored data made errors actionable for enterprise deployment rather than leaderboard-only comparison.

Models used: frontier open and closed LLMs · o4-mini · Gemini 2.5 Pro · Claude 4 Sonnet

Scale

• The benchmark synthesizes thousands of fictional companies across small-business profiles.
  “We synthesized thousands of fictional companies to represent the broad spectrum of small businesses in North America.”article, stated
• Harder tasks require at least three or four tools composed in the right order.
  “The most challenging tasks forced models to use at least 3 or 4 tools and compose them correctly (using the results from one tool to use the next, etc), probing the underwriter for information along the way.”article, stated
• Tool-call errors appeared in 36% of conversations, including 30-50% for the three most accurate models.
  “Across models, including top performers, agents made at least one tool call error in 36% of the conversations despite access to the metadata required to use tools properly.”article, stated
• Some top OpenAI models hallucinated products outside the guidelines 15-45% of the time.
  “the top performing models from OpenAI hallucinated several products not in the guidelines 15-45% of the time.”article, stated

Alternatives considered (roads not taken)

Use academic coding/math benchmarks as the primary signal

Rejected because: Those tasks are easier to verify and miss enterprise-specific issues like proprietary rules and business logic.

Hand-prompt every tool edge case

Rejected because: The team wanted realistic enterprise behavior where dozens of tools create too many combinations for prompt-by-edge-case control.

Trust pretrained insurance domain knowledge

Rejected because: Models imported generic insurance products that were not in the provided guidelines, creating subtle but high-risk inaccuracies.

Patterns

eval-driven-development (quality)

The project centers on a vertical benchmark with task correctness, tool correctness, tool efficiency, and hallucination analysis.

tool-use-orchestration (agent-architecture)

Each benchmarked copilot must choose resources, execute tools, and sequence underwriting logic over multiple turns.

Tradeoffs

How much test-time compute to spend.

Chose higher-turn, higher-token reasoning for some models over lower-cost shorter trajectories.

Rationale: Accuracy improved with more turns/tools for some models, but inefficient exploration could still underperform.

Benchmark realism.

Chose CPCU-authored vertical scenarios with proprietary-style rules over generic academic benchmarks or simple verifiable tasks.

Rationale: The goal was actionable enterprise insight into domain, tool-use, and hallucination failures that generic benchmarks do not expose.

Prompt engineering posture.

Chose simulate realistic deployed tool complexity over engineering prompts for every known tool-use edge case.

Rationale: Dozens of tools and combinations make exhaustive prompt hand-holding impractical in real copilot systems.

Gotchas (what bit them)

Strong models still made frequent tool errors and vertical hallucinations.

Root cause: Off-the-shelf reasoning ability did not reliably transfer to proprietary underwriting workflows and schemas.

Resolution: Use realistic enterprise benchmark data and expert-created scenarios to expose error modes.

Models hallucinated generic insurance products that were not in the proprietary underwriting guidelines.

Root cause: Pretrained domain knowledge overrode the provided enterprise-specific rules.

Resolution: Evaluate for vertical hallucinations against proprietary guideline documents, not only final task accuracy.

Evaluation

Snorkel measured task correctness, tool correctness, efficiency, hallucinations, and per-task accuracy by model.

Metrics: accuracy single digits to ~80% · 36% tool error conversations · 15-45% hallucinated products

Outcomes

• The benchmark revealed actionable failure modes that generic reasoning benchmarks would miss.

This is an evaluation platform, not an underwriting production deployment.