Agentic AI Copilot for Insurance Underwriting with Multi-Tool Integration

snorkel · Insurance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance underwriting evaluation

Snorkel needed a realistic benchmark for underwriting agents that reason over proprietary knowledge and tools across multi-turn conversations.

Constraints:

Specialized insurance rules
Tool-use failures
Hallucination from pretrained knowledge
Expert verification

Architecture

Snorkel's underwriting case is a benchmark architecture for evaluating agentic insurance copilots, not a deployed underwriting product. CPCU experts design realistic company profiles, business rules, guidelines, and appropriate responses; the simulated All National Insurance environment gives junior-underwriter conversations incomplete applicant information; a LangGraph/MCP ReAct agent wraps each benchmarked model; tools expose databases and long-form underwriting guidelines; Snorkel's evaluation suite scores task correctness, conciseness, tool-use correctness, and tool-use efficiency while surfacing hallucination and tool-call failure modes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  cpcu-expert-network[CPCU expert network] --> benchmark-dataset[Benchmark dataset]
  benchmark-dataset --> junior-underwriter-conversation[Junior underwriter conversation]
  junior-underwriter-conversation --> react-underwriting-agent[ReAct underwriting agent]
  react-underwriting-agent --> langgraph-mcp-harness[LangGraph MCP harness]
  react-underwriting-agent --> underwriting-tools[Underwriting tools]
  react-underwriting-agent --> evaluation-suite[Evaluation suite]
  evaluation-suite --> failure-analysis[Failure analysis]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`CPCU expert network`	Experts who create and verify realistic underwriting data, rules, and responses.	`Benchmark dataset`
`Benchmark dataset`	Simulated All National Insurance scenarios, company profiles, databases, and guidelines.	`CPCU expert network`, `Junior underwriter conversation`
`Junior underwriter conversation`	Multi-turn interaction with incomplete applicant information.	`Benchmark dataset`, `ReAct underwriting agent`
`ReAct underwriting agent`	Model-specific agent wrapper that reasons and uses tools.	`Junior underwriter conversation`, `LangGraph MCP harness`, `Underwriting tools`
`LangGraph MCP harness`	Agentic system framework and tool protocol used across benchmarked models.	`ReAct underwriting agent`
`Underwriting tools`	Databases, metadata, and long-form underwriting guidelines.	`ReAct underwriting agent`
`Evaluation suite`	Measures correctness, conciseness, tool use, efficiency, and failure modes.	`ReAct underwriting agent`, `Failure analysis`
`Failure analysis`	Model-specific analysis of tool errors, hallucinations, turns, and token consumption.	`Evaluation suite`

Data flow

CPCU experts create realistic underwriting scenarios, tables, guidelines, and answers. A junior-underwriter conversation starts with incomplete applicant data, the AI copilot asks clarifying questions, selects database or guideline tools through LangGraph/MCP, solves the underwriting task, and Snorkel evaluates the conversation for correctness, conciseness, tool use, efficiency, and hallucination.

3 supporting quotes

“We developed the agentic system in LangGraph with [Model Context Protocol (MCP)]”article, stated

“The AI copilot, in turn, has access to resources that include databases and underwriting guidelines in long-form documents.”article, stated

“we developed several scalable measures, including task solution correctness, task solution conciseness, tool use correctness, and tool use efficiency.”article, stated

Why this architecture

Snorkel built a realistic tool-and-data benchmark because generic benchmarks and off-the-shelf checks do not capture proprietary underwriting logic. The LangGraph/MCP harness lets many open and closed models face the same multi-turn, tool-using tasks, while CPCU expert data makes the benchmark closer to real enterprise underwriting than toy scenarios.

3 supporting quotes

“in specialized settings, off-the-shelf approaches fail and expert data is critical. To illustrate this idea, Snorkel Research is developing a series of benchmarks that include challenging features, such as: * Reasoning over specific and proprietary knowledge”article, stated

“We note that expert data and verification are key, as without these, the resulting benchmark is highly unrealistic and unlikely to match real-life use cases.”article, stated

“correct sequence of tool use behavior was sometimes very complex. Example: determine whether an applicant even qualifies as a small business, AI copilots had to: 1.”article, stated

Models used: o4-mini · Gemini 2.5 Pro · Claude 4 Sonnet · frontier models

Scale

Model accuracy ranged from single digits to about 80%.
“Our leaderboard highlights a wide range of accuracies across frontier models, from the single digits up to ~80%.”article, stated
Agents made at least one tool-call error in 36% of conversations.
“* Tool use errors: Across models, including top performers, agents made at least one tool call error in 36% of the conversations despite access to the metadata required to use tools properly.”article, stated
Some OpenAI models hallucinated non-guideline products 15-45% of the time.
“For example, in the task involving insurance product recommendations, the top performing models from OpenAI hallucinated several products not in the guidelines 15-45% of the time depending on the model.”article, stated

Alternatives considered (roads not taken)

Use generic enterprise agent benchmarks

Rejected because: Snorkel argues specialized underwriting requires proprietary knowledge, expert data, and tool-use tasks.

Prompt-engineer every edge case

Rejected because: Real deployed copilots have many tools and combinations, making exhaustive prompt hand-holding impractical.

“Real, deployed copilot systems include dozens of possible tools and a combinatorially high and unwieldy number of combinations, making it impractical to simply engineer system prompts for every edge case.”article, stated

Patterns

`eval-driven-development` (benchmarking)

The benchmark evaluates whether models can select tools, use resources, and solve underwriting tasks.

“### Evaluating Agent Capabilities”article, stated

`tool-use-orchestration` (benchmarking)

Tasks require reasoning across proprietary knowledge and specialized tools.

“* Reasoning over specific and proprietary knowledge that models have never seen before * Tool use in complex, noisy, and specialized ecosystems”article, stated

Tradeoffs

How realistic to make the benchmark.

Chose expert-authored proprietary underwriting tasks over toy or easily verifiable generic tasks.

Rationale: The benchmark is meant to expose domain-specific tool-use and hallucination errors that generic checks miss.

“meaningful real-world settings—rather than toy scenarios”article, stated

Gotchas (what bit them)

Pretrained model knowledge can hallucinate non-existent domain products.

Root cause: Models may answer from memorized general insurance knowledge instead of tools.

Resolution: The benchmark exposes hallucination and tool-use error modes.

“to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.”summary, stated

Evaluation

The benchmark measures agent capabilities across tool choice, tool use, and problem solving, with expert-created data.

Metrics: tool-use failures · accuracy · hallucination rate

“* Asking the right questions of the underwriter * Determining which tools and resources were relevant to a given task”article, stated

Outcomes

The summary reports model accuracy ranging from single digits to about 80% and 36% tool-use failure conversations.

This is a benchmark/dataset result rather than a deployed underwriting copilot.

“tier models (single digits to ~80% accuracy), with notable error modes including tool use failures (36% of conversations) and hallucinations from pretrained domain knowledge, particularly from OpenAI models which hallucinated non-existent insurance products 15-45% of the time.”summary, stated