Using LLMs to Scale Insurance Operations at a Small Company

anzen · Insurance · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Small-company insurance operations

Anzen needed to automate insurance underwriting intake and compliance document review with a sub-20-person team competing against larger insurance incumbents.

Constraints:

• Insurance applications arrived through email or fax in many dense, inconsistent formats.
• The team had limited data, capital, and headcount.
• Underwriting workflows still required human final decisions.
• Compliance review needed high accuracy to avoid misleading users.
• Evaluation at the company's scale was largely manual.

Architecture

Anzen built two narrow AI pipelines. For underwriting intake, a BERT classifier identifies insurance applications from inbox attachments, duplicate detection filters repeats, AWS Textract question-answering extracts fields, and underwriters receive alerts for final decisions. For compliance review, attorney-defined features guide sentence-embedding search and QA extraction over documents such as offer letters and separation agreements, with business rules generating positive and negative compliance insights.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  team-inbox["Team inbox"] --> bert-application-classifier["BERT application classifier"]
  bert-application-classifier --> duplicate-detection["Duplicate detection"]
  duplicate-detection --> textract-qa-extraction["Textract QA extraction"]
  textract-qa-extraction --> underwriting-alert["Underwriting alert"]
  compliance-feature-definitions["Compliance feature definitions"] --> embedding-search-and-qa["Embedding search and QA"]
  embedding-search-and-qa --> compliance-business-rules["Compliance business rules"]
  compliance-business-rules --> manual-review-channel["Manual review channel"]

Components

Data flow

Insurance attachments enter the inbox, a BERT classifier detects applications, duplicate logic removes repeats, Textract extracts key fields, and the underwriting team receives an alert. In the compliance feature, attorneys define document features, embedding search finds relevant passages, QA extracts values, business logic applies compliance rules, and outputs are manually monitored in Slack as the system matures.

Why this architecture

Anzen chose narrow, task-specific models and extraction pipelines because a small team could get high leverage from focused automation without taking on the reliability risks of broad generative agents. Humans stay in the loop for underwriting and manual evaluation while the system removes intake and extraction labor.

Models used: BERT · AWS Textract question-answering · MiniLM · Sentence embeddings · Question-answering models

Scale

• Anzen had fewer than 20 employees.
  “Anzen is a small insurtech startup with fewer than 20 employees”summary, stated
• Employment-related lawsuits were framed as a $23 billion annual problem.
  “tackles the $23 billion annual problem of employment-related lawsuits”summary, stated
• The presenter had approximately 18 months of company experience.
  “shared insights from approximately 18 months of experience at the company”summary, stated
• The application classifier used approximately 300 manually labeled attachments.
  “Pulled approximately 300 attachments from their team inbox”summary, stated
• The classifier reached 95% accuracy, 90% precision, and 100% recall.
  “Trained a classifier achieving 95% accuracy, 90% precision, and 100% recall”summary, stated
• The underwriting classifier was built end-to-end in under a week.
  “was accomplished in under a week end-to-end.”summary, stated

Alternatives considered (roads not taken)

Build a large manual underwriting intake team

Rejected because: A small team could automate intake and extraction instead of staffing manual preprocessing.

Optimize precision over recall for application detection

Rejected because: Anzen accepted some noise to avoid missing applications.

Use the largest model for semantic search

Rejected because: MiniLM outperformed larger models for the compliance semantic-search component.

Patterns

fine-tuning-adaptation (classification)

Anzen adapted pretrained BERT with a small labeled attachment set to detect insurance applications.

human-in-the-loop-gate (operations)

Underwriters make final decisions and Slack review catches model errors while automation handles intake and extraction.

eval-driven-development (monitoring)

The team uses manual Slack review, flagged errors, accumulated feedback, and the original labeled dataset as a regression baseline.

safety-policy-enforcement (compliance)

Compliance review uses attorney-provided rules and avoids bogus outputs in legal-risk contexts.

Tradeoffs

Recall-optimized intake classifier

Chose Accept some false positives over Risk missing insurance applications.

Rationale: Missing an application is worse than letting a few non-applications reach downstream review.

Narrow model composition

Chose Combine BERT, Textract QA, MiniLM, embeddings, and business rules over Use one broad generative model for the entire workflow.

Rationale: Specific components solved specific workflow steps with more control and lower risk.

Manual evaluation at early scale

Chose Slack review and feedback accumulation over Fully automated evaluation infrastructure upfront.

Rationale: The team balanced ideal LLMOps against small-startup resource constraints.

Gotchas (what bit them)

Small teams may need manual evaluation before automated eval infrastructure is justified.

Root cause: At Anzen's scale, building full automated evaluation was not yet the top constraint.

Resolution: Outputs flow to Slack, humans flag mistakes, and the labeled dataset acts as a regression baseline.

External API iteration costs can become prohibitive.

Root cause: Prompt and business-logic experiments require repeated calls to paid APIs.

Resolution: Teams need to factor repeated experimentation costs into LLM architecture choices.

LLMs can degrade systems when used where simpler models suffice.

Root cause: They are slower, less deterministic, and may not outperform narrow traditional models.

Resolution: Anzen favored pretrained classifiers, embeddings, and QA for well-scoped tasks.

Evaluation

Anzen measured the underwriting classifier with labeled attachment metrics, uses manual Slack review for production monitoring, and treats the original labeled dataset as a regression baseline.

Metrics: 95% classifier accuracy · 90% precision · 100% recall · 300 labeled attachments · Manual Slack review

Outcomes

• The intake classifier reached 95% accuracy, 90% precision, and 100% recall from about 300 labels.
• The underwriting intake prototype was completed end-to-end in under a week.
• Anzen replaced what would normally require a dedicated intake team with an automated pipeline plus human final decision.
• The compliance review prototype was also built in approximately one week with positive early feedback.

A sub-20-person startup automated parts of underwriting and compliance review for a $23B employment-lawsuit insurance problem using hundreds, not thousands, of labeled examples.