Building Production LLM Pipelines for Insurance Risk Assessment and Document Processing

vouch · Insurance · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance underwriting risk classification and document AI pipelines

Vouch needed production LLM pipelines that could classify underwriting risk, extract useful information from insurance documents and web sources, serve predictions to product engineers, report prediction quality back to the warehouse, and control LLM cost and development friction in a modern data-stack environment.

Constraints:

• Insurance underwriting depends on assessing risks and appetite to insure, so predictions must fit business workflows rather than exist as isolated model outputs.
• The source data includes structured and semi-structured numeric/text data plus PDFs and other documents.
• LLM responses often need post-processing because output parsers can fail or return imperfect structure.
• Risk-classification workflows run hourly, while document AI workflows run when documents arrive.
• LLM calls are expensive, so the pipeline must avoid redundant predictions and manage token usage.
• Developers work across different local machines and CPU architectures, creating local-development reproducibility problems.

Architecture

Vouch built Metaflow-centered LLM pipelines on AWS Batch. DBT prepares data, Metaflow orchestrates transformations and LLM calls through LangChain/OpenAI, post-processing steps repair or enforce structure when output parsers fail, predictions are written to PostgreSQL, a FastAPI service exposes them, a custom SDK lets product engineers retrieve predictions with a few lines of code, and reverse ETL sends predictions back into Snowflake for reporting. AWS Cognito secures the ALB-level sign-in flow, Docker Compose gives pipeline developers a local service stack, hourly risk-classification jobs poll for new work, and document AI jobs run when documents hit services.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  insurance-data-sources[Insurance data sources] --> dbt-transformations[DBT transformations]
  dbt-transformations --> metaflow-pipelines[Metaflow pipelines]
  metaflow-pipelines --> langchain-openai-calls[LangChain OpenAI calls]
  prediction-cache[Prediction cache] --> langchain-openai-calls
  langchain-openai-calls --> post-processing-steps[Post-processing steps]
  post-processing-steps --> postgresql-predictions[PostgreSQL predictions]
  postgresql-predictions --> fastapi-service[FastAPI service]
  fastapi-service --> product-sdk[Product SDK]
  product-sdk --> product-applications[Product applications]
  postgresql-predictions --> snowflake-reporting[Snowflake reporting]
  metaflow-pipelines --> aws-batch-runtime[AWS Batch runtime]
  aws-batch-runtime --> aws-cognito-alb-authentication[AWS Cognito ALB authentication]
  docker-compose-dev-stack[Docker Compose dev stack] --> metaflow-pipelines
  docker-compose-dev-stack --> fastapi-service

Components

Data flow

Metaflow starts by preparing data transformations that are run through DBT. Prepared data is sent to LLM providers through LangChain, initially OpenAI, to generate risk or document predictions. Returned LLM outputs pass through Metaflow post-processing to enforce transformations when parsers fail. Final predictions are written to PostgreSQL, served through FastAPI, retrieved by product engineers through a custom SDK, and reverse-ETL'd into Snowflake for reporting on prediction quality. Risk pipelines run hourly and check for new data; document AI pipelines trigger as needed when documents arrive.

Why this architecture

The architecture matches Vouch's modern data stack and product-integration needs: Metaflow fits the pipeline orchestration layer, DBT and Snowflake fit existing analytics workflows, FastAPI and the SDK expose predictions to engineers, and AWS Batch gave a fast production path. The team also wrapped LLM calls with post-processing and caching because raw LLM outputs and repeated calls are not production-safe enough on their own.

Models used: Metaflow · AWS Batch · DBT · LangChain · OpenAI APIs · FastAPI · PostgreSQL · Snowflake · AWS Cognito · Docker Compose · Kubernetes

Scale

• Risk-classification pipelines run every hour.
  “for our risk classification use cases we can run those pipelines every hour”transcript, stated
• Document AI workflows run when documents hit services.
  “for the document AI workflows those will be kind of on an as needed basis or as soon as the documents hit our services then we kick out the process”transcript, stated
• Feature-engineering tables are at terabytes or hundreds of terabytes scale.
  “we're talking terabytes or hundreds of terabytes maybe in like a set of interested in um feature engineering”transcript, stated
• The source data includes PDFs and non-PDF documents in addition to structured and semi-structured data.
  “structured and semi-structured numeric and Text data but we also have again those documents usually PDFs but not always”transcript, stated
• SDK consumers retrieve predictions with only a couple lines of code.
  “retrieve those predictions with a couple lines of code”transcript, stated

Alternatives considered (roads not taken)

Polling-style hourly Metaflow pipelines

Rejected because: The team says the current hourly cadence works, but as the project matures they probably need event-driven pipelines.

Relying on raw LLM parser output

Rejected because: The team found LLM responses often need work, so they use Metaflow transformations to enforce expected structure when parsers fail.

Let every consumer run the full LLM project locally

Rejected because: SDK users point at the dev instance instead of running pipelines, API, and databases locally.

Native local installs across many developer machines

Rejected because: Mac/architecture differences caused enough friction that the team containerized Metaflow for local development.

Patterns

batch-offline-processing (execution)

Risk classification runs as scheduled Metaflow pipelines that check hourly for new data to process.

structured-output-contract (reliability)

The pipeline treats LLM responses as intermediate outputs that must be transformed into the expected prediction structure before being written and served.

semantic-caching (cost-control)

Before making an LLM call, the system checks whether it already has a prediction, reducing duplicate work and spend.

cost-aware-routing (operations)

The team controls LLM spend by caching predictions, narrowing the amount of work, and managing tokens on LLM steps.

observability-tracing (quality-monitoring)

Predictions are reverse-ETL'd back into the warehouse so the team can report on how well the LLM predictions are working.

guardrails-io-filtering (output-repair)

Post-processing steps act as an output guardrail by enforcing transformations after parser failures instead of trusting raw LLM responses.

Tradeoffs

Use AWS Batch Metaflow infrastructure first

Chose Extending the AWS Batch Terraform template over Building a bespoke production workflow platform.

Rationale: The template got the team into production quickly, and the team extended it with Cognito/ALB authentication rather than replacing the foundation.

Serve predictions through FastAPI and an SDK

Chose A prediction service plus SDK abstraction over Requiring product engineers to know the pipeline internals.

Rationale: The SDK lets product engineers retrieve predictions with a couple lines of code while hiding Metaflow and database complexity.

Run risk predictions hourly

Chose Hourly polling/checking for new work over Immediate event-driven processing for risk classification.

Rationale: Hourly execution fits the current business volume, but the team recognizes event-driven pipelines may be better as the system matures.

Containerize local development

Chose Docker Compose running the pipeline, API, and databases over Each developer configuring a native local stack.

Rationale: The team accepted some container quirks to insulate developers from platform-specific Mac/CPU issues.

Gotchas (what bit them)

LLM responses are not reliably ready to write directly to production stores.

Root cause: Output parsers can fail or not entirely work as expected, leaving responses that still need transformation.

Resolution: Run additional Metaflow post-processing transformations before writing predictions to the database.

The initial polling/scheduled architecture may not remain ideal as the project matures.

Root cause: Hourly polling works at current volume, but document arrivals and pipeline maturity point toward event-driven execution.

Resolution: Plan to adopt event-driven Metaflow/Kubernetes patterns once documentation and implementation gaps are addressed.

Local development across developer machines caused enough friction to change the dev environment.

Root cause: Mac/AMD and cross-machine differences made running Metaflow locally inconsistent.

Resolution: Put Metaflow inside a container and use Docker Compose to spin up pipelines, API, and databases locally.

LLM API cost can grow from avoidable duplicate work.

Root cause: External LLM calls are expensive, and naive pipelines may re-run predictions that already exist.

Resolution: Check for existing predictions and narrow work before making LLM calls; manage tokens and timeouts on LLM steps.

Evaluation

The source describes reporting on prediction quality through reverse ETL into the data warehouse, operational checks for existing predictions before new LLM calls, hourly workflow cadence monitoring, and developer feedback from SDK users. It does not describe a formal offline evaluation harness.

Metrics: Prediction quality/performance reporting in the data warehouse · Hourly risk-classification cadence · Existing-prediction checks before LLM calls · LLM cost/token-management checks · Developer usability through SDK usage

Outcomes

• Vouch has LLM technologies in production for classification and document AI.
• Risk-classification pipelines run hourly without overwhelming the pipeline at current cadence.
• Document AI workflows trigger when documents hit services.
• Predictions are served through FastAPI and consumed through a custom SDK.
• Reverse ETL into the warehouse supports reporting on prediction quality.
• Docker Compose gives pipeline developers a reproducible local service stack.

The system operates on hourly risk-classification workflows, on-demand document AI, terabytes or hundreds of terabytes of relevant feature-engineering data, structured/semi-structured data plus PDFs, and SDK integration that requires only a couple lines of product code.