Enterprise-Wide LLM Assistant Deployment and Evolution Towards Fine-Tuned Models

marsh_mclennan · Insurance · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise productivity assistant and task automation

Marsh McLennan needed to make generative AI broadly available across a global professional-services organization while keeping experimentation cheap, avoiding premature infrastructure complexity, and later improving task-specific automation accuracy and cost through fine-tuned smaller models.

Constraints:

• The organization has roughly 90,000 people globally and diverse work that is hard to capture through a central use-case sweep.
• Early use cases had diffuse productivity value rather than single multimillion-dollar business cases.
• RAG preserves normal data-access controls, while fine-tuning can embed access-sensitive data in model weights.
• Dedicated fine-tuned model infrastructure per use case would break the economics of small automation opportunities.
• The accuracy threshold for full task replacement is much higher than for augmentation.

Architecture

Marsh McLennan made LLM APIs and then an organization-wide assistant available so colleagues could experiment cheaply by renting model calls instead of hosting large models. The assistant handles productivity augmentation and surrounding helper applications, while trends from broad usage inform enabling capabilities. For more specific automation, the company moved toward LoRA-style fine-tuned adapters via Predibase, sharing infrastructure across adapters so task-specific models can improve accuracy and cost without one infrastructure stack per use case.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  secured-llm-apis["Secured LLM APIs"] --> global-llm-assistant["Global LLM assistant"]
  global-llm-assistant --> productivity-suite-integrations["Productivity suite integrations"]
  global-llm-assistant --> rag-data-access-layer["RAG data access layer"]
  rag-data-access-layer --> generic-model-calls["Generic model calls"]
  generic-model-calls --> automation-candidate-pipeline["Automation candidate pipeline"]
  automation-candidate-pipeline --> lora-fine-tuned-adapters["LoRA fine-tuned adapters"]
  lora-fine-tuned-adapters --> specialized-task-models["Specialized task models"]

Components

Data flow

Employees use the global assistant and related helper applications for diffuse productivity tasks. Usage volume and task trends reveal where automation can go deeper. For those processes, teams start with LLMs, prompting, and RAG to get value quickly, collect data about what is accurate, and then feed that data into fine-tuning so smaller specialized models can replace generic calls where cost and accuracy warrant it.

Why this architecture

The architecture starts broad and cheap because a use-case sweep would force fuzzy productivity ideas into heavyweight project business cases. Rented model calls and a global assistant let employees find value. RAG keeps data access manageable. Once repeatable task patterns appear, LoRA adapters make specialization economical by sharing infrastructure and avoiding a separate model stack per small use case.

Models used: LLMs · RAG · LoRA · Predibase · Fine-tuned small models · GPT-4

Scale

• Marsh McLennan has about 90,000 people globally and operates in over 130 countries.
  “we're an organization of about 90,000 people globally we operate in over 130 countries”transcript, stated
• The technology organization has about 5,000 people.
  “we have about 5,000 people in that organization around the world”transcript, stated
• The organization-wide assistant runs about 25 million requests per year.
  “at the moment we run at about 25 million requests a year um through that tool”transcript, stated
• The speaker conservatively estimates at least one million hours saved per year.
  “we think we've saved and this is a very conservative assumption in my view at least a million hours a year”transcript, stated
• Some LoRA training cycles cost about $20.
  “we're ending up spending I know 20 bucks to run a training cycle on some of this sort of stuff”transcript, stated
• The small fine-tuned model path was handling about half a million requests per week.
  “our main tool are now run through um the small fine tune model so we're putting in probably half a million requests a week now”transcript, stated

Alternatives considered (roads not taken)

Start with a traditional enterprise use-case sweep.

Rejected because: It would push fuzzy diffuse productivity opportunities into heavyweight steering-committee and business-case processes too early.

Host large language models internally from the start.

Rejected because: Renting model calls made experimentation cheap and avoided unnecessary infrastructure effort.

Fine-tune early for broad productivity use cases.

Rejected because: Prompting and RAG had lower-hanging fruit, while fine-tuning raised cost and data-governance complexity.

Partition fine-tuned models for every access group.

Rejected because: That would multiply infrastructure and maintenance overhead.

Patterns

cost-aware-routing (model economics)

The company began by renting model calls for broad experimentation, then moved repeatable high-volume subtasks toward smaller fine-tuned models to cut cost.

fine-tuning-adaptation (task specialization)

LoRA-based fine-tuning lets Marsh McLennan specialize models for subtasks without one infrastructure stack per use case.

rag-retrieval (data access)

RAG is used to bring data to models while preserving existing access-control patterns.

data-quality-loop (fine-tuning flywheel)

Teams collect accuracy data from initial LLM/prompting workflows and feed it into later fine-tuning.

Tradeoffs

Bias near-term GenAI toward augmentation.

Chose Assistant-led productivity support over Immediate full job replacement.

Rationale: The accuracy bar for replacing work is much higher than the bar for making people more efficient.

Rent model usage before hosting models.

Chose Model calls by API over Owning large-language-model hosting infrastructure.

Rationale: API calls kept experimentation cheap while the organization learned where value appeared.

Use prompting/RAG first and fine-tune later.

Chose Fast baseline value followed by task-specific fine-tuning where warranted over Fine-tuning every use case from the start.

Rationale: Prompting/RAG delivered low-hanging fruit; fine-tuning became attractive only once adapters changed the cost/infrastructure equation.

Gotchas (what bit them)

A use-case sweep can make experimentation feel threatening and bureaucratic.

Root cause: Each opportunity becomes a project, steering committee, and ROI discussion before people have learned where value appears.

Resolution: Make experimentation cheap and broad, then build enabling capabilities when usage trends emerge.

Fine-tuned models can complicate data governance.

Root cause: Data trained into model weights may surface later in less controllable ways than RAG-retrieved data.

Resolution: Use RAG where access control matters and reserve fine-tuning for cases with justified economics and governance.

Dedicated infrastructure per fine-tuned use case breaks small-opportunity economics.

Root cause: Many automation opportunities are tens or hundreds of thousands of dollars, not large enough to carry annual infrastructure cost.

Resolution: Use shared LoRA adapter infrastructure to lower the entry cost.

Evaluation

Marsh McLennan evaluates the program through usage volume, estimated hours saved, response time, accuracy relative to GPT-4, unit economics, and whether specialized models improve task-specific automation.

Metrics: Requests per year · Requests per week through fine-tuned model · Estimated hours saved · Training-cycle cost · Response time · Accuracy exceeding GPT-4 · Cost reduction from switching models

Outcomes

• The assistant runs about 25 million requests per year.
• Marsh McLennan estimates at least one million hours saved annually.
• Some LoRA training cycles cost about $20 and produced accuracy described as exceeding GPT-4.

The assistant spans a 90,000-person organization in more than 130 countries, with fine-tuned model traffic already around half a million requests per week.