Enterprise-Scale Deployment of AI Ambient Scribes Across Multiple Healthcare Systems

memorial_sloan_kettering_/mcleod_health/_ucla · Healthcare · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Healthcare ambient clinical documentation

McLeod Health, UCLA Health, and Memorial Sloan Kettering needed to select, evaluate, roll out, and justify AI ambient scribe systems that reduce documentation burden and burnout without compromising clinical workflow, note quality, privacy, ROI, or patient trust.

Constraints:

• AI scribe vendors had to pass security, scalability, usability, and clinical relevance checks.
• Healthcare deployments need training, onboarding, EHR integration, and workflow fit.
• LLM-generated notes can contain omissions or inaccuracies.
• Not every clinician, specialty, or visit type is a good fit for ambient scribes.
• Hospitals needed financial ROI evidence as well as burnout and patient-experience improvements.

Architecture

The systems used structured vendor selection and rollout processes rather than a single technical stack. McLeod filtered vendors, used 15 scripted patient encounters with actors, collected unedited notes, had physicians/RevCycle/patients score notes, then tested Epic usability before selecting Suki. UCLA ran a randomized controlled trial with two intervention groups and one control group before scaling Nabla in monthly waves with mandatory training and support. MSK piloted vendors head-to-head, focused on oncology complexity and EHR workflow, and evaluated Abridge with broader metrics and workflow lessons.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  clinical-encounter-audio["Clinical encounter audio"] --> ai-scribe-vendor-system["AI scribe vendor system"]
  ai-scribe-vendor-system --> ehr-workflow["EHR workflow"]
  ai-scribe-vendor-system --> clinician-review["Clinician review"]
  ehr-workflow --> clinician-review["Clinician review"]
  clinician-review --> evaluation-program["Evaluation program"]
  evaluation-program --> training-and-support["Training and support"]
  training-and-support --> clinician-adoption["Clinician adoption"]

Components

Data flow

Clinical encounters are captured by ambient scribe tools, converted into draft notes or visit summaries, reviewed in clinical workflow/EHR context, and evaluated through note scoring, surveys, adoption/usage metrics, burnout instruments, financial measures, patient satisfaction, and technical monitoring for errors such as hallucinations.

Why this architecture

The organizations used rigorous evaluation and staged rollout because ambient scribes are expensive, clinically sensitive, workflow-dependent, and not universally appropriate; vendor choice and adoption depend as much on clinical fit as model capability.

Models used: Suki · Nabla · Abridge · AI scribes · LLMs · Epic · Haiku

Scale

• McLeod Health is a seven-hospital system in South Carolina.
  “McLeod Health, which is a 7-hospital system in South Carolina.”transcript, stated
• UCLA Health has about 5,000 ambulatory physicians.
  “We have about 5,000 ambulatory physicians”transcript, stated
• McLeod scaled from an initial pilot of 35 doctors to around 150.
  “scaling from our initial pilot of 35 doctors up to around 150.”transcript, stated
• UCLA continued with about 200 pilot physicians and planned another 600 in July.
  “continued on with all of our pilot users, which was about 200 physicians.”transcript, stated
• UCLA saw about a 20% reduction in burnout prevalence in its RCT.
  “In our RCT, we saw about a 20% reduction in burnout prevalence.”transcript, stated
• McLeod reported 6.3% and 5.9-point improvements on two patient-satisfaction questions.
  “That metric went up by 6.3%. And the other question. Did you trust the provider with your care went up by 5.9.”transcript, stated

Alternatives considered (roads not taken)

Pick the expected or popular AI scribe vendor.

Rejected because: McLeod's structured testing overturned the presenter's initial bias.

Evaluate vendors without randomized controls.

Rejected because: UCLA chose a randomized controlled trial to get actionable decision data.

Use monthly per-provider pricing.

Rejected because: Encounter-based pricing aligned vendor incentives and enabled lower-use clinicians to participate.

Deploy without workflow mapping.

Rejected because: MSK wished it had mapped workflows earlier to tailor rollout by specialty and visit type.

Patterns

eval-driven-development (evaluation)

Healthcare systems used scripted scenarios, RCTs, head-to-head pilots, surveys, and ROI metrics before scaling.

human-in-the-loop-gate (governance)

Clinicians review and use AI-generated notes as a support tool rather than an autonomous clinical authority.

safety-policy-enforcement (safety)

Training explicitly covers LLM imperfections such as omissions and inaccuracies.

observability-tracing (operations)

Rollouts track adoption, utilization, qualitative/quantitative outcomes, ROI, burnout, EHR time, and hallucination feedback.

Tradeoffs

Run rigorous vendor evaluation before selection.

Chose Scripted encounters, head-to-head pilots, RCTs, and stakeholder review. over Selecting based on market reputation or initial assumptions..

Rationale: Clinical note quality, workflow fit, and vendor support varied in non-obvious ways.

Scale through waves with training and support.

Chose Monthly expansion, mandatory training, super users, and elbow-to-elbow support. over A single broad rollout after pilot completion..

Rationale: The organizations needed adoption without ignoring LLM limitations or clinician workflow differences.

Prioritize tech stack and future workflow integration.

Chose A vendor whose stack could move beyond commodity notes toward clinical decision support and conversational AI. over Choosing only on current note output..

Rationale: Switching costs are significant, and scribe notes may become commoditized.

Gotchas (what bit them)

Vendor preference bias can mislead selection.

Root cause: Leaders may assume a favored vendor is best before structured testing.

Resolution: Use scripted encounters, unedited notes, and multi-stakeholder scoring.

Ambient scribes do not fit every clinician or specialty.

Root cause: Some clinicians prefer paper/Dragon or have workflows where the tool is not useful.

Resolution: Treat non-adoption as acceptable and learn fit by patient, visit, clinician, and specialty type.

Simple-looking scribe tools require deep training.

Root cause: The note appears magically, but clinicians must learn how to speak/use it and handle LLM error modes.

Resolution: Make training mandatory and add super-user and elbow-to-elbow support.

Pricing can block scaling to clinicians who use the tool intermittently.

Root cause: Per-provider monthly licenses make low-frequency but valuable use cases unattractive.

Resolution: Use encounter-based pricing with a cap to align vendor and hospital incentives.

Evaluation

The panel describes evaluation through scripted encounters, unedited vendor notes, physician/RevCycle/patient review, RCTs, pilots, Mini Z, NASA TLX, burnout prevalence, work RVUs, documentation time, level of service, EHR time, hallucination feedback, adoption/utilization, patient satisfaction, and financial ROI.

Metrics: 15 scripted encounters · Top four vendors · 35 to 150 McLeod doctors · About 200 UCLA pilot physicians · 20% burnout reduction · 30-40% encounter usage · 6.3% patient-satisfaction increase · $1,000 per provider per month net after subscription

Outcomes

• McLeod selected Suki and scaled from 35 pilot doctors toward around 150.
• UCLA selected Nabla and saw about 20% burnout reduction in its RCT.
• MSK selected Abridge and emphasized oncology workflow complexity and future workflow mapping.
• McLeod reported patient-satisfaction gains and about $1,000/provider/month net after subscription cost in one ROI analysis.

The case spans McLeod's seven-hospital system, UCLA's roughly 5,000 ambulatory physicians, and MSK's oncology-focused deployment/evaluation work.