Medical Transcript Summarization Using Multiple LLM Models: An Evaluation Study

oracle · Healthcare · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Medical transcript summarization evaluation

Healthcare care coordinators spend substantial preparation time reviewing medical transcripts, but summarization models must preserve critical clinical information, avoid hallucinations, and operate under healthcare privacy constraints.

Constraints:

• Medical data access is constrained by HIPAA privacy requirements.
• Medical language includes jargon, abbreviations, specialty terminology, and regional variation.
• Summaries must balance conciseness with retention of diagnoses, treatment plans, and history.
• Hallucinated details can create misinformation in a high-stakes clinical domain.
• The study used sample transcripts rather than live patient records.

Architecture

Oracle's presentation evaluated multiple LLMs for medical transcript summarization. A public MT Samples transcript dataset was used to avoid patient-privacy exposure; prompts extracted structured patient-summary fields; Claude outputs were manually evaluated and used as a reference baseline; GPT-4, Llama 3.1, and Phi 3.1 summaries were compared with ROUGE and cosine-similarity metrics. The study surfaced deployment tradeoffs between accuracy, conciseness, latency, and edge-device feasibility.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Dataset["MT Samples dataset"] --> Workflow["Prompted summarization workflow"]
  Workflow --> Claude["Claude baseline"]
  Workflow --> GPT["GPT-4 model"]
  Workflow --> Llama["Llama 3.1 model"]
  Workflow --> Phi["Phi 3.1 model"]
  Claude --> Metrics["Evaluation metrics"]
  GPT --> Metrics
  Llama --> Metrics
  Phi --> Metrics

Components

Data flow

Medical transcript samples are scraped from MT Samples and passed through structured prompts for multiple LLMs. Claude-generated summaries serve as the reference set after manual review. Other model outputs are compared against that reference with ROUGE-1, ROUGE-2, ROUGE-L, cosine similarity, and qualitative assessment of extracted patient fields and hallucinations.

Why this architecture

The study used a comparative evaluation architecture because model choice depends on deployment constraints: GPT-4 achieved stronger quantitative performance, Phi 3.1 offered a lightweight accuracy/conciseness balance for mobile devices, and Claude provided the manually reviewed reference baseline. Public sample data reduced privacy risk while still allowing controlled model comparison.

Models used: Claude · OpenAI GPT-4 · Llama 3.1 · Phi 3.1

Scale

• The work targeted over 50% reduction in care-coordinator preparation time.
  “our work demonstrates the potential to cut down the time spent by care coordinates to preparing for patient checked in by over 50%”transcript, stated
• The MT Samples dataset contained around 5,000 transcripts.
  “the data set has around 5,000 transcript I was playing around uh 5,000 different specialty uh 5,000 transcript from the different speciality”transcript, stated
• The quantitative comparison used around 500+ medical transcripts.
  “yeah think overall I extracted think around 500 plus uh medical transcripts uh among the four close to 5,000 transcripts”transcript, stated
• GPT-4 scored ROUGE-1 0.821, ROUGE-2 0.7, ROUGE-L 0.76, and cosine similarity 0.879.
  “the Rogue is 0.821 Rogue 2 is 0.7 um Rog L is 0.76 coine similarity is close to 0.879”transcript, stated
• Phi 3.1 achieved cosine similarity of 0.819.
  “the cosine similarity score uh 0.819 shows that the uh the p3.1 is quite effective at generating summaries”transcript, stated

Alternatives considered (roads not taken)

Use real patient records for evaluation

Rejected because: Healthcare data access is constrained by HIPAA, so the study used public sample transcripts.

Choose GPT-4 solely for highest accuracy

Rejected because: Phi 3.1 may be preferable when concise summaries, latency, or edge deployment matter.

Use model outputs without clinician oversight

Rejected because: The presenter identified human-in-the-loop validation as a future safety direction for medical summarization.

Patterns

eval-driven-development (evaluation)

The study compares model outputs with ROUGE, cosine similarity, and qualitative assessment before making deployment recommendations.

structured-output-contract (output_control)

Prompts ask models to extract medical transcript content into consistent fields such as symptoms, histories, medications, diagnostics, assessment, and follow-up instructions.

human-in-the-loop-gate (control)

The presenter recommends clinician involvement to guide and validate model outputs for safer medical summarization.

safety-policy-enforcement (control)

The study treats hallucinations, omitted clinical details, and medical-context misinterpretation as safety risks that must constrain deployment choices.

Tradeoffs

Prioritize GPT-4 for comprehensive accuracy

Chose GPT-4 as the strongest quantitative performer over Lower-scoring alternatives for maximum accuracy cases.

Rationale: GPT-4 led ROUGE and cosine metrics and was judged best for comprehensive narrative summaries.

Consider Phi 3.1 for constrained deployment

Chose Phi 3.1 for concise, lightweight summarization scenarios over Always using the most accurate large model.

Rationale: Phi 3.1 scored below GPT-4 but offered a better balance for short summaries and device deployment.

Use public sample data for research

Chose MT Samples transcripts over Real patient transcripts.

Rationale: Public sample data enabled evaluation without violating patient privacy, but it limits confidence about real-world clinical generalization.

Gotchas (what bit them)

Using Claude as ground truth creates circularity risk.

Root cause: The study compares other model outputs to Claude summaries rather than clinician-authored references.

Resolution: The presenter says Claude was manually evaluated and performed well, but future clinical deployment should add human validation.

Medical summarization cannot safely optimize only for brevity.

Root cause: Critical diagnosis, treatment, and history details may be omitted if summaries become too concise.

Resolution: The evaluation considers accuracy, completeness, semantic similarity, and future clinician validation rather than length alone.

Model hallucinations appeared in medical follow-up details.

Root cause: Some models generated information absent from the transcript context.

Resolution: The presenter highlights hallucination as a model limitation and points to domain knowledge and human-in-the-loop validation as future work.

Public sample transcripts may not represent clinical production data.

Root cause: Privacy constraints pushed the evaluation toward MT Samples rather than real patient records.

Resolution: The work frames the dataset as a privacy-preserving research resource and identifies larger/diverse medical datasets as future improvement.

Evaluation

The study used Claude-generated summaries as a manually reviewed baseline, then compared GPT-4, Llama 3.1, and Phi 3.1 with ROUGE-1, ROUGE-2, ROUGE-L, cosine similarity, and qualitative inspection of structured medical-summary fields.

Metrics: ROUGE-1 · ROUGE-2 · ROUGE-L · cosine similarity · qualitative assessment · hallucination examples · accuracy versus conciseness tradeoff

Outcomes

• GPT-4 led across reported ROUGE and cosine-similarity metrics.
• Phi 3.1 was identified as a strong lightweight option for concise summaries and potential iPad-style edge deployment.
• Llama 3.1 underperformed GPT-4 and Phi 3.1 on the reported metrics and showed hallucination examples.
• The study estimated potential care-coordinator preparation-time reduction above 50%.
• Future work calls for medical fine-tuning, domain-specific knowledge integration, and clinician-in-the-loop validation.

The transcript reports a 5,000-transcript MT Samples dataset, about 500+ evaluated medical transcripts, GPT-4 scores of ROUGE-1 0.821 and cosine 0.879, Phi 3.1 cosine 0.819, and a potential over-50% reduction in care-coordinator preparation time.