Automating Radiology Report Generation with Fine-tuned LLMs

heidelberg_university · Healthcare · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Radiology report generation

Heidelberg University researchers wanted to generate detailed radiology reports from radiologist impressions and medical imaging context while reducing radiologist workload and fitting training onto limited GPU resources.

Constraints:

• radiology report generation must preserve medical detail and recommendations
• detailed reports average 1,000-5,000 tokens
• training had to run on a single RTX 5000 GPU
• quantization can reduce precision and accuracy
• human radiologist evaluation remains necessary

Architecture

The research pipeline frames the radiologist impression as an instruction and the detailed report as the response using the Alpaca prompt format for Llama. The team fine-tuned Llama 3 with supervised fine-tuning, 4-bit quantization, LoRA-style parameter-efficient fine-tuning, a 5K-token sequence length, and Hugging Face tooling on a single RTX 5000 GPU. Evaluation used training/validation loss, BLEU scores, and human review by senior radiologists.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  radiologist-impression-input[Radiologist impression input] --> alpaca-prompt-formatter[Alpaca prompt formatter]
  alpaca-prompt-formatter --> fine-tuned-llama-3-model[Fine-tuned Llama 3 model]
  quantization-and-lora-stack[Quantization and LoRA stack] --> fine-tuned-llama-3-model
  fine-tuned-llama-3-model --> evaluation-harness[Evaluation harness]

Components

Data flow

Radiologist impressions are formatted as Alpaca instructions, paired with detailed reports as responses, and used for supervised fine-tuning. Quantization reduces model size, LoRA updates a small parameter subset, and the fine-tuned Llama 3 model generates detailed report text that is evaluated with loss, BLEU, and radiologist review.

Why this architecture

The team chose instruction fine-tuning from impressions to detailed reports because they wanted the model to expand a concise radiologist summary into a full report, not summarize a report. Quantization and LoRA were necessary to make Llama 3 fine-tuning practical on a single RTX 5000 GPU, with an explicit tradeoff between efficiency and possible accuracy loss.

Models used: Llama 3 · 4-bit quantized Llama 3 · LoRA parameter-efficient fine-tuning

Scale

• Radiologist impressions are about 500 to 1,000 tokens.
  “minimal text of about 500 to th000 tokens then we have radiologist report which is analysis of the 3D image with”transcript, stated
• Detailed radiology reports average 1,000 to 5,000 tokens.
  “recommendations this could be between 1,000 to 5,000 tokens on average uh for more on generating uh the detailed report from”transcript, stated
• The model used a maximum sequence length around 5K tokens.
  “maximum sequence length as 5 5K tokens as on average our reports have less than 5,000 tokens we trained the model using”transcript, stated
• Training ran on an RTX 5000 GPU for 2,000 epochs.
  “5,000 tokens we trained the model using RTX 5000 GPU uh for 2,000 EPO we used supervised fine tuning trainer from”transcript, stated

Alternatives considered (roads not taken)

Generate summary impressions from detailed reports

Rejected because: The researchers explicitly tried the opposite direction: detailed report from impression.

Full-precision fine-tuning

Rejected because: Quantization reduced model size, inference time, training cost, and power consumption.

Fine-tune all parameters

Rejected because: Parameter-efficient fine-tuning reduced compute burden while preserving model capacity.

Patterns

fine-tuning-adaptation (modeling)

The team fine-tuned Llama 3 on radiology impression-to-report examples.

structured-output-contract (prompting)

The Alpaca format structures each training example as instruction and response.

cost-aware-routing (operations)

Quantization and LoRA reduce compute, memory, and training cost.

eval-driven-development (evaluation)

The team reported training/validation loss, BLEU scores, and human radiologist evaluation.

Tradeoffs

Use impressions as input

Chose generate detailed reports from radiologist impressions over generate impressions from full reports.

Rationale: The team wanted to avoid overloading the architecture and target detailed-report generation.

Use PEFT/LoRA

Chose train low-rank matrices while freezing original weights over full model updates.

Rationale: LoRA reduced trainable parameters and compute burden while preserving capacity.

Gotchas (what bit them)

Quantization trades efficiency against accuracy.

Root cause: Reducing numerical precision can slightly reduce model accuracy.

Resolution: The team used 4-bit quantization while acknowledging the tradeoff.

BLEU is an incomplete radiology quality metric.

Root cause: BLEU only compares whether words appear in the generated text.

Resolution: The case also mentions human evaluation by senior radiologists.

Evaluation

The research evaluates with training loss, validation loss, BLEU on training and validation data, and human senior-radiologist evaluation.

Metrics: training loss · validation loss · training BLEU · validation BLEU · human radiologist review

Outcomes

• The team trained a fine-tuned Llama 3 model on a single RTX 5000 GPU.
• Reported training loss was 0.72 and validation loss was 1.36.
• Reported BLEU was 0.69 on training data and 0.33 on validation data.
• The source frames BLEU as limited and points to radiologist evaluation.

Reports average up to 5,000 tokens, and the setup uses a 5K-token sequence length on a single GPU.