Automating Weather Forecast Text Generation Using Fine-Tuned Vision-Language Models

uk_metoffice · Government · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Government weather forecasting

The UK Met Office wanted to automate the last-mile drafting of the Shipping Forecast: turning large numerical weather-prediction outputs into concise maritime forecast text that follows strict operational language.

Constraints:

• Forecast text covers 31 sea areas and must follow a specific length, terminology, and format.
• The upstream data is large, multidimensional weather-model output rather than ordinary business text.
• Directional and wording errors can be safety-critical, so semantic near-matches are not good enough.
• The prototype had to compare text-based LLM and vision-language fine-tuning approaches within a short experimentation window.

Architecture

The Met Office/AWS prototype converts numerical weather prediction outputs into Shipping Forecast text through two foundation-model paths: a text-intermediary LLM approach and a vision-language approach over rendered forecast frames. Weather data is reduced from full atmospheric/ocean model output into the Northwest European shelf variables needed for the forecast, staged for training/inference, fine-tuned with Amazon Nova models through SageMaker/Bedrock tooling, and evaluated against meteorologist-written bulletins with strict word-level metrics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  numerical-weather-model-output["numerical-weather-model-output"]
  shipping-forecast-cropper["shipping-forecast-cropper"]
  text-intermediary-pipeline["text-intermediary-pipeline"]
  vision-frame-pipeline["vision-frame-pipeline"]
  nova-pro-llm-path["nova-pro-llm-path"]
  claude-comparison-path["claude-comparison-path"]
  nova-lite-vlm-finetune["nova-lite-vlm-finetune"]
  sagemaker-training-job["sagemaker-training-job"]
  meteorologist-bulletin-evaluator["meteorologist-bulletin-evaluator"]
  numerical-weather-model-output --> shipping-forecast-cropper
  shipping-forecast-cropper --> text-intermediary-pipeline
  shipping-forecast-cropper --> vision-frame-pipeline
  text-intermediary-pipeline --> nova-pro-llm-path
  text-intermediary-pipeline --> claude-comparison-path
  vision-frame-pipeline --> nova-lite-vlm-finetune
  nova-lite-vlm-finetune --> sagemaker-training-job
  nova-pro-llm-path --> meteorologist-bulletin-evaluator
  claude-comparison-path --> meteorologist-bulletin-evaluator
  nova-lite-vlm-finetune --> meteorologist-bulletin-evaluator

Components

Data flow

Numerical weather model output is cropped to the Shipping Forecast variables and region, transformed into either textual summaries or 24-hour visual frames, passed to Nova/Claude model experiments, fine-tuned through SageMaker jobs where appropriate, and evaluated by comparing generated bulletins against expert meteorologist bulletins with word-based F1 rather than embedding-similarity metrics.

Why this architecture

The task is narrow, structured, and safety-sensitive: a Shipping Forecast is short, highly formatted, and semantically brittle because words like north and south have opposite operational meaning. The architecture therefore reduces the meteorological data to task-specific inputs, tests both text and vision representations, uses fine-tuning for a domain-specific format, and evaluates with strict word-level metrics rather than broad semantic similarity.

Models used: Amazon Nova Pro 1.0 · Amazon Nova Lite · Claude 3.7 Sonnet

Scale

• A single full atmospheric forecast run generates upward of 45GB, while ocean output adds about 5GB.
  “we're typically generating upwards of 45 gigabytes of data per individual forecast from the atmospheric models and about five gigabytes of data per individual forecast from our ocean models.”transcript, stated
• Cropping reduces the Shipping Forecast data to about 152MB atmospheric and 7MB ocean data.
  “this results in about 152 megabytes for our atmospheric data and about seven megabytes for our ocean data.”transcript, stated
• The three-month fine-tuning period still represents about 85GB of data.
  “this is about sort of 85 gigabytes of data for three month period.”transcript, stated
• The prototype reduced forecast production from hours-to-a-day for experts to less than five minutes at 52-62% accuracy.
  “it usually takes an expert meteorologist a few hours to a day to produce this forecast. And in just four weeks of prototyping”transcript, stated

  “with between 52 and 62% accuracy in less than five minutes.”transcript, stated
• The experimental SageMaker fine-tuning jobs ran for three hours on a p5.48xlarge with four GPUs.
  “our training jobs were running for three hours on p5.48xlarge GPU, using four GPUs.”transcript, stated
• Nova Pro scored 62% average word F1 versus 57% for Claude 3.7 Sonnet in the LLM comparison.
  “with the Nova Pro scoring an average word F1 score of 62% to Claude 3.7 Sonnet of 57%”transcript, stated

Alternatives considered (roads not taken)

Use embedding/BERT-style semantic scoring for forecasts

Rejected because: Embedding similarity can reward opposite weather directions, making it misleading for operational forecasts.

Rely only on text-intermediary LLM prompting

Rejected because: The team also explored a vision-language path to learn from rendered spatial-temporal weather frames.

Use SageMaker HyperPod for short experiments

Rejected because: The team wanted quick dirty runs, so three-hour experiments used SageMaker training jobs instead of a cluster.

Use full raw forecast data without cropping

Rejected because: The team cropped the forecast to only the variables and area required for the Shipping Forecast to reduce the data burden.

Patterns

structured-output-contract (output-control)

The target forecast text was treated as a tightly constrained product rather than free-form generation, with model outputs expected to match established maritime forecast wording.

fine-tuning-adaptation (model-operations)

The VLM approach converted gridded weather attributes into short videos and fine-tuned Amazon Nova Lite on forecast examples, while the team compared LoRA and full fine-tuning behavior.

eval-driven-development (evaluation)

The team chose word-based F1 over semantic metrics because the operational task requires exact directional and terminology matches, then used that metric to compare model and fine-tuning variants.

batch-offline-processing (architecture)

The proposed architecture emphasizes batch-style processing of large historical weather datasets and forecast runs before calling foundation models.

Tradeoffs

How to represent weather data to a language model.

Chose A text-intermediary LLM path for one approach, and a video-based VLM path for another. over Relying on a single representation for all experiments..

Rationale: The text path gave models summarized domain features, while the VLM path tested whether spatial-temporal patterns could be learned more directly from rendered weather data.

How aggressively to adapt the foundation model.

Chose Full fine-tuning for higher task performance in the experiments. over LoRA-only adaptation..

Rationale: Full fine-tuning scored better on this narrow task, but the team also called out the risk of losing broader model capabilities and the latency/cost implications of alternatives.

How to make the production architecture resilient.

Chose A decoupled processing architecture with SQS and dead-letter handling. over A tightly coupled preprocessing-to-inference chain..

Rationale: The workflow needs to identify failed forecast records and support investigation instead of silently losing work.

Gotchas (what bit them)

Optimizing for generic model-training loss did not align with the operational forecast-quality metric.

Root cause: The task rewards exact meteorological wording, so perplexity and embedded probability can prefer outputs that score worse under word-level F1.

Resolution: The team compared early stopping with overfitting behavior and kept the evaluation focus on word-based F1 against expert bulletins.

Semantic similarity metrics can hide dangerous directional errors.

Root cause: Embedding-level metrics may score opposite weather directions as similar even when the operational meaning is wrong.

Resolution: The team rejected BERTScore-style evaluation for this use case and used strict word matching instead.

Evaluation

Generated forecasts were compared word-by-word to expert meteorologist bulletins using precision, recall, and F1. The team explicitly discussed why embedding-level scores were too weak for maritime forecast wording.

Metrics: Word-based F1 score · Precision and recall from true positives, false positives, and false negatives · Average word F1 comparison between Nova Pro and Claude 3.7 Sonnet · Accuracy ranges for the four-week Nova prototype

Outcomes

• In four weeks, the team taught Amazon Nova to draft Shipping Forecast text with 52-62% accuracy.
• The prototype reduced forecast text generation from a few hours to a day down to under five minutes.
• The experiments produced concrete findings about individual versus combined models, categorical versus continuous representations, overfitting versus early stopping, and LoRA versus full fine-tuning.

A single numerical weather forecast run can generate upward of 45 GB of atmospheric model data plus ocean data, and the Met Office ingests about 215 billion observations daily, so the proposed architecture emphasizes batch preparation, decoupling, and managed model deployment.