Training a 70B Japanese Large Language Model with Amazon SageMaker HyperPod

institute_of_science_tokyo · Research & Academia · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Sovereign Japanese LLM training

The Institute of Science Tokyo needed to adapt Llama 3.3 70B for Japanese language tasks while operating a reliable, efficient 256-GPU distributed training environment.

Constraints:

• The target model had 70 billion parameters.
• Training used primarily Japanese data at hundreds-of-billions-token scale.
• The cluster needed low-latency inter-node communication.
• Storage had to support high-throughput active training and durable checkpoint storage.
• Checkpointing and monitoring had to preserve reliability during long distributed runs.

Architecture

The team continually pre-trained Meta's Llama 3.3 70B into Llama 3.3 Swallow on SageMaker HyperPod. The architecture combined a 32-node P5/H100 cluster, S3 plus FSx for Lustre storage, Megatron-LM 4D parallelism, communication overlap, asynchronous distributed checkpointing, and monitoring across the training environment.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  japanese_training_corpus["Japanese training corpus"] --> amazon_s3_storage["Amazon S3 storage"]
  amazon_s3_storage --> fsx_for_lustre_filesystem["FSx for Lustre filesystem"]
  fsx_for_lustre_filesystem --> hyperpod_h100_cluster["HyperPod H100 cluster"]
  hyperpod_h100_cluster --> megatron_lm_training_stack["Megatron-LM training stack"]
  megatron_lm_training_stack --> distributed_checkpointing["Distributed checkpointing"]
  distributed_checkpointing --> amazon_s3_storage
  monitoring_systems["Monitoring systems"] --> hyperpod_h100_cluster

Components

Data flow

Japanese training data and checkpoints live durably in S3, synchronize into FSx for Lustre for high-throughput access, feed distributed Megatron-LM training across a 256-H100 HyperPod cluster, and checkpoint asynchronously through Distributed Checkpoint so training can continue with less interruption.

Why this architecture

A 70B continual-pretraining run needed distributed compute, fast shared storage, communication optimization, and non-blocking checkpointing; SageMaker HyperPod, FSx, S3, and Megatron-LM map directly to those bottlenecks.

Models used: Meta Llama 3.3 70B · Llama 3.3 Swallow · Megatron-LM

Scale

• The model had 70 billion parameters.
  “developed Llama 3.3 Swallow, a 70-billion-parameter large language model”summary, stated
• Training used 314 billion primarily Japanese tokens.
  “using 314 billion tokens of primarily Japanese training data over 16 days across 256 H100 GPUs.”summary, stated
• Training ran for 16 days across 256 H100 GPUs.
  “using 314 billion tokens of primarily Japanese training data over 16 days across 256 H100 GPUs.”summary, stated
• The cluster used 32 EC2 P5 instances with 8 H100 GPUs each.
  “The team deployed 32 EC2 P5 instances, each equipped with 8 NVIDIA H100 GPUs, creating a 256-GPU cluster”summary, stated

Alternatives considered (roads not taken)

Train from scratch

Rejected because: The project adapted Meta's Llama 3.3 through continual pre-training instead.

Single-layer storage

Rejected because: The team used S3 for durability and FSx for high-performance active training to avoid storage bottlenecks.

Traditional blocking checkpointing

Rejected because: The team used asynchronous checkpointing to reduce checkpoint bottlenecks.

Patterns

fine-tuning-adaptation (model adaptation)

The team continually pre-trained Llama 3.3 70B on Japanese data to produce Llama 3.3 Swallow.

durable-execution (training reliability)

Asynchronous distributed checkpointing protected long-running training without frequent blocking pauses.

observability-tracing (training operations)

Monitoring systems were part of the production LLMOps training stack.

Tradeoffs

Continual pre-training from Llama 3.3 70B.

Chose Adapt an existing foundation model over Train a Japanese model from scratch.

Rationale: This targeted Japanese capability while maintaining computational efficiency at 70B scale.

Use hierarchical storage.

Chose S3 plus FSx for Lustre over A single storage layer.

Rationale: The design balanced long-term durability with active-training throughput.

Gotchas (what bit them)

Storage can bottleneck large-scale training.

Root cause: Hundreds of GPUs need high-throughput data and checkpoint access.

Resolution: Use S3 for durability and FSx for Lustre for active parallel file access.

Checkpointing can interrupt long training jobs.

Root cause: Traditional checkpointing creates blocking bottlenecks.

Resolution: Use asynchronous Distributed Checkpoint to parallelize checkpoint operations.

GPU utilization suffers if communication blocks computation.

Root cause: Distributed parallelism introduces communication across data, tensor, and pipeline domains.

Resolution: Overlap communication across all parallelism domains.

Evaluation

The model was evaluated on Japanese language benchmarks against GPT-4o-mini and other leading models; operations were judged by training efficiency and production readiness.

Metrics: 70B parameters. · 314B training tokens. · 16 training days. · 256 H100 GPUs. · Superior Japanese benchmark performance versus GPT-4o-mini and other leading models.

Outcomes

• The team produced Llama 3.3 Swallow, a Japanese-specialized 70B model.
• The model outperformed GPT-4o-mini and other leading models on Japanese benchmarks.
• The training run demonstrated production-scale distributed LLM training on SageMaker HyperPod.

Training used 314B tokens over 16 days on a 256-H100 cluster.