Training and Deploying MPT: Lessons Learned in Large Scale LLM Development

mosaicml · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Open-source LLM training operations

MosaicML wanted to train commercially usable open-source MPT models while proving that large language model training could be planned, budgeted, evaluated, and operated more economically than many practitioners assumed.

Constraints:

• Training runs were expensive enough to require rigorous planning.
• Organizations needed privacy, quality, cost, or latency reasons to justify custom models.
• Evaluation criteria had to be defined before committing compute.
• The team needed distributed training infrastructure and fault tolerance.
• Some fine-tuning data licensing choices affected commercial usability.

Architecture

MosaicML trained MPT decoder-only transformer models from scratch using its own optimized architecture and training platform. The lifecycle covered problem definition, scaling-law compute budgeting, evaluation planning, data preparation, distributed training, fault tolerance, fine-tuning, and variants such as chat, code, and long-context models.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  planning_and_evaluation_process["Planning and evaluation process"] --> compute_budgeting_process["Compute budgeting process"]
  planning_and_evaluation_process --> training_data_pipeline["Training data pipeline"]
  compute_budgeting_process --> distributed_training_platform["Distributed training platform"]
  training_data_pipeline --> distributed_training_platform
  distributed_training_platform --> fault_tolerance_operations["Fault tolerance operations"]
  distributed_training_platform --> mpt_model_family["MPT model family"]
  mpt_model_family --> evaluation_suite["Evaluation suite"]

Components

Data flow

The team defines the model goal and evaluation criteria, budgets compute with scaling laws, prepares pretraining data, runs distributed pretraining from randomized weights, fine-tunes variants for tasks like chat or code, and uses benchmarks plus informal vibe checks to assess behavior.

Why this architecture

MosaicML trained MPT both to provide permissive open models and to battle-test the training platform, so the architecture emphasizes end-to-end LLMOps discipline rather than a single model artifact.

Models used: MPT-7B · MPT-30B · MPT chat-tuned variant · MPT long-context variants

Scale

• MPT-7B pretraining cost approximately 250,000.
  “they successfully trained MPT-7B for approximately 250,000 on pre-training alone”summary, stated
• The talk contrasts that with common expectations of 500K-$1M+.
  “significantly lower than the 500K-$1M+ that most practitioners expected”summary, stated
• The primary variants included MPT-7B and MPT-30B.
  “The result was two primary variants: MPT-7B and MPT-30B”summary, stated
• MPT included 8K and 64K context length variants.
  “with additional specialized versions including 8K and 64K context length models”summary, stated

Alternatives considered (roads not taken)

Use existing APIs or open-source models

Rejected because: Organizations may need custom models for privacy, quality, cost, or latency reasons.

Apply RLHF to MPT

Rejected because: MosaicML pre-trained and fine-tuned MPT but did not apply RLHF.

Train before defining evaluation criteria

Rejected because: The presentation stresses evaluation planning before committing expensive compute.

Patterns

eval-driven-development (training planning)

MosaicML emphasized defining evaluations before expensive LLM training runs.

fine-tuning-adaptation (model variants)

MPT used pretraining and fine-tuning to produce general and specialized model variants.

durable-execution (training operations)

The case highlights distributed training infrastructure and fault tolerance as core lifecycle concerns.

Tradeoffs

Train MPT from scratch.

Chose Own optimized architecture and randomized weights over Continued pretraining from another base model.

Rationale: This tested MosaicML's platform and produced commercially permissive open-source models.

Use both formal evals and vibe checks.

Chose Benchmarks plus conversational qualitative tests over Formal metrics alone.

Rationale: Model quality includes capabilities that are not captured by one benchmark.

Gotchas (what bit them)

Training an LLM may not be necessary.

Root cause: Smaller models or alternative approaches can solve some problems with less complexity.

Resolution: Define the problem clearly and question the need for an LLM before training.

Fine-tuning data licensing can limit commercial usability.

Root cause: The chat-tuned variant used Alpaca data generated by ChatGPT.

Resolution: Track dataset licensing separately from model technical quality.

Evaluation

MosaicML planned evaluations up front, used benchmarks such as HumanEval for coding use cases, and supplemented with qualitative vibe checks.

Metrics: MPT-7B pretraining cost approximately 250,000. · Practitioner expectations were 500K-$1M+. · Model variants included MPT-7B, MPT-30B, 8K, and 64K context models.

Outcomes

• MosaicML trained the MPT open-source model family.
• The project demonstrated a lower-than-expected cost for GPT-3-quality 7B training.
• The work battle-tested MosaicML's training platform.

The case reports MPT-7B pretraining around 250,000 and variants including MPT-30B and 64K context models.