Large-Scale Deployment of On-Device and Server Foundation Models for Consumer AI Features

apple · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Consumer-scale foundation model deployment across devices and cloud

Apple needed to deliver generative AI features across its software platforms while preserving privacy, speed, power efficiency, structured developer integration, multilingual support, multimodal capability, and safety. The core challenge was choosing when to run a compact model on-device and when to use a scalable server model, then optimizing both for production latency, memory, quality, and responsible deployment.

Constraints:

• Apple Intelligence features need to run across apps and everyday user experiences
• privacy principles require on-device processing and Private Cloud Compute for appropriate tasks
• the on-device model must run efficiently on Apple silicon with low latency and minimal resource usage
• the server model must scale for more complex tasks while maintaining low latency
• the models support 15 languages and multimodal image/text capabilities
• developer APIs require reliable structured outputs and type-safe integration
• model training excludes users' private personal data and user interactions
• foundation models and user-facing features must pass safety, bias, locale, and feature-specific evaluation

Architecture

Apple deploys a complementary model family: an approximately 3B-parameter on-device foundation model optimized for Apple silicon and a server-based parallel-track mixture-of-experts model for Private Cloud Compute. The on-device model uses block-level KV cache sharing, quantization-aware training, 2-bit decoder weights, 4-bit embeddings, 8-bit KV cache, and adapter recovery. The server model uses PT-MoE track parallelism, ASTC compression, and dedicated decompression support. The training pipeline combines licensed, public, Applebot-crawled, filtered, multilingual, code/math, image-text, synthetic caption, SFT, tool-use, and RLHF data. The Foundation Models framework exposes the on-device model to Swift developers through guided generation, constrained decoding, speculative decoding, tool calling, and adapter training, with safety guardrails and feedback loops across users and developers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  features["Apple Intelligence features"] --> device["On-device foundation model"]
  features --> server["Server foundation model"]
  server --> pcc["Private Cloud Compute"]
  data["Training data pipeline"] --> device
  data --> server
  data --> vision["Vision encoder stack"]
  vision --> device
  vision --> server
  sft["SFT and RLHF pipeline"] --> device
  sft --> server
  compression["Compression pipeline"] --> device
  compression --> server
  framework["Foundation Models framework"] --> device
  framework --> guided["Guided generation runtime"]
  framework --> tools["Tool calling runtime"]
  guided --> apps["Developer apps"]
  tools --> apps
  safety["Responsible AI and safety system"] --> features
  safety --> data
  eval["Evaluation system"] --> safety
  feedback["Feedback channels"] --> eval
  features --> feedback

Components

Data flow

Data is sourced from licensed, public, open-source, and Applebot-crawled content, filtered for quality, safety, legal compliance, PII, and unsafe material. Text and multimodal data train on-device and server models through staged pretraining, visual encoder alignment, continued pretraining, SFT, tool-use annotation, multilingual expansion, and RLHF. At deployment time, Apple Intelligence routes appropriate work to the compact on-device model or the server model on Private Cloud Compute. Developers use the Foundation Models framework to define Swift output types, have the framework inject the response format, and rely on OS-level constrained and speculative decoding to produce type-conformant outputs. User, developer, evaluation, and safety signals feed continuous improvement.

Why this architecture

Apple splits on-device and server inference because local execution provides privacy and low-latency resource efficiency, while a server MoE model provides higher accuracy and scalability for more complex tasks. Vertical integration with Apple silicon, Swift, the OS daemon, guided generation, and Private Cloud Compute lets Apple optimize the whole stack from model architecture through developer APIs and safety guardrails.

Models used: Apple on-device foundation language model · Apple server foundation model · parallel-track mixture-of-experts · Private Cloud Compute · ViT-g · ViTDet-L · CLIP models · OCR tools · in-house grounding model · in-house image captioning model · teacher model · GPT-4o as a judge · Qwen-2.5-3B · Qwen-3-4B · Gemma-3-4B · Llama-4-Scout · Qwen-3-235B · InternVL-2.5-4B · Qwen-2.5-VL

Scale

• The on-device model is approximately 3 billion parameters.
  “a compact, approximately 3-billion-parameter model”article, stated
• The models are designed to support 15 languages.
  “designed to support 15 languages.”article, stated
• On-device KV cache sharing reduces memory usage by 37.5%.
  “reducing the KV cache memory usage by 37.5% and significantly improving the time-to-first-token.”article, stated
• PT-MoE can reduce synchronization overhead by 87.5% when track-block depth is 4.
  “For example, if D = 4, PT reduces 87.5% of the synchronization overhead.”article, stated
• Applebot training data spans hundreds of billions of pages.
  “Applebot, spanning hundreds of billions of pages and covering an extensive range of languages, locales, and topics.”article, stated
• The image-text data pipeline yielded over 10 billion image-text pairs.
  “After de-duplication, this process yielded over 10B high-quality image-text pairs.”article, stated
• The interleaved image-text dataset contains 175 million documents and over 550 million images.
  “this resulted in 175M interleaved image-text documents, containing over 550M images.”article, stated
• Synthetic image captioning generated over 5 billion image-caption pairs.
  “generating over 5B image-caption pairs that we used across the pre-training stages.”article, stated
• Sparse-upcycling reduced teacher model training cost by 90%.
  “This reduced the cost of training the teacher model by 90%.”article, stated
• The server model was trained from scratch on 14 trillion text tokens.
  “we trained the sparse server model from scratch on 14T text tokens.”article, stated
• The tokenizer grew from 100k to 150k vocabulary with 25% more tokens for many languages.
  “we extended the text tokenizer from a vocabulary size of 100k to 150k, achieving representation quality for many additional languages with just 25% more tokens.”article, stated
• Final continued pretraining handled sequences up to 65K tokens.
  “we trained the model to handle significantly longer context lengths using sequences up to 65K tokens”article, stated

Alternatives considered (roads not taken)

Use only cloud/server inference for all Apple Intelligence features

Rejected because: Apple emphasizes local privacy, low latency, minimal resource usage, and the on-device Foundation Models framework.

Use only on-device inference for every task

Rejected because: The server model is kept for higher accuracy and scalable handling of more complex tasks.

Train a large dense teacher model from scratch for on-device distillation

Rejected because: Apple sparse-upcycled a MoE teacher from a pretrained 3B model to cut teacher training cost.

Rely mainly on heuristic web-data filtering

Rejected because: Apple reduced overly aggressive heuristics and added model-based filtering signals to retain more informative content.

Use unconstrained text generation for developer outputs

Rejected because: The Foundation Models framework uses guided generation and constrained decoding to conform to Swift-defined output types.

Patterns

model-routing (deployment)

Apple uses complementary on-device and server foundation models for different privacy, efficiency, accuracy, and complexity requirements.

privacy-preserving-processing (privacy)

Training excludes private user data, deployment uses on-device processing and Private Cloud Compute, and responsible AI principles include privacy protection.

model-distillation (training)

The on-device model is trained with a distillation loss using a sparse-upcycled MoE teacher to reduce teacher-training cost.

latency-optimization (serving)

Apple uses KV-cache sharing, PT-MoE track parallelism, quantization, ASTC compression, constrained decoding, and speculative decoding to reduce latency and overhead.

structured-output-contract (developer-framework)

Guided generation maps Swift types to a response format and uses constrained decoding so model outputs conform to developer-defined types.

tool-use-orchestration (developer-framework)

The Foundation Models framework lets developers provide tools and automatically handles serial and parallel tool-call graphs.

eval-driven-development (evaluation)

Apple uses human graders, auto benchmarks, locale-specific evals, image evaluations, feature-specific adapter evaluations, safety datasets, and red teaming.

safety-policy-enforcement (responsible-ai)

Apple uses responsible AI principles, safety taxonomies, built-in guardrails, legal/cultural review, feature-specific safety datasets, and locale-specific red teaming.

Tradeoffs

Where to run foundation model inference

Chose Hybrid on-device plus Private Cloud Compute server models over A single universal deployment target.

Rationale: On-device inference handles privacy and efficiency; server inference handles high-accuracy, scalable complex tasks.

How to build the on-device teacher cheaply

Chose Sparse-upcycle a 64-expert MoE from a pretrained 3B model over Pretrain a large dense teacher from scratch.

Rationale: Sparse-upcycling reduced teacher-training cost by 90% while supporting distillation for the compact model.

How to expose model outputs to developers

Chose Swift guided generation with constrained decoding over Free-form text outputs that developers parse manually.

Rationale: The vertical integration creates Swift-type outputs with format guarantees and simpler developer code.

How to compress models for production

Chose Aggressive quantization and ASTC compression with recovery adapters over Keeping uncompressed weights for maximum raw quality.

Rationale: Compression enables practical deployment with only slight quality regressions or occasional minor improvements.

How to evaluate multilingual expansion

Chose Locale-specific native-speaker and human evaluation over Simple translation-based evaluation.

Rationale: Apple evaluates whether responses sound native and use appropriate local terms, not just whether they are translated.

Gotchas (what bit them)

On-device models must reduce memory and token latency without losing too much quality.

Root cause: Mobile deployment faces KV-cache, power, and resource constraints.

Resolution: Use block-level KV cache sharing, quantization-aware training, embedding/KV-cache quantization, and adapter recovery.

Server model parallelism can waste latency on synchronization.

Root cause: Traditional tensor parallelism synchronizes more frequently across layers.

Resolution: Use PT-MoE track blocks that process independently and synchronize only at input/output boundaries.

Web-scale training data needs more than heuristic filtering.

Root cause: Web data is noisy, and overly aggressive rules remove informative content.

Resolution: Add model-informed filtering signals, high-fidelity HTML extraction, dynamic rendering, legal/safety filters, and quality checks.

Multimodal and multilingual expansion can damage existing text capability if adaptation is not controlled.

Root cause: New visual, long-context, code, math, and multilingual training stages change the model distribution.

Resolution: Use staged continued pretraining and explicitly train multimodal adaptation without damaging text capabilities.

Locale support is not just translation accuracy.

Root cause: Responses must be native-sounding and culturally appropriate for each locale.

Resolution: Use native-speaker prompts, locale-specific evaluation categories, legal/cultural experts, regional datasets, and locale red teaming.

Adapters are powerful but create version-management overhead.

Root cause: Adapters are tied to the base model version.

Resolution: Reserve adapter training for advanced use cases after exploring the base model, and retrain adapters for each new base model version.

Evaluation

Apple evaluates on-device and server models with human graders, standard language/reasoning categories, locale-specific human evaluation, GPT-4o-as-judge auto evals, side-by-side comparisons with external models, image-understanding evals, feature-specific adapter evals, targeted safety datasets, internal/external human evaluation, auto-grading, human red teaming, and user/developer feedback.

Metrics: human preference rates · standard language and reasoning capability categories · tool-use evaluation · locale-specific native-sounding quality · GPT-4o judge alignment with human evals · image-question evaluation categories · feature-specific extraction accuracy · safety high-risk content datasets · culture-specific risk and bias evaluations · thumbs up/down user feedback · developer Feedback Assistant reports

Outcomes

• Apple introduced a new generation of foundation models for Apple Intelligence and direct developer access to the on-device model.
• The on-device model uses an approximately 3B-parameter compact architecture with 37.5% lower KV-cache memory usage.
• The server PT-MoE architecture can reduce synchronization overhead by 87.5% at track-block depth 4.
• Apple reduced teacher training cost by 90% through sparse-upcycling for on-device distillation.
• The server model trained on 14T text tokens, and multimodal pretraining used over 10B image-text pairs plus over 5B synthetic image-caption pairs.
• The framework provides Swift guided generation, constrained decoding, speculative decoding, tool calling, and adapters.
• Apple reports favorable on-device text comparisons against Qwen-2.5-3B and competitive English performance against larger Qwen/Gemma models, while the server model remains behind larger GPT-4o/Qwen-3-235B-class models.
• Safety, locale, feedback, and responsible AI loops are integrated into feature and model deployment.

The deployment combines 15-language support, a 3B on-device model, a server MoE model, hundreds of billions of crawled pages, 14T server pretraining text tokens, over 10B image-text pairs, 175M interleaved documents, 550M images, 5B generated caption pairs, 65K-token long-context training, and cross-device developer access via the Foundation Models framework.