GPU Resource Optimization for Multi-Model LLM Deployment

salesforce · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM model serving infrastructure

Salesforce needed to serve multiple proprietary LLMs with different sizes, traffic patterns, latency needs, and cost constraints while improving GPU utilization and avoiding over-provisioned SageMaker endpoints.

Constraints:

• Salesforce's AI Platform Model Serving team hosts proprietary LLMs such as CodeGen and XGen
• models were distributed across single model endpoints
• model sizes ranged from a few GB to 30 GB
• larger 20-30 GB models had lower traffic and underutilized multi-GPU instances
• medium-sized approximately 15 GB models had higher-traffic, low-latency, high-throughput needs
• the platform had to maintain performance and stability while reducing infrastructure cost
• the serving foundation needed to scale to hundreds of models and future high-end GPU generations

Architecture

Salesforce adopted Amazon SageMaker AI inference components alongside existing single-model endpoints. A SageMaker endpoint defines instance type and baseline capacity; each model becomes an inference component with its own accelerator allocation, memory allocation, model artifacts, container image, copies, and scaling policy. SageMaker packs multiple models onto GPU instances, scales model copies and instances dynamically, and lets Salesforce keep SMEs for stable critical workloads while using inference components for variable workloads and efficient GPU sharing.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  salesforce_ai_applications["Salesforce AI applications"] --> sagemaker_ai_endpoint["SageMaker AI endpoint"]
  salesforce_model_serving_team["Salesforce Model Serving team"] --> sagemaker_ai_endpoint
  sagemaker_ai_endpoint --> inference_components["Inference components"]
  sagemaker_ai_endpoint --> single_model_endpoints["Single model endpoints"]
  model_packages["Model packages"] --> inference_components
  inference_components --> auto_scaling_policies["Auto scaling policies"]
  auto_scaling_policies --> managed_instance_auto_scaling["Managed instance auto scaling"]
  managed_instance_auto_scaling --> gpu_instance_fleet["GPU instance fleet"]
  inference_components --> gpu_instance_fleet

Components

Data flow

Salesforce applications such as ApexGuru or Agentforce send inference traffic to SageMaker-hosted model endpoints. For inference-component endpoints, SageMaker loads model packages as per-model components, assigns accelerator and memory resources, places and packs model copies onto GPU instances, and scales model copies or backing instances based on traffic. Critical steady workloads can remain on single-model endpoints, while variable model workloads share GPU capacity on inference-component endpoints.

Why this architecture

Inference components solve Salesforce's mismatch between model size, traffic, and GPU allocation by packing multiple models on shared endpoints with per-model resource controls and scaling policies. The hybrid architecture keeps dedicated SMEs where predictable performance is valuable, while inference components reduce idle GPU capacity and operational overhead for variable workloads.

Models used: CodeGen · XGen · Inline · BlockGen · FlowGPT · TextEval

Scale

• Salesforce model sizes ranged from a few gigabytes to 30 GB.
  “supporting a diverse range of model sizes from a few gigabytes (GB) to 30 GB”article, stated
• The underutilized large-model class was around 20-30 GB.
  “Their larger models (20–30 GB) with lower traffic patterns were running on high-performance GPUs”article, stated
• Medium-sized high-traffic models were approximately 15 GB.
  “their medium-sized models (approximately 15 GB) handling high-traffic workloads demanded low-latency, high-throughput processing capabilities.”article, stated
• Salesforce achieved up to an eight-fold reduction in deployment and infrastructure costs.
  “achieving up to an eight-fold reduction in deployment and infrastructure costs while maintaining high performance standards.”article, stated
• The platform can scale to hundreds of models.
  “Salesforce can maximize the utilization of compute instances, scale to hundreds of models, and optimize costs while providing predictable performance.”article, stated
• Salesforce currently operates on P4d instances and plans to use P5en instances with NVIDIA H200 GPUs.
  “Operating on Amazon EC2 P4d instances today, with plans to use the latest generation [P5en instances]”article, stated

Alternatives considered (roads not taken)

Keep all models on single model endpoints

Rejected because: SMEs provide predictable performance for critical steady workloads but create underutilized or over-provisioned GPU capacity for variable workloads.

Move entirely to inference components

Rejected because: Salesforce introduced inference component endpoints alongside existing SMEs to preserve stability, performance, and usability where SMEs fit better.

Over-provision multi-GPU setups for medium models

Rejected because: Medium 15 GB high-traffic models incurred higher costs on similar multi-GPU setups even though they needed lower-latency, higher-throughput serving.

Use traditional blue/green deployment for future model updates

Rejected because: Salesforce is looking to rolling updates to avoid requiring doubled instances and to minimize extra capacity during updates.

Patterns

model-routing (resource-aware serving)

Salesforce routes different model sizes and traffic patterns across SMEs and inference components, with model-specific accelerator, memory, copy, and scaling settings.

cost-aware-routing (GPU utilization)

Inference components pack multiple models on shared high-performance GPU endpoints and dynamically scale copies/instances to reduce idle compute and infrastructure cost.

tool-use-orchestration (model serving control plane)

The serving architecture coordinates model artifacts, container images, accelerator allocations, memory reservations, model copies, autoscaling policies, and GPU instance placement.

Tradeoffs

Use inference components for variable workloads.

Chose Shared endpoint model packing with per-model resource controls over Separate single-model endpoints for every model.

Rationale: Inference components reduce idle GPU capacity, lower cost, and decrease endpoint-management overhead for variable traffic patterns.

Keep a hybrid endpoint strategy.

Chose SMEs plus inference component endpoints over A one-size-fits-all hosting pattern.

Rationale: SMEs fit consistent critical traffic, while inference components optimize variable workloads through dynamic scaling and GPU sharing.

Allocate resources at model granularity.

Chose Per-model accelerator, memory, copy, and scaling policy configuration over Endpoint-level resource allocation only.

Rationale: Granular allocation lets Salesforce match each model's traffic and performance profile while sharing the same endpoint.

Use high-end GPUs more selectively.

Chose High-performance GPU access through packed and autoscaled endpoints over Dedicating expensive GPUs to underutilized models.

Rationale: Better packing lets smaller models benefit from high-performance GPUs without the traditional cost overhead.

Gotchas (what bit them)

Large low-traffic models can waste high-performance multi-GPU instances.

Root cause: The 20-30 GB model class needed high-performance GPUs but did not receive enough traffic to use the whole instance efficiently.

Resolution: Use inference components to pack models and allocate resources granularly on shared endpoints.

Medium high-traffic models can be over-provisioned when hosted like larger models.

Root cause: Approximately 15 GB models needed low latency and high throughput but often ran on similar multi-GPU setups with excessive cost.

Resolution: Give each model independent scaling and resource allocation based on traffic patterns.

Full migration to a shared endpoint model could risk stability for critical workloads.

Root cause: Some critical workloads have consistent traffic and need dedicated predictable performance.

Resolution: Use a hybrid architecture with SMEs alongside inference component endpoints.

Model updates can temporarily double capacity requirements.

Root cause: Traditional blue/green deployment requires extra instances during replacement.

Resolution: Plan to use SageMaker rolling updates for inference component endpoints with batch-by-batch model updates and real-time safety checks.

Evaluation

Salesforce evaluated the serving architecture through GPU utilization, infrastructure cost, performance preservation, endpoint-management overhead, scaling behavior, and ability to host both large and smaller workloads on the same infrastructure.

Metrics: model sizes from a few GB to 30 GB · large low-traffic models around 20-30 GB · medium high-traffic models around 15 GB · up to eight-fold reduction in deployment and infrastructure costs · ability to scale to hundreds of models · support for P4d, P5, and P5en high-performance GPU instances

Outcomes

• Salesforce improved GPU utilization through SageMaker inference components.
• Multiple models can share GPU resources efficiently on the same endpoint.
• Model-specific resource allocation and independent scaling are available for CodeGen ensemble and other models.
• Infrastructure and deployment costs were reduced by up to eight-fold while maintaining high performance.
• Smaller models can use high-performance GPUs for latency and throughput without excessive cost.
• The platform can scale to hundreds of models and positions Salesforce for future P4d, P5, and P5en GPU workloads.
• Salesforce plans to use rolling updates and real-time safety checks to reduce model-update overhead.

The architecture supports a diverse model portfolio ranging from a few GB to 30 GB, with model-specific traffic patterns, and is designed to scale to hundreds of Salesforce AI models on high-end GPU infrastructure.