High-Performance LLM Deployment with SageMaker AI

salesforce · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise AI model serving and FMOps

Salesforce's AI Model Serving team needed to deploy LLMs, multimodal foundation models, speech models, computer-vision models, and traditional ML models quickly across AWS Regions while balancing latency, throughput, cost-efficiency, security, and reliability.

Constraints:

• models must meet price-performance requirements for real-time AI applications
• the team supports a wide range of model types, not just LLMs
• fast-paced AI product cycles require rapid evaluation and deployment of proprietary, open-source, and third-party models
• multiple simultaneous deployments across projects must not break each other
• deployment must be secure and protect customer data
• performance tuning must not compromise scalability, latency, multi-framework support, or user experience
• configuration changes need rapid experimentation without underlying code changes

Architecture

Salesforce built a SageMaker AI hosting framework for end-to-end model serving. Data science and research teams provide models and performance requirements; the AI Model Serving team hosts, optimizes, scales, and deploys them with SageMaker AI, Deep Learning Containers, distributed inference, multi-model deployments, rolling-batch inference, elastic load balancing, instance scaling, and real-time monitoring. The DLC architecture's engine abstraction layer, model store, and workload manager let project teams optimize isolated components such as rolling batch inference without disrupting latency or multi-framework support. CI/CD pipelines using tools such as Jenkins and Spinnaker run vulnerability, compliance, model-integrity, performance, regression, and security checks; YAML configuration stored in git allows rapid optimizer and hyperparameter experimentation. Ongoing work explores quantization, tensor parallelism, caching, AWS AI chips, Graviton, distributed inference, and more granular DJL logging metrics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Salesforce model producers"] --> B["AI Model Serving team"]
  B --> C["SageMaker AI hosting framework"]
  C --> D["Deep Learning Containers"]
  D --> E["Engine abstraction layer"]
  D --> F["Model store"]
  D --> G["Workload manager"]
  G --> H["Rolling-batch inference"]
  I["Git-backed YAML configuration"] --> J["CI/CD pipelines"]
  B --> J
  J --> K["FMOps and security review"]
  J --> C
  H --> L["SageMaker scaling and monitoring"]
  C --> L

Components

Data flow

Salesforce data science and research teams build models and supply requirements. The AI Model Serving team evaluates models across environments, packages and configures them with SageMaker AI Deep Learning Containers, tunes rolling-batch and distributed-inference settings, stores configuration in git-backed YAML alongside inference code, and deploys through automated CI/CD. During development and release, pipelines run performance tests, regression tests, security checks, vulnerability scanning, compliance validation, and model-integrity checks. In production, SageMaker AI provides elastic load balancing, instance scaling, real-time monitoring, enterprise security controls, and routing/scaling control for model endpoints.

Why this architecture

Salesforce chose a SageMaker AI and DLC-based framework because it needed a self-service, secure, multi-region model-serving path that handles many model types while optimizing latency, throughput, and cost. DLCs reduce infrastructure setup overhead and provide best-practice serving defaults, while modular container internals allow one team to tune throughput without disrupting other frameworks or latency-sensitive functionality. CI/CD, git-backed YAML config, and security-by-design practices let the team iterate quickly while preserving trust and reliability.

Models used: large language models · multi-modal foundation models · speech recognition models · computer vision models · traditional ML models · TensorRT · vLLM

Scale

• The framework deploys models quickly through automated self-service processes across multiple AWS Regions.
  “deploying them quickly through automated, self-service processes across multiple AWS Regions.”article, stated
• Salesforce reports reducing model deployment time by as much as 50%.
  “With SageMaker AI, they reduced their model deployment time by as much as 50%.”article, stated
• Iteration cycles became days or hours instead of weeks.
  “They experienced faster iteration cycles, measured in days or even hours instead of weeks.”article, stated
• The team supports multiple simultaneous deployments across projects.
  “Because the team supports multiple simultaneous deployments across projects, they needed to make sure enhancements in each project didn’t compromise others.”article, stated
• The serving scope includes LLMs, multimodal foundation models, speech recognition, computer vision, and traditional ML models.
  “including LLMs, multi-modal foundation models (FMs), speech recognition, and computer vision-based models.”article, stated

Alternatives considered (roads not taken)

Hand-build model-serving infrastructure for each deployment

Rejected because: DLCs provide optimized frameworks, CUDA settings, and best-practice defaults so engineers can spend time on model optimization rather than infrastructure setup.

Tune one project monolithically inside the serving container

Rejected because: A monolithic approach would risk breaking other deployments or capabilities; the DLC modular architecture isolates component-level optimization.

Change inference code for each optimizer or hyperparameter experiment

Rejected because: Git-backed YAML configuration allowed rapid experimentation across optimizers and hyperparameters without altering code.

Treat security as a post-deployment review

Rejected because: The team embedded secure-by-design principles, vulnerability checks, compliance validation, model-integrity checks, and development-environment security testing throughout the lifecycle.

Patterns

latency-optimization (serving)

Salesforce used SageMaker AI DLCs, distributed inference, rolling-batch inference, GPU utilization defaults, memory allocation defaults, TensorRT, vLLM, and future quantization/tensor-parallel strategies to improve latency and throughput.

cost-aware-routing (serving)

The hosting framework explicitly balances throughput, latency, hardware cost, scaling strategy, and cost-efficiency across inference environments.

eval-driven-development (quality)

Model serving changes are gated by extensive performance testing, regression testing, development-environment security testing, and regular cross-functional review.

canary-rollout (release)

The team uses small incremental deployments with frequent automated testing to identify issues early and minimize release risk.

safety-policy-enforcement (security)

Security requirements are embedded into architecture and CI/CD through vulnerability checks, compliance validation, model integrity checks, encryption, RBAC, and network isolation.

observability-tracing (operations)

Production model serving relies on real-time model monitoring, continuous deployment monitoring, and planned granular logging metrics in DJL.

Tradeoffs

Use SageMaker AI DLCs as the deployment substrate

Chose pre-built optimized containers with best-practice settings over custom per-model serving environments.

Rationale: DLCs reduce setup overhead, provide optimized frameworks and CUDA settings, and let engineers focus on model-specific optimization.

Tune rolling batch inference through exposed configuration

Chose max_rolling_batch_size and job_queue_size configuration with best-practice defaults over extensive custom batching engineering.

Rationale: The team could balance GPU utilization, throughput, and real-time latency by adjusting exposed DLC parameters.

Use modular container architecture

Chose engine abstraction layer, model store, and workload manager components over one tightly coupled serving container.

Rationale: Modularity lets teams optimize rolling batching or streaming independently without disrupting latency, multi-framework support, or other critical container behavior.

Store serving configuration in git-backed YAML

Chose configuration managed alongside inference code over hardcoded optimizer and hyperparameter settings.

Rationale: Versioned YAML lets teams experiment rapidly and coordinate changes through CI while keeping the underlying inference code stable.

Gotchas (what bit them)

Serving improvements can break unrelated deployments or model frameworks.

Root cause: The team supports multiple simultaneous projects, so tuning one component can affect latency, scalability, user experience, or multi-framework support elsewhere.

Resolution: Salesforce adopted modular DLC development and regression testing so individual optimizations can be isolated and checked before release.

Rapid model deployment can conflict with enterprise trust and security requirements.

Root cause: New models must move quickly, but Salesforce still needs secure hosting, customer-data protection, compliance, and model-integrity validation.

Resolution: Security is embedded throughout development and deployment with secure-by-design architecture, automated checks, encryption, RBAC, network isolation, monitoring, and collaboration with security teams.

Manual configuration slows model-serving experimentation.

Root cause: Optimizer and hyperparameter experiments would otherwise require code changes or bespoke infrastructure work.

Resolution: Configuration is stored in git as YAML next to inference code, enabling fast experimentation without altering the serving code.

Inference optimization is a moving target as models and hardware change.

Root cause: Salesforce's generative AI needs and the model landscape keep scaling and changing, requiring constant evaluation of optimization techniques and hardware options.

Resolution: The team continues research and development around quantization, tensor parallelism, batching, DJL caching, AWS AI chips, Graviton, FMOps automation, and distributed inference.

Evaluation

Salesforce evaluates model-serving deployments through performance testing across multiple environments, regression testing for optimizations such as TensorRT and vLLM, development-environment security testing, automated CI/CD checks for vulnerabilities, compliance, and model integrity, and regular cross-functional reviews with development, FMOps, and security teams.

Metrics: throughput · latency · cost-efficiency · scalability and reliability for inferencing on AWS · deployment time reduced by as much as 50% · iteration cycles measured in days or hours instead of weeks · vulnerability, compliance, and model-integrity checks

Outcomes

• Salesforce reported substantial improvements in deployment speed and cost-efficiency from the SageMaker AI strategy.
• Iteration cycles moved to days or hours instead of weeks.
• Model deployment time was reduced by as much as 50%.
• The framework supports secure, automated, self-service model deployment across multiple AWS Regions.
• The team can tune rolling-batch inference and other optimizations without extensive custom engineering.

The framework serves Salesforce enterprise AI across multiple AWS Regions and multiple simultaneous model deployments, covering LLMs, multimodal foundation models, speech recognition, computer vision, and traditional ML models with automated CI/CD, monitoring, scaling, and security controls.