Streamlining Custom LLM Deployment with Serverless Infrastructure

salesforce · Tech · 2025 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Custom LLM deployment infrastructure for agentic AI applications

Salesforce's AI platform team needed to reduce the operational overhead and cost of deploying fine-tuned Llama, Qwen, and Mistral models for Agentforce while preserving existing production interfaces and scalability.

Constraints:

• Teams spent months optimizing instance families, serving engines, and configurations.
• Frequent model releases made the process hard to maintain.
• GPU capacity reservations for peak usage wasted resources during lower traffic.
• Migration could not disrupt production workloads or downstream application interfaces.
• Some custom preprocessing and postprocessing logic still needed to run in existing SageMaker-style tooling.

Architecture

Salesforce added Amazon Bedrock Custom Model Import to its existing CI/CD model pipeline. CI/CD saves model artifacts to S3, Bedrock imports models through a unified API, and inference clients keep calling existing SageMaker-style endpoints through lightweight CPU proxy containers that preserve preprocessing, postprocessing, monitoring, and logging while forwarding GPU-intensive inference to Bedrock serverless endpoints.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ci_cd_model_pipeline["CI/CD model pipeline"] --> s3_model_store["S3 model store"]
  ci_cd_model_pipeline --> bedrock_custom_model_import_api["Bedrock Custom Model Import API"]
  s3_model_store --> bedrock_custom_model_import_api
  iam_and_cross_account_access["IAM and cross-account access"] --> s3_model_store
  bedrock_custom_model_import_api --> amazon_bedrock_serverless_inference["Amazon Bedrock serverless inference"]
  prediction_service_clients["Prediction service clients"] --> sagemaker_cpu_proxy_containers["SageMaker CPU proxy containers"]
  sagemaker_cpu_proxy_containers --> amazon_bedrock_serverless_inference
  iam_and_cross_account_access --> amazon_bedrock_serverless_inference

Components

Data flow

CI/CD writes model artifacts to an S3 model store, then calls the Bedrock Custom Model Import API. Bedrock pulls the model from S3 and preloads it. At inference time, client requests go through Salesforce's preprocessing layer and SageMaker CPU proxy, which forwards the actual inference call to Bedrock, receives raw model output, applies postprocessing, and returns through the existing prediction-service interface.

Why this architecture

Salesforce used a hybrid Bedrock-plus-proxy architecture to gain serverless custom-model deployment and cost benefits without forcing downstream application changes. The CPU proxies kept existing SageMaker monitoring, logging, preprocessing, and postprocessing, while Bedrock handled custom model import, autoscaling, and GPU inference.

Models used: Llama · Qwen · Mistral · ApexGuru · fine-tuned QWEN-2.5 13B · vLLM · TensorRT-LLM

Scale

• The Bedrock import step adds about five to seven minutes while the overall model release remains about one hour.
  “adding minimal overhead (5–7 mins, depending on model size) to their deployment timeline—their overall model release process remains at approximately 1 hour.”article, stated
• Salesforce achieved a 30% reduction in time to iterate and deploy models to production.
  “the team achieved a 30% reduction in time to iterate and deploy models to production.”article, stated
• Salesforce achieved up to 40% cost reduction through Bedrock.
  “Salesforce achieving up to 40% cost reduction through Amazon Bedrock.”article, stated
• Bedrock autoscaled the ApexGuru fine-tuned Qwen model from one to three copies at concurrency 32.
  “Amazon Bedrock Custom Model Import auto scaled from one to three copies as concurrency reached 32.”article, stated
• CPU proxy compatibility adds an extra five to ten milliseconds of latency.
  “The trade-off involves an additional network hop adding 5–10 millisecond latency and the cost of always-on CPU instances”article, stated

Alternatives considered (roads not taken)

Continue traditional SageMaker managed inference for all custom models

Rejected because: It required months of instance-family, serving-engine, and configuration optimization and wasted reserved GPU capacity.

Rewrite downstream applications for Bedrock directly

Rejected because: Salesforce prioritized zero downtime, no downstream application changes, and preserving existing endpoint interfaces.

Use API Gateway and Lambda for pre/post-processing

Rejected because: Salesforce found that approach less backward-compatible and saw cold-start impacts with larger processing libraries.

Patterns

model-routing (deployment)

Salesforce preserved existing prediction-service endpoints through CPU proxies while routing model inference to Bedrock serverless endpoints.

cost-aware-routing (infrastructure cost)

The migration replaced peak GPU reservations with Bedrock pay-per-use economics for variable traffic patterns.

latency-optimization (serving)

Salesforce load-tested concurrency, throughput, autoscaling, and proxy latency to validate production performance.

canary-rollout (migration)

Salesforce migrated gradually, starting with non-critical workloads before broadening serverless custom-model deployment.

safety-policy-enforcement (platform integration)

Bedrock Custom Model Import integrates with Bedrock Guardrails, Knowledge Bases, and Agents while using IAM/cross-account access controls.

Tradeoffs

Use a hybrid proxy architecture.

Chose SageMaker CPU proxies plus Bedrock serverless inference over Directly replacing existing serving endpoints.

Rationale: The hybrid path preserves integrations, monitoring, and preprocessing/postprocessing while moving expensive GPU inference to Bedrock.

Adopt serverless custom model deployment.

Chose Amazon Bedrock Custom Model Import over Reserved GPU capacity for peak usage.

Rationale: Variable traffic made reserved GPU capacity wasteful; Bedrock pay-per-use reduced cost for dev, test, staging, and production traffic mixes.

Keep SageMaker for unsupported or highly customized cases.

Chose Bedrock where compatible, SageMaker AI for unsupported architectures over A single serving platform for every model.

Rationale: Bedrock compatibility depends on model architecture support; unsupported cutting-edge architectures may still need SageMaker.

Gotchas (what bit them)

LLM serving optimization can take months when infrastructure is self-managed.

Root cause: Teams had to tune instance families, serving engines such as vLLM versus TensorRT-LLM, and configurations.

Resolution: Use Bedrock Custom Model Import to abstract infrastructure decisions and focus on model performance.

Backward compatibility can add latency and cost.

Root cause: Keeping SageMaker proxy containers preserves existing interfaces but adds an extra network hop and always-on CPU instances.

Resolution: Accept the small proxy overhead to avoid downstream changes while moving GPU-intensive inference to Bedrock.

Cold starts can be minutes for larger models.

Root cause: Serverless custom model endpoints need to spin up larger model copies.

Resolution: Keep latency-sensitive endpoints warm with at least one active model copy via health checks every 14 minutes.

Evaluation

Salesforce used load testing across concurrency scenarios, measured latency and throughput, and evaluated operational efficiency and cost optimization after migration.

Metrics: Import step overhead: five to seven minutes. · Overall model release process: approximately one hour. · Deployment iteration improvement: 30%. · Cost reduction: up to 40%. · Concurrency 32 autoscaled to three model copies. · Proxy overhead: five to ten milliseconds. · Cold starts for 26B models: a couple of minutes.

Outcomes

• Salesforce reduced time to iterate and deploy models by 30%.
• The migration reduced costs by up to 40%.
• The hybrid architecture preserved backward compatibility with existing integrations.
• ApexGuru is now deployed in production.

The case reports ApexGuru fine-tuned Qwen-2.5 13B load tests, Bedrock scaling from one to three copies at concurrency 32, and a maintained approximately one-hour release process.