Mission-Critical LLM Inference Platform Architecture

baseten · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Mission-critical LLM inference platform

Baseten needed to serve custom and open-weight LLMs for customers whose products depend on strict latency, throughput, compliance, and reliability, including very large models like DeepSeek V3 and multi-model voice-call workflows.

Constraints:

• DeepSeek V3 has 671 billion parameters and cannot fit on a standard 8-card H100 node even with FP8 weights plus KV cache.
• Customers require P95/P99 latency, time-to-first-token, throughput, security, compliance, and geography guarantees without noisy-neighbor effects.
• A single model replica or single-region Kubernetes cluster cannot always meet demand or GPU capacity requirements.
• Some workflows require multiple models, such as speech-to-text, LLM calls, and text-to-speech, where API-to-API orchestration adds too much latency.
• Framework support alone is not enough; speculative decoding, scaling, failure recovery, and workflow enablement all need production infrastructure around them.

Architecture

Baseten runs dedicated inference resources for customer models rather than shared per-token endpoints. Truss packages models and supports TensorRT-LLM, vLLM, SGLang, and custom serving components. For DeepSeek V3, Baseten used SGLang and had to support FP8 blockwise kernels, H200 or multi-node H100 deployments, and model-performance debugging despite slow load times. At platform level, Baseten frames mission-critical inference as three pillars: model-level performance, horizontal scaling infrastructure across replicas/regions/clouds, and developer experience for multi-step workflows. KV-cache-aware load balancing routes requests based on replica cache state, queue depth, and location. Truss Chains lets independently autoscaled models stream data between steps, reducing network round trips for workflows such as AI phone calls.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Workflow["Customer workflow"] --> Deployment["Customer model deployment"]
  Deployment --> Truss["Truss packaging"]
  Truss --> Framework["Serving framework layer"]
  Framework --> Replicas["Dedicated inference replicas"]
  Replicas --> KV["KV cache-aware load balancer"]
  Replicas --> Cloud["Multi-cloud scaling plane"]
  Truss --> Chains["Truss Chains"]
  Chains --> Replicas

Components

Data flow

A customer deploys a custom or open model through Truss onto dedicated Baseten inference resources. The platform selects and customizes a serving framework such as SGLang, TensorRT-LLM, or vLLM for the workload. Requests are routed across dedicated replicas with KV-cache-aware load balancing and may scale across clouds or regions. For multi-step workloads, Truss Chains makes one external call and streams data directly between hosted models such as transcription, LLM response generation, and text-to-speech.

Why this architecture

Baseten argues that production inference is more than picking an engine. Model-level optimizations solve only one part of the problem; the platform also needs fast horizontal scaling, multi-region or multi-cloud capacity, and workflow-level developer experience so customers can run latency-sensitive products without noisy-neighbor or network-hop failures.

Models used: DeepSeek V3 · SGLang · TensorRT-LLM · vLLM · Llama 70B · Whisper fine-tunes · speculative decoding draft models · Medusa heads

Scale

• DeepSeek V3 has 671 billion parameters.
  “it's 671 billion parameter Moe and I I I think it's a game changer for the open source AI”transcript, stated
• An 8-card H100 node provides 640 GB of memory, which is not enough for DeepSeek V3 FP8 weights plus KV cache.
  “even we we use the h008 cards it should be 640 GB memory so D6 V3 model it has uh 671 billion weights”transcript, stated
• Some models must scale to hundreds of replicas, each potentially using multiple H100s or a full node.
  “a single model might want to horizontally scale up to 200 replicas Each of which is let's say 2 h100s or 4 h100s or even a full node”transcript, stated
• Baseten gives examples of one model spread across 50 replicas in GCP East, 80 in AWS West, and more in Oracle London.
  “50 replicas in gcp East and uh 80 replicas in in uh AWS West and and Oracle in in London Etc”transcript, stated
• Truss Chains can support AI phone calls with sub-400 ms end-to-end latency.
  “going back to the AI phone call use case you can get sub 400 millisecond latency AI phone calls that actually feel very realistic”transcript, stated
• SGLang grew from about 2,000 to more than 7,000 GitHub stars after mid-2024 involvement.
  “when I participated in the sgan team at July it has only 2,000 stars and right now it has more than seven stars”transcript, stated

Alternatives considered (roads not taken)

Offer shared inference endpoints for popular open-source models

Rejected because: Baseten intentionally serves customers with custom models, strict latency/compliance requirements, and dedicated-resource needs instead of shared per-token API endpoints.

Quantize customer models behind the scenes

Rejected because: Baseten treats model quality as a customer requirement and only quantizes in conjunction with customer engineers and evaluations.

Treat one serving framework as the whole production answer

Rejected because: Framework choice is only part of model-level performance, while mission-critical inference also requires scaling infrastructure and workflow developer experience.

Orchestrate multi-model voice workflows as separate external API calls

Rejected because: Network round trips between models made latency too high for realistic phone-call applications.

Patterns

model-routing (operations)

Baseten selects TensorRT-LLM, SGLang, vLLM, or custom runtime pieces based on latency, throughput, customization, and model requirements.

latency-optimization (performance)

The platform optimizes model runtime, time-to-first-token, KV-cache routing, and multi-model workflow streaming for low-latency products.

cost-aware-routing (operations)

Requests are routed across replicas by KV cache, queue depth, location, committed customer resources, and optional overflow to Baseten cloud.

structured-output-contract (generation)

SGLang supports constrained decoding by converting JSON schemas to finite state machines that control output and can skip predictable token paths.

model-distillation (model_optimization)

Speculative decoding can use draft models or Medusa heads, but the throughput gains depend on training those smaller predictors well.

fine-tuning-adaptation (model_optimization)

Baseten supports fine-tuned custom models and works with customers on quantization and evaluations rather than altering weights silently.

Tradeoffs

Use dedicated inference instead of shared endpoints

Chose Customer-dedicated resources and consumption-based pricing over Shared open-source model endpoints priced per token.

Rationale: Baseten targets customers with custom models, compliance needs, latency guarantees, and noisy-neighbor intolerance.

Invest in SGLang for DeepSeek and customization

Chose SGLang for performance plus developer-friendly extension over Only TensorRT-LLM or vLLM for every workload.

Rationale: TensorRT-LLM can be best for latency-sensitive cases, but SGLang was easier to customize and had DeepSeek-specific optimizations.

Stream hosted models inside Truss Chains

Chose One external call with internal model-to-model streaming over Separate customer-orchestrated API calls for each model step.

Rationale: The streaming architecture removes network round trips that would otherwise make voice-call workflows feel too slow.

Expose runtime details transparently to customers

Chose Transparent provider posture over Opaque 'just trust our magic' deployment.

Rationale: Foundation-model companies and advanced customers care about how quality, latency, and compliance are achieved under the hood.

Gotchas (what bit them)

Huge open-weight models may not fit on familiar GPU node shapes.

Root cause: DeepSeek V3's 671B FP8 weights alone exceed 8-card H100 memory before adding KV cache.

Resolution: Baseten used H200 GPUs or multi-node H100 configurations and implemented missing FP8 support.

Debugging very large models can be slow even when the bug is not conceptually hard.

Root cause: Model load time dominates performance benchmarking and debugging loops.

Resolution: The team treats large-model debugging as slow operational work rather than assuming the implementation is inherently complex.

Kubernetes autoscaling is not enough for mission-critical LLM inference.

Root cause: LLM replicas need fast scale-out, GPU capacity across regions/clouds, and stable P95/P99 behavior under bursts.

Resolution: Baseten rebuilt infrastructure pieces for horizontal scaling and multi-cloud model replication.

Framework support for speculative decoding does not automatically produce throughput wins.

Root cause: Acceptance rate depends on training or fine-tuning the draft model or additional heads with the right data.

Resolution: Baseten treats speculative decoding as model work plus framework support, not just a flag.

Evaluation

Baseten evaluates inference through customer-specific quality preservation, latency and time-to-first-token targets, P95/P99 behavior under bursts, throughput, compliance/geography constraints, quantization evals with customer teams, and throughput improvements from techniques such as speculative decoding and cache-aware load balancing.

Metrics: time to first token · P95 and P99 latency · throughput under burst traffic · sub-400 ms AI phone call latency · KV cache hit behavior · quality preservation after quantization or runtime changes · region and compliance placement

Outcomes

• Baseten served DeepSeek V3 using SGLang with FP8 and DeepSeek-specific optimizations.
• The platform supports customers with dedicated inference rather than shared noisy-neighbor endpoints.
• KV-cache-aware load balancing improved latency compared with random replica assignment.
• Multi-cloud capabilities let a single model spread across customer and Baseten resources in multiple regions.
• Truss Chains enables sub-400 ms AI phone-call workflows by streaming data between hosted model steps.
• SGLang's community and feature set grew quickly, including DeepSeek optimizations and routing features.

The transcript cites DeepSeek V3 at 671B parameters, H100 nodes with 640 GB memory, examples of 200 replicas, cross-cloud deployments with 50 and 80 replicas in named regions, and sub-400 ms voice-call latency.