Building a Hybrid Cloud AI Infrastructure for Large-Scale ML Inference

roblox · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale ML and LLM inference infrastructure

Roblox needed unified AI infrastructure that could scale from fragmented team-specific ML pipelines to hundreds of production inference pipelines across personalization, search, safety, translation, assistant features, and generative creation.

Constraints:

• early teams built mini-platforms with disparate frameworks and duplicated feature engineering
• Roblox serves an always-on global platform with millions of daily users
• inference workloads span batch, online, CPU, GPU, cloud, and on-prem environments
• feature serving needs high throughput, low latency, cost efficiency, and rapid iteration
• generative workloads require token-aware throttling, load balancing, observability, and LLM-specific serving

Architecture

Roblox evolved through a three-phase infrastructure build. First it adopted Kubeflow, notebooks, a model registry, KServe/Triton serving, and roblox-ml to standardize training and deployment. Second it added Ray for scalable batch inference, moved CPU inference into data centers, and built a custom Feast/Flink-backed feature store. Third it added an ML gateway for centralized large-model access and a vLLM-based LLMOps platform for high-throughput LLM serving across hybrid cloud and on-prem infrastructure.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  jupyter-notebooks[Jupyter notebooks] --> roblox-ml-library[roblox-ml library]
  roblox-ml-library --> kubeflow-pipelines[Kubeflow pipelines]
  kubeflow-pipelines --> model-registry[Model registry]
  model-registry --> kserve-triton-serving[KServe Triton serving]
  ray-batch-inference[Ray batch inference] --> custom-feature-store[Custom feature store]
  custom-feature-store --> kserve-triton-serving
  ground-truth-team[Ground truth team] --> kubeflow-pipelines
  custom-feature-store --> ml-gateway[ML gateway]
  ml-gateway --> vllm-platform[vLLM platform]
  ml-gateway --> ai-applications[AI applications]
  vllm-platform --> ai-applications
  kserve-triton-serving --> ai-applications

Components

Data flow

Engineers develop and package models with notebooks, roblox-ml, Kubeflow pipelines, and a registry. Batch workloads run through Ray, online models use KServe/Triton and feature-store data, and CPU inference can run in Roblox data centers. Large-model calls route through the ML gateway, which centralizes API management, token throttling, latency-aware regional load balancing, usage tracking, and observability. vLLM serves LLM workloads with improved latency and throughput.

Why this architecture

Roblox centralized infrastructure because team-specific mini-platforms created duplicated feature engineering and scaling work. Hybrid placement gives control over privacy, latency, and cost; Ray handles batch scale; the feature store serves both streaming and batch features; the ML gateway turns many large models into a governed shared service; and vLLM improves LLM serving efficiency.

Models used: vLLM-served LLMs · Roblox Assistant models · voice safety model · translation models · recommendation and search models

Scale

• Roblox grew from fewer than 50 inference pipelines in early 2023 to about 250.
  “In early 2023, we supported fewer than 50 machine learning (ML) inference pipelines. Today, our infrastructure supports approximately 250 of these pipelines.”article, stated
• Roblox maintains tens of thousands of CPUs and more than a thousand GPUs across two data centers and hybrid cloud infrastructure.
  “We maintain tens of thousands of CPUs and more than a thousand GPUs across two data centers and our hybrid cloud infrastructure”article, stated
• Roblox served 79.5 million daily active users as of June 30, 2024.
  “We process approximately 1 billion personalization requests daily for our 79.5 million daily active users (as of June 30, 2024). Moving this workload to our own data centers has helped us maintain our efficiency without compromising user experience.”article, stated
• Personalization processes about 1 billion requests daily.
  “We process approximately 1 billion personalization requests daily for our 79.5 million daily active users (as of June 30, 2024). Moving this workload to our own data centers has helped us maintain our efficiency without compromising user experience.”article, stated
• The feature store ingests about 30 billion records and serves about 70 billion records per day at P99 50ms.
  “Our feature store now ingests approximately 30 billion records and serves approximately 70 billion records per day with a P99 latency of 50ms”article, stated
• Roblox Assistant increased token processing to 1.5 billion tokens per week.
  “With the launch of Roblox Assistant, we’ve seen the number of tokens processed increase to 1.5 billion per week.”article, stated
• vLLM currently serves about 4 billion tokens per week with nearly 2x latency and throughput improvement.
  “Since moving to vLLM, we’ve seen an almost 2x improvement in both latency and throughput, and we currently serve approximately 4 billion tokens per week.”article, stated

Alternatives considered (roads not taken)

Team-specific mini-platforms

Rejected because: They duplicated feature engineering, optimizations, and inference-scaling work.

Third-party feature store

Rejected because: Existing solutions did not meet throughput, latency, cost, and rapid-iteration needs.

Cache personalization requests to reduce cost

Rejected because: Caching would produce stale homepage recommendations for users who visit multiple times per day.

Patterns

batch-offline-processing (infrastructure)

Ray powers distributed batch inference with multistage processing, task parallelism, and fault tolerance.

model-routing (infrastructure)

The ML gateway centralizes large-model access across cloud, on-prem, CPU, GPU, open-source, and internal models.

observability-tracing (operations)

The gateway integrates monitoring tools and usage tracking while models are monitored for latency and accuracy after release.

eval-driven-development (evaluation)

Roblox uses offline experiments, shadow testing, A/B tests, monitoring, and human review before and after model release.

latency-optimization (operations)

vLLM, load balancing, feature-store latency targets, and on-prem CPU inference improve latency and throughput.

cost-aware-routing (operations)

Roblox routes workloads across data centers, cloud, CPUs, GPUs, and vLLM while avoiding stale caching for personalization.

Tradeoffs

Adopt Kubeflow and roblox-ml

Chose standardized pipelines and packaging over each team owning its own deployment path.

Rationale: A common platform made model deployment accessible and reduced duplicated infrastructure work.

Build a custom feature store

Chose Feast plus Roblox-specific DSL and Flink streaming over continuing with third-party feature-store tooling.

Rationale: Roblox needed domain-specific transformations, high throughput, low latency, and freshness.

Centralize large-model access through an ML gateway

Chose common API, throttling, load balancing, key management, usage tracking, and observability over per-application large-model integrations.

Rationale: Generative workloads needed shared governance and resilience across many teams and environments.

Gotchas (what bit them)

Fragmented ML platforms cause duplicated scaling and feature work.

Root cause: Each team built custom feature engineering, frameworks, and inference optimizations.

Resolution: Centralize around Kubeflow, roblox-ml, model registry, feature store, serving, gateway, and vLLM.

Feature freshness and throughput requirements exceeded off-the-shelf tools.

Root cause: Roblox needed very high throughput, low latency, cost efficiency, and rapid iteration across batch and streaming features.

Resolution: Build a custom Feast/Flink-backed feature store.

Caching personalization would reduce cost but harm freshness.

Root cause: Users visit the homepage multiple times a day and need current recommendations.

Resolution: Move CPU inference to Roblox data centers and optimize workload placement instead of relying on stale caches.

Evaluation

Roblox evaluates models and infrastructure with offline experiments, shadow testing, A/B testing, operational monitoring, accuracy monitoring, human moderator review, throughput, latency, and token-volume metrics.

Metrics: approximately 20 A/B tests in parallel for personalization · P99 50ms feature-store serving latency · 1 billion personalization requests daily · 4 billion tokens per week through vLLM · almost 2x latency and throughput improvement

Outcomes

• Roblox scaled from fewer than 50 to about 250 inference pipelines
• custom feature store serves tens of billions of records per day
• Roblox Assistant and other GenAI features pushed token volume into billions per week
• vLLM produced almost 2x improvements in latency and throughput
• the ML gateway centralizes large-model access, security, observability, and load balancing

Roblox reports 79.5 million daily active users, 1 billion personalization requests daily, 30B feature records ingested and 70B served per day, and 4B LLM tokens per week.