Scaling Multimodal AI for Autonomous Trucking with Ray

torc_robotics · Automotive · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: autonomous vehicle multimodal ML infrastructure

Torc Robotics needed to scale AV 3.0 multimodal training, auto-labeling, simulation, and reinforcement-learning workloads without maintaining fragmented pipelines, starving expensive GPUs, or forcing model developers to own distributed-systems tuning.

Constraints:

• The system had to support perception, prediction, planning, open-loop training, auto-labeling, closed-loop simulation, and reinforcement learning.
• Large multimodal workloads mixed CPU-heavy data preparation with GPU-heavy model operations.
• Model developers needed reusable transform contracts while the MLOps team handled scaling and optimization.

Architecture

Torc standardized on Ray and a transform-centric architecture. Common cloud/compute layers, Ray primitives, and shared transform libraries compose into graph-based training, auto-labeling, and simulation frameworks. CPU-bound transforms run horizontally on cheaper CPU instances, GPU transforms stay on GPU nodes, dashboards expose bottlenecks, and recurring tuning lessons are codified into agentic skills.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Multimodal sensor data"] --> B["Shared transform library"]
  B --> C["Ray scheduling layer"]
  C --> D["CPU worker pool"]
  D --> E["GPU worker pool"]
  C --> E
  B --> F["Training graph"]
  B --> G["Auto-labeling graph"]
  B --> H["Simulation graph"]
  F --> L["Model loop"]
  G --> L
  H --> L
  C --> I["Observability dashboards"]
  I --> J["MLOps team"]
  J --> K["Agentic optimization skills"]

Components

Data flow

Sensor and scenario data flow through shared transform graphs: data reading, voxelization, preparation, training or inference, simulation feedback loops, and output sinks. Ray schedules those transforms across heterogeneous CPU/GPU resources, while dashboards and throughput callbacks guide bottleneck tuning.

Why this architecture

Torc chose Ray-backed transform graphs because earlier AV systems accumulated duplicated, incompatible training/testing/data pipelines, while Ray let them separate reusable components from scheduling and scale CPU/GPU work independently.

Models used: AV perception models · object detection model · point cloud voxelization models · reinforcement learning policies

Scale

• A showcased workload streamed roughly 91,000 multimodal scenes.
  “it has roughly 91,000 multimodal scenes stream streaming through”transcript, stated
• Torc ran 32 GPUs on cheaper or more available G5.16x/G5.24x instances while preserving throughput.
  “we could also schedule 32 GPUs on much cheaper G516x or much more available G524X large instances and still be able to get the same throughput”transcript, stated
• The architecture was built for four core workload families across training and simulation.
  “wrangle all of these various systems and libraries into a cohesive system that can be used for all four of our core uh kind of applications”transcript, stated

Alternatives considered (roads not taken)

Keep separate ECS, SageMaker, patch-inference, and bespoke pipelines

Rejected because: They duplicated data formats, metrics, libraries, resource management, and verification processes.

Run CPU-heavy preprocessing on expensive GPU instances

Rejected because: It starved GPUs of prepared data and wasted expensive resources on CPU-bound work.

Optimize by inspecting bespoke model code first

Rejected because: Dashboard-driven bottleneck analysis let the MLOps team tune jobs without understanding every transform implementation.

Patterns

batch-offline-processing (ml-infrastructure)

Batch training and auto-labeling are represented as reusable Ray transform graphs over large multimodal datasets.

observability-tracing (operations)

Dashboards and throughput callbacks identify bottlenecks and separate Ray Data pipeline tuning from model-loop tuning.

tool-use-orchestration (automation)

Recurring optimization workflows are codified as agentic skills that help users diagnose, patch, and redeploy jobs.

Tradeoffs

Infrastructure backend

Chose Ray as a unified scheduling and graph backend over multiple specialized pipeline systems.

Rationale: Ray let Torc reuse transforms across training, auto-labeling, and simulation while scheduling heterogeneous resources.

Resource scheduling

Chose schedule by GPU/CPU type with heterogeneous pools over rigid dense GPU instance types.

Rationale: The same throughput could be achieved on cheaper or more available instances by separating CPU and GPU work.

Developer/infrastructure boundary

Chose model developers write transforms; MLOps tunes jobs over requiring every model developer to be a Ray optimization expert.

Rationale: The shared transform contract let infrastructure specialists optimize from dashboards and configuration.

Gotchas (what bit them)

Fragmented ML systems created platform debt.

Root cause: Training, testing, extraction, annotation, and preparation systems used different backends, formats, metrics, and libraries.

Resolution: Unify workloads around Ray primitives and shared transform libraries.

Expensive GPUs were starved or misused.

Root cause: CPU-heavy IO and preprocessing work was colocated with GPU work, leaving GPUs waiting for data.

Resolution: Distribute CPU-bound transforms horizontally on cheaper CPU instances and feed GPUs a hot stream.

Optimization lessons became tribal knowledge.

Root cause: Repeated tuning patterns lived in team memory rather than reusable automation.

Resolution: Codify recurring bottleneck fixes as agentic skills.

Shuffling and data movement can dominate throughput.

Root cause: Shuffling loaded unstructured data causes expensive cross-node movement.

Resolution: Shuffle metadata before loading unstructured data and avoid global shuffles when possible.

Evaluation

Torc evaluated workloads through throughput, GPU utilization, multi-instance/multi-GPU performance, dashboard bottleneck analysis, and resource utilization across nodes.

Metrics: throughput · GPU utilization · GPU instance count · multi-instance multi-GPU performance · resource utilization · pipeline bottlenecks

Outcomes

• Torc consolidated fragmented AV ML workloads into reusable Ray transform graphs.
• The team reduced GPU starvation by distributing CPU-bound work to cheaper CPU instances.
• The same throughput could be achieved using 32 GPUs on cheaper or more available instance shapes.
• Recurring optimization lessons started becoming agentic skills for diagnose-patch-redeploy loops.

This is MLOps for autonomous trucking rather than a language-agent product, but it contributes reusable infrastructure lessons for multimodal AI systems.