End-to-End Foundation Models for Self-Driving Vehicles at Scale

wayve · Automotive · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous-driving foundation models

Wayve needed self-driving technology that generalized across countries, vehicle types, sensor sets, and use cases without bespoke per-city maps or manually programmed perception/planning rules.

Constraints:

The system must work with many cars and different sensor sets.
Traditional intermediate representations are brittle and expensive to scale geographically.
High-definition maps are difficult to maintain and can go stale after road changes.
Vehicle deployment requires compression into a 75-watt automotive runtime.
The system needs validation and safety-driver supervision during testing.

Architecture

Wayve uses an end-to-end learning architecture centered on a foundation/world model trained on diverse driving and related data. Instead of hand-programming object-level intermediate representations, the model learns driving-relevant structure from data, supports offline tasks such as simulation and scenario classification, and is distilled into a validated driving model compressed to run in production vehicles under a 75-watt power budget.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  fleet-and-partner-data["Fleet and partner data"] --> foundation-world-model["Foundation world model"]
  foundation-world-model --> offline-simulation-and-curation["Offline simulation and curation"]
  foundation-world-model --> driving-model-pipeline["Driving model pipeline"]
  offline-simulation-and-curation --> driving-model-pipeline
  driving-model-pipeline --> compressed-vehicle-model["Compressed vehicle model"]
  compressed-vehicle-model --> production-vehicle-runtime["Production vehicle runtime"]
  production-vehicle-runtime --> fleet-and-partner-data

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Fleet and partner data`	Driving and related data gathered across vehicles, countries, sensors, OEMs, and partners.	`Foundation world model`
`Foundation world model`	Learns holistic geometry, kinematics, and scene understanding from diverse data.	`Offline simulation and curation`, `Driving model pipeline`
`Offline simulation and curation`	Uses next-frame prediction and scenario classification to organize data and power simulation.	`Driving model pipeline`
`Driving model pipeline`	Produces and validates the vehicle driving model, including coverage of relevant traffic signs and scenarios.	`Compressed vehicle model`
`Compressed vehicle model`	Single deployed model that drives, selects trajectories, and emits auxiliary safety signals within vehicle constraints.	`Production vehicle runtime`
`Production vehicle runtime`	Runs the compressed model on the car across locations, cars, and sensor sets.	`Fleet and partner data`

Data flow

Driving data from many cars, countries, sensors, and partners trains a foundation/world model. The foundation model supports multiple offline tasks, including next-frame prediction, simulation, scenario classification, and curation, then a driving model is produced, validated, compressed, and deployed as a single vehicle model.

“we are fitting this foundation model many many sources of diverse data.”transcript, stated

“Then um the output of this model can also be multiple things. One can be driving but it can be something else. U it can be a next frame prediction.”transcript, stated

Why this architecture

Wayve chose end-to-end learned driving to avoid brittle manually maintained state and intermediate object models. A single trained network can run across countries and cars, while the foundation-model layer supports simulation, curation, and downstream driving-model validation.

“rather than um understanding what every object does. Uh the car should learn to drive from example like as a human.”transcript, stated

“So in contrast what we do is we have a network that we train and then we run it everywhere. we have one model that runs across countries across cars.”transcript, stated

Models used: foundation world model · compressed driving model · next-frame prediction model · scenario classification model

Scale

The target deployment spans many cars and different sensor sets.
“So we have to work with many cars. We have to work with different sensor sets and so on.”transcript, stated
Wayve expanded from London to Japan, Europe, the UK, and the US.
“A year ago we were driving primarily in London. Now our cars drive across the world. Uh Japan, Europe, UK, US.”transcript, stated
The city target increased from 100 cities to 500 cities.
“this year we started on with a goal to be in 100 cities then we raised it to 500 cities.”transcript, stated
The validation curriculum must cover 2,000 traffic signs in Europe.
“there are 2,000 uh traffic signs in Europe. So it has to see them.”transcript, stated
The deployed model must fit in a 75-watt vehicle power budget.
“then you need to deploy. So you need to compress it to 75 watts.”transcript, stated

Alternatives considered (roads not taken)

Build bespoke autonomous-driving stacks per car and sensor set.

Rejected because: Wayve needed one data-trained model to generalize across vehicles, countries, and sensor configurations.

“This is counter to where the industry is at now where there's quite a bit of bispoke development for every car for every sensor set.”transcript, stated

Use manually managed intermediate representations such as object lists and behavior models.

Rejected because: The speaker argues that programmatically managed intermediate state becomes a house of cards and is hard to expand.

“if you try to manage the state programmatically you're going to end up with a house of cards. It's very hard. It will work in one place but it's very hard to expand.”transcript, stated

Rely on high-definition maps.

Rejected because: High-definition maps are hard to maintain and can miss road changes after a map is finalized.

“we gave that up because high definitionition maps are hard to maintain. Uh and you never know whether there will be a change after the map was finalized.”transcript, stated

Deploy the full foundation model directly to the car.

Rejected because: The foundation model may be too large and must be compressed into the constrained vehicle runtime.

“now the foundation model can be large it may not fit in a car and it's not necessarily tuned to drive in a particular location.”transcript, stated

Patterns

`fine-tuning-adaptation` (modeling)

Wayve adapts a foundation model into a validated driving model through data curriculum and local driving data.

“then we produce and validate a driving model that you know there are 2,000 uh traffic signs in Europe. So it has to see them.”transcript, stated

`model-distillation` (modeling)

The foundation model may be too large for the car, so the production model is compressed to the vehicle power budget.

“then you need to deploy. So you need to compress it to 75 watts.”transcript, stated

`data-quality-loop` (data)

Deployment, data collection, learning, and iteration are framed as the way to scale physical AI across environments.

“You need to deploy at scale and then you have to learn from it and iterate through this.”transcript, stated

`batch-offline-processing` (data)

Offline models and foundation-model tasks organize data, classify scenarios, and power simulation before the single car model is deployed.

“All these models are only used offline. When we go to the car, we only deploy one model.”transcript, stated

Tradeoffs

Driving stack representation

Chose End-to-end learned driving. over Object-list and rule-based intermediate representations..

Rationale: The learned approach avoids hand-managed state and lets the model decide what matters to the driving task.

“we're letting the model figure out what it needs to pay attention to and of course we validate it.”transcript, stated

Map dependency

Chose No high-definition maps. over Precisely located HD maps..

Rationale: Avoiding HD maps made city-to-city scaling an operations problem rather than a map-maintenance dependency.

“we don't have um high definitionition maps. Uh this industry started with high definitionition maps being very precisely located.”transcript, stated

Deployment model

Chose One compressed vehicle model. over Deploying multiple offline models or the full foundation model in the car..

Rationale: The production runtime needs one reliable, power-constrained model while richer models can remain offline.

“When we go to the car, we only deploy one model. That model drives, it does all the driving tasks.”transcript, stated

Gotchas (what bit them)

Intermediate representations can become brittle at global scale.

Root cause: Programmatically managed state works in one place but is hard to expand across countries and contexts.

Resolution: Train a network end to end and run one model across countries and cars.

“It's very hard. It will work in one place but it's very hard to expand.”transcript, stated

HD maps introduce maintenance risk.

Root cause: Road changes such as lane closures can happen after a map is finalized.

Resolution: Give up HD maps and scale operations around learned driving instead.

“you never know whether there will be a change after the map was finalized. Maybe it's a lane closure that that has to be dealt with.”transcript, stated

A world model does not automatically satisfy vehicle-runtime constraints.

Root cause: The foundation model may be large and not tuned to a particular deployment location.

Resolution: Produce, validate, and compress a driving model for the car.

“the foundation model can be large it may not fit in a car and it's not necessarily tuned to drive in a particular location.”transcript, stated

Geographic expansion exposes unseen traffic conventions.

Root cause: A model trained in the UK may not have seen Japan-specific signs such as triangular stop signs.

Resolution: Use local data, safety-driver testing, data partnerships, and curriculum coverage checks.

“if you've never seen a triangular stop sign, then well, how would the car know? So, it has to learn.”transcript, stated

Evaluation

Wayve validates via supervised road testing, data curriculum coverage, simulation/scenario tooling, and production deployment constraints rather than reporting a single benchmark score.

Metrics: Goal raised from 100 to 500 cities · 2,000 European traffic signs in curriculum coverage · 75-watt deployment budget · Hands-off drive from central London to Birmingham mentioned as an example