User Foundation Models for Personalization at Scale

grab · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Superapp personalization and recommendation infrastructure

Grab needed a reusable user representation layer that could capture multimodal, sequential behavior across mobility, delivery, financial, merchant, and driver ecosystems without relying on siloed task-specific feature engineering.

Constraints:

• Grab operates across diverse service verticals whose user signals interact across sessions.
• Legacy personalization relied on hundreds to thousands of manually engineered task-specific features.
• The model had to handle text, numeric, categorical ID, and location modalities.
• Massive ID vocabularies with hundreds of millions of entities made naive output heads infeasible.
• Training and inference had to scale to terabytes of data and millions of daily embeddings.

Architecture

Grab built a transformer-based user foundation model that consumes tabular profile data and sequential clickstream events through a shared key-value token format. Modality-specific adapters encode text, numerical values, IDs, and locations, alignment layers project them into one vector space, and the transformer pre-trains with masked reconstruction plus next-action prediction. Ray-based CPU/GPU pipelines train on terabytes of data and regenerate fresh long-term and short-term embeddings for downstream personalization systems.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  raw-superapp-data["Raw superapp data"] --> key-value-tokenizer["Key-value tokenizer"]
  key-value-tokenizer --> modality-adapters["Modality adapters"]
  modality-adapters --> alignment-layer["Alignment layer"]
  alignment-layer --> transformer-foundation-model["Transformer foundation model"]
  transformer-foundation-model --> pre-training-heads["Pre-training heads"]
  pre-training-heads --> hierarchical-id-classifier["Hierarchical ID classifier"]
  transformer-foundation-model --> user-id-adapter["User ID adapter"]
  transformer-foundation-model --> sequence-aggregation-module["Sequence aggregation module"]
  user-id-adapter --> downstream-personalization-models["Downstream personalization models"]
  sequence-aggregation-module --> downstream-personalization-models
  ray-cpu-workers["Ray CPU workers"] --> key-value-tokenizer
  ray-cpu-workers --> ray-gpu-workers["Ray GPU workers"]
  ray-gpu-workers --> transformer-foundation-model
  ray-gpu-workers --> downstream-personalization-models

Components

Data flow

Grab tokenizes user attributes and recent events as key-value pairs, sends each token through a modality adapter, aligns all vectors into a shared representation space, and feeds them into the transformer. Pre-training reconstructs masked tokens and predicts next actions, while Ray separates CPU tokenization from GPU model computation. The resulting long-term and short-term embeddings are served to ad optimization, recommendations, fraud, churn, and other downstream models.

Why this architecture

Grab chose a foundation-model architecture because a superapp creates cross-vertical behavior that task-specific models miss. The shared key-value token format and modality adapters let one transformer learn from unordered profile data and ordered clickstreams, while dual embeddings give downstream systems both persistent preference and current-session intent.

Models used: Custom transformer foundation model · Masked language modeling objective · Next-action prediction objective · Pre-trained language models for text adapters · Ray Data distributed inference

Scale

• Grab operates in over 800 cities.
  “Grab, the Southeast Asian superapp operating across mobility, food delivery, and financial services in over 800 cities”summary, stated
• The system handles vocabularies with hundreds of millions of unique users, merchants, and drivers.
  “With hundreds of millions of unique merchants, users, and drivers, a standard cross-entropy loss would require prediction layers with enormous output dimensions.”summary, stated
• A flat 100 million ID vocabulary with 768-dimensional embeddings would imply nearly 80 billion parameters.
  “a vocabulary of 100 million IDs with 768-dimension embeddings would create a prediction head with nearly 80 billion parameters”summary, stated
• Training processes terabytes of multimodal data.
  “massive-scale training involving terabytes of diverse multimodal data requiring distributed computing frameworks”summary, stated
• Daily inference regenerates embeddings for millions of active users.
  “high-throughput inference necessitating regeneration of embeddings for millions of active users daily.”summary, stated
• Batch inference processes thousands of users simultaneously.
  “a batch-oriented approach that processes thousands of users' data simultaneously, maximizing throughput.”summary, stated

Alternatives considered (roads not taken)

Continue manually engineered task-specific features

Rejected because: Siloed features required substantial effort and missed cross-vertical sequential behavior.

Semi-supervised training on one downstream objective

Rejected because: Single-objective training would bias embeddings toward one vertical and limit utility elsewhere.

Flat ID prediction over all entities

Rejected because: Hundreds of millions of IDs would make output heads computationally intractable.

Bundle CPU preprocessing with GPU computation

Rejected because: Bundling would leave expensive GPUs idle during tokenization.

Patterns

model-distillation (model architecture)

Grab turns diverse superapp behavior into reusable embeddings that distill long-term and short-term user signals for many downstream models.

batch-offline-processing (infrastructure)

Grab uses Ray distributed batch training and inference pipelines to process terabytes of data and regenerate embeddings for millions of users.

context-engineering (representation)

The model context is engineered through key-value tokenization, custom positional embeddings, attention masks, and modality adapters that preserve tabular versus sequential semantics.

latency-optimization (infrastructure)

Ray decouples CPU preprocessing from GPU computation to keep expensive accelerators fed and reduce cost/time.

data-quality-loop (data operations)

Grab identifies ongoing corpus unification and enrichment as necessary for reliable foundation-model behavior across downstream applications.

Tradeoffs

Unsupervised ecosystem-wide pre-training

Chose Masked reconstruction and next-action prediction over all Grab verticals over Semi-supervised training on one downstream objective.

Rationale: The foundation model needed general embeddings useful across heterogeneous tasks.

Dual long-term and short-term embeddings

Chose Serve both stable persona embeddings and recent-intent sequence embeddings over A single user representation.

Rationale: Different downstream tasks need different balances between historical behavior and current intent.

Modular adapters

Chose Specialized encoders per modality plus an alignment layer over One shared encoder for all text, IDs, numbers, and locations.

Rationale: Different modalities needed different encoders while still contributing to one shared representation space.

Gotchas (what bit them)

Flat categorical ID reconstruction becomes computationally infeasible at marketplace scale.

Root cause: Hundreds of millions of user, merchant, and driver IDs imply enormous prediction heads.

Resolution: Grab used hierarchical classification to predict meaningful groups first, then smaller subgroups.

Task-specific objectives would bias a superapp foundation model.

Root cause: Grab's downstream tasks span food, transport, financial services, ads, fraud, and churn.

Resolution: The team used unsupervised pre-training across the full ecosystem.

CPU preprocessing can starve GPU training and inference workers.

Root cause: Complex multimodal tokenization is CPU-intensive while transformer computation is GPU-intensive.

Resolution: Grab separated CPU and GPU worker pools under Ray.

A shared embedding product can propagate data-quality defects to many teams.

Root cause: Downstream applications reuse the same foundation embeddings.

Resolution: Grab treats dataset unification and low-noise behavior views as ongoing operational work.

Evaluation

The case study reports architectural reasoning and downstream adoption but discloses limited quantitative lift metrics, leaving cost-benefit and performance comparisons as open evaluation gaps.

Metrics: Terabytes of data processed · Millions of daily embeddings · Hundreds of millions of categorical IDs · No disclosed CTR/conversion/fraud lift metrics

Outcomes

• Grab created reusable long-term and short-term embeddings for downstream personalization.
• Initial embeddings are already used for ad optimization, dual app prediction, fraud detection, churn probability estimation, and recommendations.
• The architecture reduces duplicated feature-engineering effort by making embeddings a platform product.
• The public case study does not provide concrete downstream lift numbers.

The model serves a superapp spanning more than 800 cities, terabytes of multimodal behavior data, millions of daily embeddings, and categorical vocabularies with hundreds of millions of entities.