Fine-Tuning Transaction Foundation Models with Joint Fusion

nubank · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: financial customer behavior modeling

Nubank wanted transaction foundation models to perform better on specific downstream customer-behavior tasks while still incorporating useful non-sequential tabular features, because general transaction embeddings were not task-optimized and late fusion with tabular models learned embeddings separately from tabular context.

Constraints:

self-supervised transaction embeddings were general-purpose rather than optimized for individual tasks
customer modeling used both sequential transaction history and tabular information such as bureau data
gradient-boosted tree models were strong for tabular data but not differentiable, blocking joint end-to-end optimization
DNN-based tabular models varied drastically across problems and initially underperformed GBT baselines
the solution needed to avoid manual feature engineering from sequential transaction data

Architecture

Nubank starts with a causal transformer pre-trained on transaction data to produce user embeddings. For task-specific adaptation, they add a prediction head and supervised-fine-tune the transformer. To incorporate tabular features, they move beyond late fusion and build a joint-fusion architecture: DCNv2 processes embedded numerical and categorical tabular features, projects them into a low-dimensional feature embedding, concatenates that with the transformer user embedding, and uses an MLP for final prediction while simultaneously fine-tuning the transformer and blending model.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  sequential-transaction-data[Sequential transaction data]
  causal-transaction-transformer[Causal transaction transformer]
  prediction-head[Prediction head]
  tabular-feature-inputs[Tabular feature inputs]
  numerical-and-categorical-embeddings[Numerical and categorical embeddings]
  dcnv2-tabular-network[DCNv2 tabular network]
  joint-fusion-mlp[Joint fusion MLP]
  lightgbm-baseline[LightGBM baseline]
  benchmark-evaluation-suite[Benchmark evaluation suite]
  sequential-transaction-data --> causal-transaction-transformer
  causal-transaction-transformer --> prediction-head
  causal-transaction-transformer --> joint-fusion-mlp
  tabular-feature-inputs --> numerical-and-categorical-embeddings
  numerical-and-categorical-embeddings --> dcnv2-tabular-network
  dcnv2-tabular-network --> joint-fusion-mlp
  joint-fusion-mlp --> benchmark-evaluation-suite
  lightgbm-baseline --> benchmark-evaluation-suite

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Sequential transaction data`	Customer transaction history used by the foundation model to learn behavioral embeddings.	`Causal transaction transformer`
`Causal transaction transformer`	Pre-trained transaction foundation model that produces the final-token user embedding and is fine-tuned for downstream tasks.	`Prediction head`, `Joint fusion MLP`
`Prediction head`	Linear layer used in standard supervised fine-tuning for binary classification, multi-class classification, or regression targets.	`Causal transaction transformer`
`Tabular feature inputs`	Non-sequential numerical and categorical customer features such as bureau information.	`Numerical and categorical embeddings`
`Numerical and categorical embeddings`	Periodic numerical embeddings and trainable categorical embedding tables used to make DNN tabular modeling competitive.	`DCNv2 tabular network`
`DCNv2 tabular network`	Processes embedded tabular features and projects them into a low-dimensional feature embedding.	`Joint fusion MLP`
`Joint fusion MLP`	Combines transformer user embeddings with tabular feature embeddings for the final prediction during joint fine-tuning.	`Benchmark evaluation suite`
`LightGBM baseline`	Gradient-boosted tree baseline trained on tabular features for comparison.	`Benchmark evaluation suite`
`Benchmark evaluation suite`	Measures relative AUC gains across internal benchmark tasks for fine-tuning, DNN variants, late fusion, and joint fusion.	`Joint fusion MLP`, `LightGBM baseline`

Data flow

Sequential transaction data enters the causal transformer, whose final token embedding becomes the user embedding. Tabular numerical and categorical features are embedded and processed through DCNv2, then projected into a feature embedding. The feature embedding is concatenated with the transformer user embedding, an MLP predicts the task label, and supervised fine-tuning jointly updates the transformer and blending model to optimize the downstream loss.

“The input to this prediction head is the final token embedding, denoted the user embedding, from the output of the causal transformer.”article, stated

“This feature embedding is concatenated with the transformer-based embedding, and a multi-layer perceptron makes the final prediction.”article, stated

Why this architecture

Nubank chose joint fusion because late fusion combines tabular features with embeddings learned separately, while joint fusion lets the transformer learn task-specific transaction representations in the presence of tabular features. A DNN tabular stack was required because GBTs are not differentiable and therefore cannot be jointly optimized with the transformer.

“late fusion is suboptimal because the finetuned embeddings are learned separately from the tabular features.”article, stated

“Unfortunately, GBTs are not differentiable and, hence, are incompatible with joint fusion.”article, stated

Models used: causal transformer transaction foundation model · DCNv2 · LightGBM · XGBoost

Scale

The article is the third part of Nubank's transaction foundation model series.
“This is the third part of a series of blog posts about modeling customer finances through foundation models.”article, stated
Standard supervised fine-tuning produced a 1.68% relative AUC improvement across several benchmark tasks.
“the right-hand figure below shows the 1.68% relative improvement in AUC achieved by finetuning across several benchmark tasks.”article, stated
A cited survey evaluated 19 tabular models across 176 datasets, motivating caution about one-size-fits-all DNN tabular modeling.
“one survey paper [5] evaluated 19 tabular feature models (NN + GBT) on 176 datasets”article, implied
Initial DCNv2 tabular-only results were 0.40% worse than GBTs before numerical/categorical embedding improvements.
“initial results showed much worse performance (-0.40%) for the DNN-based DCNv2 models compared to the GBTs.”article, stated

Alternatives considered (roads not taken)

use general pre-trained transaction embeddings as-is

Rejected because: the embeddings were not optimized for any particular downstream task.

“These embeddings are not optimized for any particular task and can be applied to a variety of problems.”article, stated

late fusion with GBT models

Rejected because: late fusion learns the embeddings separately from tabular features, limiting task-specific optimization.

“Alongside these tabular features, finetuned embeddings can be passed into the GBT training, a process denoted late fusion [8].”article, stated

jointly optimize GBTs with the transformer

Rejected because: gradient-boosted trees are not differentiable and therefore incompatible with joint fusion.

“Unfortunately, GBTs are not differentiable and, hence, are incompatible with joint fusion.”article, stated

use a plain DCNv2 tabular network without numerical embedding improvements

Rejected because: initial DCNv2-only tabular results were worse than GBTs.

“initial results showed much worse performance (-0.40%) for the DNN-based DCNv2 models compared to the GBTs.”article, stated

Patterns

`fine-tuning-adaptation` (model-operations)

Nubank fine-tunes a pre-trained transaction transformer with a task-specific prediction head so the user embedding is optimized for the downstream label.

“Our supervised fine-tuning process is a secondary step in which we take a pre-trained model and add a linear layer, called the prediction head”article, stated

`eval-driven-development` (evaluation)

Architecture choices were compared through relative AUC gains across internal benchmark tasks, including tabular-only DNN variants and late versus joint fusion.

“we visualize the relative gain in AUC when using joint fusion vs late fusion on the same benchmark tasks as above.”article, stated

Tradeoffs

Supervised-fine-tune transaction embeddings for each task.

Chose task-specific transformer and prediction-head optimization over using general unsupervised embeddings unchanged.

Rationale: Task-specific fine-tuning tailors the embedding to the target label and improves downstream AUC.

“We then optimize the transformer simultaneously with the prediction layer to minimize the loss (i.e., cross entropy, MSE).”article, stated

Use joint fusion instead of late fusion.

Chose simultaneous transformer and blending-model optimization over passing separately learned embeddings into a GBT.

Rationale: Joint optimization lets the transformer learn signals not already present in tabular features.

“In contrast to late fusion, we propose a _joint fusion-_based [8] training procedure.”article, stated

Use DCNv2 with embedded tabular features.

Chose differentiable DNN tabular network with numerical and categorical embeddings over GBT blending for joint training.

Rationale: Differentiability enables joint fusion, while the embedding strategy closed the tabular-performance gap with GBTs.

“Incorporating this embedding strategy into the DCNv2 model allowed us to achieve parity with GBTs on many of our internal problems.”article, stated

Gotchas (what bit them)

General transaction embeddings were useful but not task-optimal.

Root cause: Self-supervised pre-training learns broad behavioral representations rather than the loss for a specific classification or regression target.

Resolution: Nubank added a prediction head and supervised-fine-tuned the transformer for downstream labels.

“if we want to achieve optimal performance on a specific task, we can further refine the transformer and its embeddings through a process called supervised fine-tuning.”article, stated

Late fusion left transaction embeddings blind to tabular context during fine-tuning.

Root cause: Embeddings and tabular blending were trained separately.

Resolution: Joint fusion simultaneously optimizes the transformer and the blending model.

“This procedure jointly optimizes the transformer alongside the blending model, enabling the transformer to better capture information not present in the tabular features.”article, stated

Strong GBT tabular models could not participate in end-to-end joint fusion.

Root cause: GBTs are not differentiable.

Resolution: Nubank invested in DNN-based tabular feature networks and selected DCNv2 with numerical and categorical embeddings.

“Motivated by this incompatibility, we invested in DNN-based tabular feature networks.”article, stated

DNN tabular models were inconsistent across problems.

Root cause: Tabular model performance varies drastically by dataset and architecture.

Resolution: The team added periodic numerical embeddings, categorical embeddings, regularization, and transformer-embedding normalization to make DCNv2 reliable enough.

“the performance of DNN tabular feature networks can vary drastically between problems.”article, stated

Evaluation

Nubank evaluated supervised fine-tuning and fusion architectures using relative AUC gains across internal benchmark tasks, comparing fine-tuned transaction embeddings, tabular-only LightGBM baselines, DNN tabular variants, late fusion, and joint fusion.

Metrics: 1.68% relative AUC improvement from supervised fine-tuning across benchmark tasks · -0.40% initial DCNv2 performance compared with GBT tabular models · relative AUC gain for joint fusion versus late fusion · relative AUC gains across several DNN tabular model versions

“the right-hand figure below shows the 1.68% relative improvement in AUC achieved by finetuning across several benchmark tasks.”article, stated

“This demonstrates the advantage of finetuning jointly with tabular features.”article, stated

Outcomes

Supervised fine-tuning improved AUC by 1.68% relative across several benchmark tasks.
DCNv2 with numerical and categorical embeddings reached parity with GBTs on many internal tabular problems.
Joint fusion improved performance over late fusion on the same benchmark tasks.
The joint-fusion lift came from learning informative task features during fine-tuning rather than adding new data sources.
The approach combines sequential transaction data and tabular features in one end-to-end training process.

The post reports internal benchmark tasks rather than public production volumes; the scale signal is architectural reuse across Nubank transaction-foundation-model tasks.

“the improvement is achieved by automatically learning informative features for the task at hand through finetuning our transaction foundation models.”article, stated

“joint fusion—a method for combining sequential transaction data with tabular features in a single, end-to-end training process.”article, stated