BERT-Based Sequence Models for Contextual Product Recommendations

instacart · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Contextual product recommendation retrieval

Instacart needed one contextual retrieval system that could power multiple ads and organic recommendation surfaces using in-session shopping behavior instead of maintaining many ad hoc retrieval systems.

Constraints:

• different surfaces have different downstream ranking goals
• the catalog has millions of products across thousands of retailers
• production vocabularies are much larger than many published BERT4Rec-style datasets
• recommendations must react to in-session user actions in real time
• sequence information, popularity bias, and canonical product identity all affect recommendation quality

Architecture

Instacart built a centralized contextual retrieval layer that uses a BERT-like sequence model trained with masked language modeling over historical product-ID sequences. At inference time, the transformer encodes a session sequence and predicts probabilities over product IDs, returning top-K candidates for downstream ranking and re-ranking across search, item details, cart, pre-checkout, post-checkout, ads, and organic surfaces. The preliminary version uses product-ID sequences only, with future work on ANN scaling, canonical identity, popularity bias, and richer context.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  session-events[Session events] --> product-sequence-encoder[Product sequence encoder]
  historical-interaction-data[Historical interaction data] --> mlm-training-pipeline[MLM training pipeline]
  mlm-training-pipeline --> product-sequence-encoder
  product-sequence-encoder --> next-product-predictor[Next-product predictor]
  next-product-predictor --> unified-retrieval-layer[Unified retrieval layer]
  unified-retrieval-layer --> downstream-rankers[Downstream rankers]
  downstream-rankers --> recommendation-surfaces[Recommendation surfaces]
  historical-interaction-data --> recall-k-evaluator[Recall@K evaluator]

Components

Data flow

User session actions such as cart adds and product page views become ordered product-ID sequences. The BERT-like model encodes the sequence and predicts next-product probabilities over the catalog vocabulary. The retrieval layer returns top-K candidates to downstream ranking/re-ranking systems for many Instacart surfaces, and offline Recall@K evaluates whether the held-out last product is retrieved.

Why this architecture

Instacart unified retrieval because ads and organic systems shared a need for session-aware candidates even though their rankers optimize different goals. A BERT-like MLM setup was chosen after evaluating language-model architectures, and product-ID-only sequences were enough for a strong preliminary system before adding richer features.

Models used: BERT-like model · BERT4Rec · Transformers4Rec · XLNet

Scale

• Instacart uses the system across search, item details, cart, pre-checkout, and post-checkout surfaces.
  “We use this contextual recommendation system across many surfaces at Instacart such as search, item details page, cart, pre-checkout and post-checkout recommendations.”article, stated
• Production models have about 10x more products than BERT4Rec/Transformers4Rec examples with tens of thousands of products.
  “But unlike BERT4Rec and Transformers4Rec, which show results on tens of thousands of products, our production models have an order of 10x more products.”article, stated
• The broader catalog contains millions of products across multiple retailers.
  “Moreover, our catalog has millions of products spread across multiple retailers. Press enter or click to view image in full size”article, stated
• Cart recommendations saw a 30% lift in user cart additions.
  “We also saw a 30% lift in user cart additions when we launched this system on cart recommendations. ## Discussion: Does sequence information matter?”article, stated
• Randomizing sequence training data hurt Recall@K by 10-40%; randomizing evaluation order hurt by 20-45%.
  “Even though we may have gotten meaningful recommendations with completely randomized sequence training data, the relevance metrics (Recall@K) for the model trained on randomized training data were 10–40% worse depending on K.”article, stated

  “Evaluation metrics degraded by 20–45% depending on the value of K. This indicates that during model inference, if we do not have proper sequence information, prediction quality(and recommendations) will suffer.”article, stated

Alternatives considered (roads not taken)

Maintain separate ad hoc retrieval systems per surface

Rejected because: Unified retrieval lowered maintenance cost and deprecated legacy systems.

Use co-occurrence, similarity, and popularity signals only

Rejected because: Next-product prediction was a more principled and empirically successful path.

Ignore product order and treat sessions as product bags

Rejected because: Randomized training/evaluation showed sequence order materially affects quality.

Patterns

retrieval-ranking (recommendations)

The BERT-like model retrieves top-K product candidates before downstream rankers and re-rankers.

model-routing (ranking-stack)

A common retrieval layer feeds diverse downstream ranking systems with different ads/organic objectives.

eval-driven-development (model-selection)

Offline Recall@K and sequence-ablation experiments drove architecture and sequence-length decisions.

Tradeoffs

Start with product-ID sequences only

Chose simpler sequence model over including all user/product/context features immediately.

Rationale: The preliminary version already showed significant impact and reduced complexity.

Use sequence length of 20

Chose bounded recent session context over unbounded full-session history.

Rationale: Recent 3-5 products matter most and prior products diminish with sequence length.

Use unified retrieval across ads and organic

Chose shared contextual retrieval layer over surface-specific retrieval stacks.

Rationale: Shared retrieval reduces maintenance while downstream rankers still optimize surface-specific goals.

Gotchas (what bit them)

Catalog scale makes naive BERT-style retrieval difficult.

Root cause: Instacart has millions of products across thousands of retailers, far beyond tens-of-thousands product benchmarks.

Resolution: The first iteration restricts vocabulary under one million and explores ANN scaling.

Product identity varies across retailers.

Root cause: Some products share canonical identity but others, like non-branded produce, do not.

Resolution: Instacart points to text-content-inclusive approaches such as TiGER.

Popularity bias can dominate recommendations.

Root cause: Most interaction data comes from a small subset of the catalog.

Resolution: Instacart gets top-K products for a particular retailer and continues to iterate.

Evaluation

Instacart evaluates with offline Recall@K next-product prediction, randomized sequence training/evaluation ablations, online cart-add lift, and impact across transaction volume and ad marketplace metrics.

Metrics: Recall@K · 10-40% worse Recall@K from randomized training sequence · 20-45% worse metrics from randomized evaluation order · 30% lift in user cart additions · last 3-5 products have outsized influence

Outcomes

• Instacart deprecated old ad hoc retrieval systems.
• The unified retrieval layer serves multiple ads and organic recommendation surfaces.
• The BERT-like sequence model delivered significant offline uplift and a 30% cart-add lift on cart recommendations.
• Sequence order proved materially important for recommendation quality.

Non-agentic ML/retrieval case: millions of products, thousands of retailers, multiple surfaces, product vocabulary constrained under one million in v1, and explicit lift/ablation metrics.