Building a Commonsense Knowledge Graph for E-commerce Product Recommendations

amazon · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce recommendation with commonsense reasoning

Amazon needed recommendation models to infer commonsense relationships between customer queries, products, audiences, functions, and contexts, such as understanding why slip-resistant shoes might fit a query for pregnant women.

Constraints:

• commonsense relationships are implicit in query-purchase and co-purchase behavior
• raw customer interaction data is noisy
• knowledge graph candidates need human plausibility and typicality assessment
• recommendation models must improve both frozen-encoder and fine-tuned settings
• generated relationships must be usable as entity-relation-entity triples

Architecture

Amazon built COSMO, a recursive LLM and human-in-the-loop framework for generating e-commerce commonsense knowledge graphs. An LLM proposes hypotheses from query-purchase and co-purchase data, machine learning and heuristics filter candidates, human annotators assess plausibility and typicality, reviewers extract guiding principles, and refined instructions prompt the LLM to produce higher-quality entity-relation-entity triples used by recommendation models.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer-interaction-pairs[Customer interaction pairs] --> noise-pruning[Noise pruning]
  noise-pruning --> llm-hypothesis-generator[LLM hypothesis generator]
  llm-hypothesis-generator --> human-ml-filter[Human ML filter]
  human-ml-filter --> guiding-principles[Guiding principles]
  guiding-principles --> knowledge-graph[Knowledge graph]
  knowledge-graph --> recommendation-model[Recommendation model]

Components

Data flow

Query-purchase and co-purchase pairs are pruned for noise, passed to an LLM to generate commonsense relationship hypotheses, filtered by machine learning models and human annotation, converted into entity-relation-entity triples, assembled into a knowledge graph, and supplied to cross-encoder product-query relevance models.

Why this architecture

The architecture converts implicit behavior into explicit commonsense relationships while keeping humans in the loop for plausibility. Feeding these relationships into cross-encoder relevance models helps product search match queries that require reasoning beyond lexical similarity.

Models used: large language model · bi-encoder · cross-encoder

Scale

• COSMO was presented at SIGMOD in June 2024.
  “In a paper we’re presenting at the Association for Computing Machinery’s annual Conference on Management of Data (SIGMOD) in June 2024, we describe COSMO, a framework that uses large language models (LLMs) to [discern the commonsense relationships](https://www.”article, stated
• The evaluation used the Shopping Queries Data Set created for KDD Cup 2022.
  “we used the Shopping Queries Data Set we created for KDD Cup 2022”article, stated
• With frozen encoders, the COSMO model achieved a 60 percent macro-F1 improvement over the best baseline.
  “When the models’ encoders were fixed — so the only difference between the cross-encoders was that one included COSMO relationships as inputs and the other didn’t — the COSMO-based model dramatically outperformed the best-performing baseline, achieving a 60% increase in macro F1 score.”article, stated
• After fine-tuning, COSMO retained a 28 percent macro-F1 and 22 percent micro-F1 edge.
  “When the encoders were fine-tuned on a subset of the test dataset, the performance of all three models improved significantly, but the COSMO-based model still held a 28% edge in macro F1 and a 22% edge in micro F1 over the best-performing baseline.”article, stated

Alternatives considered (roads not taken)

Use query-product pairs without COSMO relationships

Rejected because: The COSMO-seeded cross-encoder outperformed the baseline when relevant triples were added.

Rely only on model-generated hypotheses

Rejected because: Human annotation and reviewer-derived principles filter and improve low-quality hypotheses.

Patterns

human-in-the-loop-gate (data-quality)

Human annotators assess plausibility and typicality of candidate commonsense relationships.

synthetic-data-generation (knowledge-generation)

LLMs generate commonsense hypotheses and triples from behavior-derived pairs.

retrieval-ranking (ranking)

COSMO relationships are evaluated by product-query relevance ranking models.

Tradeoffs

Model architecture for evaluation

Chose cross-encoder with COSMO relationships over bi-encoder or cross-encoder without graph relationships.

Rationale: The COSMO-seeded model produced stronger F1 results.

Knowledge generation loop

Chose recursive LLM plus human/ML filtering over one-pass LLM graph generation.

Rationale: Guiding principles from high-quality hypotheses refine subsequent LLM prompts.

Gotchas (what bit them)

Customer behavior data contains noisy co-purchase relationships.

Root cause: Some co-purchases are too far apart in the product graph to imply useful commonsense.

Resolution: Prune pairs and filter candidate hypotheses with ML and human annotation.

Commonsense relationships need both plausibility and typicality.

Root cause: A relationship can be reasonable but not commonly associated with the query or source product.

Resolution: Ask human annotators to assess both criteria.

Evaluation

Amazon evaluated COSMO on the Shopping Queries Data Set with bi-encoder, cross-encoder, and COSMO-augmented cross-encoder models, using macro-F1 and micro-F1 in frozen and fine-tuned settings.

Metrics: macro F1 · micro F1 · 60% macro-F1 improvement · 28% macro-F1 edge · 22% micro-F1 edge

Outcomes

• COSMO improved macro-F1 by 60 percent in a frozen-encoder experiment.
• After fine-tuning, COSMO still retained 28 percent macro-F1 and 22 percent micro-F1 edges over the best baseline.

The work is evaluated on Amazon's Shopping Queries Data Set and presented as a SIGMOD 2024 framework.