Multi-modal LLM Platform for Catalog Attribute Extraction at Scale

instacart · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce catalog attribute extraction

Instacart needed to extract structured product attributes from millions of SKUs where SQL rules and text-only models were slow, low quality, and blind to image-only information.

Constraints:

Millions of products span thousands of categories.
Attributes may appear in images but not text descriptions.
Each attribute previously required heavy engineering work.
Production quality requires confidence scoring, monitoring, and human review.

Architecture

Instacart built PARSE, a self-serve multimodal LLM platform with a configuration UI, ML extraction endpoint, confidence estimation through self-verification, quality screening, human-in-the-loop correction, and catalog data pipeline integration.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  platform-ui["Platform UI"]
  backend-orchestration-layer["Backend Orchestration Layer"]
  ml-extraction-endpoint["ML Extraction Endpoint"]
  quality-screening["Quality Screening"]
  catalog-data-pipeline["Catalog Data Pipeline"]
  platform-ui --> backend-orchestration-layer
  backend-orchestration-layer --> platform-ui
  backend-orchestration-layer --> ml-extraction-endpoint
  ml-extraction-endpoint --> quality-screening
  ml-extraction-endpoint --> backend-orchestration-layer
  quality-screening --> ml-extraction-endpoint
  quality-screening --> catalog-data-pipeline
  catalog-data-pipeline --> quality-screening

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Platform UI`	Lets teams configure extraction tasks, attributes, algorithms, parameters, SQL queries, and few-shot examples.	`Backend Orchestration Layer`
`Backend Orchestration Layer`	Executes product-data SQL queries and routes data to extraction.	`Platform UI`, `ML Extraction Endpoint`
`ML Extraction Endpoint`	Runs text, image, cascade, or multimodal LLM extraction strategies.	`Quality Screening`, `Backend Orchestration Layer`
`Quality Screening`	Evaluates outputs through human review, LLM-as-judge, confidence monitoring, and low-confidence routing.	`ML Extraction Endpoint`, `Catalog Data Pipeline`
`Catalog Data Pipeline`	Ingests validated attributes into customer-facing catalog systems.	`Quality Screening`

Data flow

Teams configure an attribute in the UI, backend orchestration fetches product data, the extraction endpoint prompts a multimodal LLM over text and images, self-verification scores confidence, low-confidence outputs go to human or LLM evaluation, and validated attributes flow into the catalog pipeline.

“PARSE implements a comprehensive end-to-end platform with four main components that orchestrate the entire attribute extraction workflow from data retrieval through production deployment and quality monitoring.”summary, stated

“Final validated extraction results flow into Instacart's catalog data pipeline for ingestion.”summary, stated

Why this architecture

PARSE turns attribute extraction into a self-service platform because product teams need many attributes across many categories, and multimodal reasoning captures product details missing from text-only inputs.

“a self-serve multi-modal LLM platform that enables teams to extract attributes from both text and images with minimal engineering effort.”summary, stated

“Traditional text-only systems, whether rule-based or ML-based, consistently miss these cases, while PARSE's multi-modal LLM extracts the value directly from the image.”summary, stated

Models used: multi-modal LLMs · LLM-as-a-judge evaluators · LLM cascade models

Scale

Instacart extracts attributes from millions of SKUs.
“extract structured product attributes from millions of SKUs in their e-commerce catalog.”summary, stated
The catalog spans thousands of categories.
“With millions of products across thousands of categories”summary, stated
PARSE reduced development time from weeks to days.
“reduced attribute extraction development time from weeks to days”summary, stated
Multimodal reasoning improved recall by 10% over text-only approaches.
“achieved 10% higher recall through multi-modal reasoning compared to text-only approaches”summary, stated
PARSE achieved 95% accuracy on simpler attributes with one day of effort.
“delivered 95% accuracy on simpler attributes with just one day of effort versus one week with traditional methods.”summary, stated

Alternatives considered (roads not taken)

SQL-based rules

Rejected because: Rules were low quality, slow to develop, and could not handle image-only evidence.

“previous approaches using SQL-based rules and traditional machine learning models faced significant limitations: low quality for complex attributes requiring contextual understanding”summary, stated

Text-only machine learning

Rejected because: Text-only models miss attributes visible only on packaging images.

“extracting sheet count from household products often requires reading packaging images where "80 sheets" appears visibly on the product but the accompanying text description lacks this detail entirely.”summary, stated

Patterns

`human-in-the-loop-gate` (governance)

Human auditors label gold standards, review low-confidence outputs, and correct production extractions.

“This human-in-the-loop approach ensures production quality while focusing expensive human review on cases where the model expresses uncertainty.”summary, stated

`llm-as-judge` (quality)

PARSE uses LLM auto-evaluation alongside human review during quality screening.

“PARSE incorporates LLM-as-a-judge auto-evaluation capabilities alongside human review.”summary, stated

`model-routing` (architecture)

PARSE supports single-model, cascade, and multimodal extraction strategies to balance cost and accuracy.

“endpoint supports multiple extraction strategies to balance cost and accuracy tradeoffs.”summary, stated

Tradeoffs

Use confidence-based review

Chose Self-verification and human review for low-confidence outputs over Accepting all LLM extractions automatically.

Rationale: This preserves quality while focusing human review on uncertain cases.

“proactive error detection by identifying extracted values with low confidence scores and routing them to human auditors for review and correction.”summary, stated

Build self-service extraction

Chose Configurable platform UI and versioned configurations over Custom engineering project for each attribute.

Rationale: Self-service reduces development overhead and supports many attributes.

“The Platform UI serves as the self-service configuration layer where teams define extraction tasks without writing custom code.”summary, stated

Gotchas (what bit them)

Text-only attribute extraction misses packaging facts.

Root cause: Important attributes may only be visible in product images.

Resolution: Use multimodal LLM extraction over both text and image inputs.

“For example, extracting sheet count from household products often requires reading packaging images where "80 sheets" appears visibly on the product”summary, stated

Attribute-specific model development does not scale across catalog breadth.

Root cause: Separate models and datasets for each attribute create high development overhead.

Resolution: Build a self-serve platform where teams configure attributes, prompts, data queries, and examples.

“high development overhead requiring separate models and datasets for each attribute, slow iteration cycles taking weeks for new attributes”summary, stated

Evaluation

PARSE combines human gold labels, LLM-as-judge auto-evaluation, confidence scoring, periodic production sampling, and low-confidence human review.

Metrics: accuracy · recall · confidence score · development time

“The component offers a human evaluation interface where auditors label gold-standard attribute values and compute quality metrics.”summary, stated

Outcomes

PARSE reduced development time from weeks to days.
Multimodal reasoning improved recall by 10% versus text-only extraction.
The platform achieved 95% accuracy on simpler attributes with one day of effort.

Millions of SKUs, thousands of categories, and production catalog ingestion for customer-facing search, badges, and personalization.

“where they immediately become available for search refinement, product variation selection, badge displays, and other personalization features.”summary, stated