Semi-Supervised Fine-Tuning of Compact Vision-Language Models for Product Attribute Extraction

flipkart · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce product attribute extraction with vision-language models

Flipkart needed accurate product attribute extraction across massive catalogs while avoiding the cost of fully supervised labeling and large VLM API calls.

Constraints:

• Product attributes such as color, pattern, material, and style are essential for search and recommendations.
• Manual labeling millions of products is expensive and error-prone.
• Large VLM APIs are costly at modern e-commerce scale.
• Compact models must learn effectively from scarce labels and abundant unlabeled product images.

Architecture

Flipkart trains compact 2-3B parameter VLMs with a semi-supervised pipeline: bootstrap with a small labeled set labeled by large multimodal models, apply PEFT adapters for efficient fine-tuning, generate multiple reasoning-and-answer chains on unlabeled products, select preferred outputs through self-consistency, and use Direct Preference Optimization to favor preferred reasoning chains over dis-preferred ones across iterative cycles.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Product listings"] --> B["Small labeled set"]
  A --> E["Unlabeled product pool"]
  B --> C["PEFT adapter training"]
  C --> D["Compact VLM"]
  E --> F["Self-consistency labeling"]
  D --> F
  F --> G["DPO training"]
  G --> D
  D --> H["Attribute predictions"]

Components

Data flow

A compact pre-trained VLM is first adapted on a small curated labeled dataset with PEFT. It then generates multiple reasoning-answer chains for unlabeled product listings, self-consistency selects preferred outputs, DPO trains the model to prefer those outputs, and the process iterates so the compact model improves from unlabeled catalog data.

Why this architecture

The architecture uses compact VLMs plus PEFT/DPO because Flipkart needs production-suitable speed and cost while still exploiting abundant unlabeled catalog images that supervised fine-tuning cannot use alone.

Models used: Qwen2.5-VL-3B-Instruct · compact VLMs · Gemini/GPT · Direct Preference Optimization

Scale

• The business problem spans millions of e-commerce items.
  “But with millions of items, manually labelling every single product is a monumental task.”article, stated
• The approach targets compact 2-3B parameter VLMs.
  “using compact VLMs (with 2–3 billion parameters)”article, stated
• Experiments covered twelve e-commerce verticals.
  “The experiments were conducted on a dataset spanning twelve e-commerce verticals, including everything from sandals and sarees to suitcases and watches.”article, stated
• Qwen2.5-VL-3B-Instruct improved from 75.1% to 85.7% accuracy after DPO training.
  “the accuracy of the Qwen2.5-VL-3B-Instruct model jumped from 75.1% to 85.7% after DPO training.”article, stated

Alternatives considered (roads not taken)

Manual product-attribute labeling at full scale

Rejected because: Manual labeling is expensive, error-prone, and impractical for millions of products.

Large VLM APIs for attribute extraction

Rejected because: API cost becomes a major hurdle at modern e-commerce platform scale.

Self-learning on high-confidence predictions

Rejected because: Self-learning caused accuracy drops in smaller VLMs, likely by reinforcing model bias.

Patterns

fine-tuning-adaptation (model-adaptation)

Flipkart adapts compact VLMs with PEFT and then further tunes them with DPO preference data from unlabeled products.

synthetic-data-generation (data)

The compact model generates multiple reasoning-and-answer chains on unlabeled products to create preference data.

confidence-thresholding (routing)

Self-consistency treats the most frequent generated answer as the preferred output for preference training.

Tradeoffs

Train compact VLMs.

Chose 2-3B parameter models over large VLM APIs for every product.

Rationale: Compact models are more suitable when speed and cost matter in real e-commerce systems.

Use DPO instead of self-learning.

Chose preference optimization over reasoning chains over retraining on high-confidence predictions.

Rationale: DPO was more stable for smaller VLMs while self-learning dropped accuracy.

Gotchas (what bit them)

Label scarcity blocks supervised VLM training.

Root cause: Manual product-attribute labeling is costly and error-prone across massive diverse catalogs.

Resolution: Use a small labeled set for initialization and exploit unlabeled product listings with self-consistency plus DPO.

Self-learning can collapse compact models.

Root cause: Retraining on a model's own high-confidence predictions can reinforce its biases.

Resolution: Use preference optimization over preferred and dis-preferred reasoning chains instead of simple self-learning.

Large VLM APIs are not necessarily production-economical.

Root cause: At catalog scale, per-call VLM API cost becomes a major hurdle.

Resolution: Train compact VLMs that are faster and cheaper for e-commerce attribute extraction.

Evaluation

Flipkart evaluates compact VLM training on a manually labeled test set across twelve verticals, comparing pre-trained checkpoints, PEFT supervised fine-tuning, DPO unsupervised fine-tuning, and self-learning.

Metrics: accuracy · vertical coverage · labeled listings per vertical · unlabeled data volume · DPO vs self-learning

Outcomes

• Qwen2.5-VL-3B-Instruct improved from 75.1% to 85.7% accuracy after DPO training.
• DPO outperformed the initial supervised model using unlabeled data.
• Self-learning caused smaller VLM accuracy to drop, while DPO remained stable.
• The approach can improve product listings, search, recommendations, and cataloging cost.

The source cites millions of items, twelve verticals, compact 2-3B parameter models, and accuracy gains on a manually labeled test set.