Fine-tuning LLMs for Market Research Product Description Matching

kantar_worldpanel · Consulting · 2024 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: market research product-description matching

Kantar Worldpanel needed a more flexible way to experiment with GenAI for matching paper receipt descriptions to product barcode names, because legacy systems were inflexible, engineering-heavy, and dependent on outdated specialized skills.

Constraints:

• legacy systems were not flexible or scalable enough for new AI use cases
• manual and engineering resources were expensive bottlenecks for generating training data
• receipt descriptions had to be matched to reference product barcode names with enough accuracy to support client insights
• the team wanted to compare several GenAI models and then fine-tune a smaller model for production use
• data sharing and collaboration needed secure governance across different environments

Architecture

Kantar used the Databricks Data Intelligence Platform as a shared experimentation and ML lifecycle environment. Data scientists compared Llama, Mistral, GPT-4, and GPT-3.5, selected GPT-4 for high-quality generated labels, produced about 120,000 receipt-description/barcode-name pairs, and planned to use that generated training data to fine-tune a smaller model for a production pipeline. MLflow supported experiment tracking and reproducibility, while Unity Catalog provided secure data sharing and governance.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  receipt-description-data[Receipt description data]
  databricks-data-intelligence-platform[Databricks Data Intelligence Platform]
  databricks-marketplace-models[Databricks Marketplace models]
  mlflow-experiment-tracking[MLflow experiment tracking]
  model-comparison-labels[Model comparison labels]
  gpt-4-training-data-generator[GPT-4 training-data generator]
  smaller-fine-tuned-production-model[Smaller fine-tuned production model]
  manual-coding-teams[Manual coding teams]
  unity-catalog-governance[Unity Catalog governance]
  receipt-description-data --> databricks-data-intelligence-platform
  receipt-description-data --> gpt-4-training-data-generator
  databricks-data-intelligence-platform --> databricks-marketplace-models
  databricks-data-intelligence-platform --> mlflow-experiment-tracking
  databricks-data-intelligence-platform --> unity-catalog-governance
  databricks-marketplace-models --> model-comparison-labels
  mlflow-experiment-tracking --> model-comparison-labels
  model-comparison-labels --> gpt-4-training-data-generator
  gpt-4-training-data-generator --> smaller-fine-tuned-production-model
  gpt-4-training-data-generator --> manual-coding-teams

Components

Data flow

Receipt descriptions and product barcode names are brought into Databricks, where data scientists can download candidate models, run experiments, and use result labels to compare model quality. GPT-4 generated a training dataset of receipt-description/barcode-name pairs, which Kantar planned to use to fine-tune a smaller model for production. MLflow tracks experiments and results, Unity Catalog governs data sharing, and manual coding teams focus on discrepant results instead of mass data generation.

Why this architecture

Kantar chose a Databricks-centered experimentation path because it let data scientists compare models, manage MLflow experiments, access clusters with minimal engineering support, generate labeled training data quickly, and then fine-tune a smaller model for lower-cost production serving.

Models used: GPT-4 · GPT-3.5 · Llama · Mistral

Scale

• Kantar works in the fast-moving consumer goods sector and sells insights from consumer data.
  “Kantar Worldpanel is a prominent international market research company that collects and analyzes consumer data, primarily in the fast-moving consumer goods (FMCG) sector.”article, stated
• The POC generated about 120,000 receipt-description/barcode-name pairs.
  “we’ve automatically generated a training dataset of about 120,000 pairs of receipt descriptions and barcode names with an accuracy of 94% in just a couple of hours”article, stated
• The reported best model accuracy for training-data generation was 94%.
  “Ultimately, GPT-4 provided better answers, with an accuracy of 94%.”article, stated
• The team used one or two lines of code to download models and work with a cluster in a single platform.
  “With one or two lines of code, we can download models, work and experiment with them and have a cluster that can handle the models — all in a single place”article, stated

Alternatives considered (roads not taken)

continue relying on legacy systems

Rejected because: they were less flexible and scalable, resource-intensive to maintain, and required outdated specialized skills.

use human resources to generate training data at scale

Rejected because: the team wanted faster generation without heavy human effort, leaving manual coders to focus on discrepant cases.

serve GPT-4 directly for the task

Rejected because: GPT-4 was used to identify and generate high-quality training data, while Kantar planned to fine-tune a smaller production model for cost and performance.

run model experiments across disconnected tools

Rejected because: Kantar valued a single platform for downloading models, running experiments, managing clusters, tracking results, and governance.

Patterns

fine-tuning-adaptation (model-operations)

Kantar used GPT-4 to generate task-specific training data so a smaller model could be fine-tuned for product-description matching in the production pipeline.

eval-driven-development (evaluation)

The data science team compared candidate models using correctness labels and selected GPT-4 based on reported task accuracy.

human-in-the-loop-gate (human-oversight)

Automation shifted manual coding teams toward discrepant results rather than removing human review from the workflow entirely.

cost-aware-routing (serving)

The system used a strong model for high-quality training data while targeting a smaller fine-tuned model for production cost and performance.

Tradeoffs

Use GPT-4 for generating training data, then fine-tune a smaller model.

Chose strong-model data generation plus smaller-model production serving over serving the strongest model directly for every production call.

Rationale: GPT-4 produced the best-quality outputs for the task, but a smaller fine-tuned model was expected to be more cost-effective and performant.

Centralize experimentation on Databricks.

Chose one integrated platform for models, clusters, MLflow, governance, and experiments over separate legacy systems and scattered experiment infrastructure.

Rationale: A unified platform reduced engineering dependency and made model comparison easier.

Use automation to redirect rather than eliminate manual coding work.

Chose manual review of discrepant cases over manual generation of masses of training data.

Rationale: The generated dataset handled volume while manual coders focused on cases where human judgment was more valuable.

Gotchas (what bit them)

Legacy systems limited experimentation with GenAI use cases.

Root cause: The older systems were inflexible, resource-intensive, and depended on outdated specialized skills.

Resolution: Kantar moved experimentation and ML lifecycle work onto Databricks so data scientists could run POCs and compare models more directly.

Data scientists were dependent on engineers for cluster and service setup.

Root cause: The experimentation environment required infrastructure setup before model work could happen.

Resolution: Kantar used Databricks so data scientists could download models, run experiments, and use clusters with minimal setup.

Generating training data manually consumed scarce human and engineering resources.

Root cause: The product matching task needed large numbers of receipt-description/barcode-name pairs.

Resolution: The team used GenAI to generate the bulk dataset and focused manual coders on discrepant results.

Evaluation

Kantar compared Llama, Mistral, GPT-4, and GPT-3.5 inside Databricks, using labels that identified whether outputs were correct or wrong. The customer story reports GPT-4 as the best performer with 94% accuracy for the task.

Metrics: GPT-4 reported at 94% accuracy · about 120,000 generated receipt-description/barcode-name training pairs · training data generated in a couple of hours

Outcomes

• The Databricks customer story reports enhanced data accuracy, streamlined workflows, and optimized resource utilization.
• GPT-4 reportedly reached 94% accuracy for generating product-description matching training data.
• Kantar generated about 120,000 receipt-description/barcode-name pairs in a couple of hours.
• Manual coding teams could focus on discrepant results instead of mass training-data generation.
• Data scientists could experiment with models and clusters in one platform with less engineering dependency.

The POC sits upstream of Kantar's client-facing market insights pipeline, matching receipt descriptions to barcode names so purchased products can be identified and converted into insights.