Scaling Product Categorization from Manual Tagging to LLM-Based Classification

getyourguide · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: travel marketplace product categorization

GetYourGuide needed to classify hundreds of thousands of travel products into hundreds of multi-label categories, improving relevance and conversion beyond manual tagging, supplier rules, and vector-threshold systems.

Constraints:

• Products can belong to multiple categories simultaneously.
• A naive product-category LLM matrix would require 250,000 x 600 category decisions.
• The system needed better recall and niche-category coverage without making API cost or latency explode.

Architecture

GetYourGuide uses a hybrid pipeline: OpenAI embeddings pre-rank products for each category, products are grouped into OpenAI batch jobs, GPT-4-mini evaluates one product/category pair with structured reasoning/confidence/decision fields, custom early stopping halts low-yield category processing, and Airflow-like automation handles errors and recurring updates. Offline evaluation and A/B testing validate quality and business impact.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Product catalog"] --> C["Embedding pre-ranker"]
  B["Category catalog"] --> C
  C --> D["Batch job builder"]
  D --> E["LLM classifier"]
  B --> E
  E --> F["Structured output schema"]
  F --> G["Early stopping monitor"]
  G --> H["Category assignments"]
  H --> I["Offline evaluation"]
  H --> J["A/B testing"]

Components

Data flow

For each category, embeddings rank products by semantic similarity; ranked product-category pairs are batched and sent to OpenAI; the LLM returns structured reasoning, confidence, and yes/no decisions; early stopping watches connection rates and stops low-yield batches; results update category assignments and flow into offline/online evaluation.

Why this architecture

The winning architecture reduced an enormous classification matrix by using embeddings to prioritize candidates, batch jobs to cut cost, early stopping to avoid long tails, and structured reasoning/confidence fields to improve LLM classification quality.

Models used: OpenAI embeddings · GPT-4-mini

Scale

• The catalog has roughly 250,000 products.
  “roughly have to classify 250,000 products”transcript, stated
• The taxonomy has about 600 categories.
  “and we now have 600 categories.”transcript, stated
• The system handles roughly 1,500 new or updated products.
  “mentioned before there were 1,500 no 100 1,500 products or updated products”transcript, stated
• OpenAI batch jobs offer a 50% discounted price relative to normal calls.
  “You pay 50% of the uh discounted price of the normal calls.”transcript, stated
• OpenAI batch jobs have a 24-hour SLA.
  “they have a 24-hour SLA for the answers.”transcript, stated

Alternatives considered (roads not taken)

Manual categorization

Rejected because: Manual tagging was too slow and expensive to scale.

Semantic vector thresholding alone

Rejected because: Different categories needed different threshold angles and quality was insufficient for excluded categories.

One product plus all categories

Rejected because: Quality/cost tradeoffs were worse than a one-product one-category decision with a smaller model.

Patterns

rag-retrieval (candidate-generation)

Embedding similarity retrieves and prioritizes likely product-category pairs before expensive LLM classification.

structured-output-contract (output-control)

The classifier returns reasoning, confidence, and decision fields in a schema to improve quality and debugging.

batch-offline-processing (cost-optimization)

OpenAI batch jobs process many product-category decisions asynchronously at lower cost.

eval-driven-development (evaluation)

Offline MCC/precision/recall and online A/B metrics validated the categorization system.

Tradeoffs

Classification unit

Chose one product and one category per LLM decision over larger multi-product/multi-category prompts.

Rationale: The small yes/no task worked well with GPT-4-mini and structured outputs.

Cost control

Chose embedding pre-ranking plus early stopping over classify every product-category combination.

Rationale: The full 250k x 600 matrix would create too many API calls.

Quality validation

Chose offline metrics plus A/B experiment over offline model metrics alone.

Rationale: Business impact needed conversion, quote-rate, and bounce-rate validation.

Gotchas (what bit them)

Structured output order affected reasoning quality.

Root cause: Putting the decision before the reason reduced the model's thinking step.

Resolution: Put reasoning before decision and include confidence in the schema.

Batch savings come with a latency SLA.

Root cause: OpenAI batch jobs are cheaper but asynchronous with a 24-hour answer window.

Resolution: Use batch for offline categorization and parallelize by category.

Long-tail niche categories make exhaustive processing wasteful.

Root cause: Some categories may have only about 100 relevant products.

Resolution: Use embedding pre-ranking and stop category batches when positive assignment rates drop below threshold.

Evaluation

GetYourGuide evaluated categorization with MCC, recall, precision, category coverage, and A/B tests on conversion, quote rate, and bounce rate.

Metrics: MCC · recall · precision · category coverage · conversion rate · quote rate · bounce rate

Outcomes

• MCC improved noticeably, driven by recall gains and slight precision gains.
• 50 existing categories previously excluded by vector quality were restored to automation.
• 295 new categories were enabled.
• A/B testing showed a 1.3% conversion-rate improvement plus improved quote rate and reduced bounce rate.

The system is a production batch classification pipeline for marketplace categorization, not an agentic user-facing workflow.