Implementing Product Comparison and Discovery Features with LLMs at Scale

idealo · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM-powered product discovery and comparison for e-commerce

idealo wanted to improve product comparison and category discovery for shoppers without building hype-driven chatbots or relying on an LLM's ungrounded world knowledge. The team needed LLM features that solved concrete user needs, used idealo's proprietary product and expert-content data, produced structured downstream outputs, and could be evaluated through both quality checks and business KPIs.

Constraints:

• users had short attention spans and did not want to type into or read from a generic shopping chatbot
• product pages could expose 60 to 70 attributes, making manual comparison hard
• category pages had long filter lists that non-expert shoppers did not know how to use
• LLMs can hallucinate convincingly when treated as knowledge databases
• product finder quality is fuzzy and difficult to evaluate with a single deterministic metric
• business success still had to be measured through A/B-tested KPIs such as bounce rate and conversion

Architecture

idealo treated LLMs as language interfaces over product data and expert content. For product comparison, the system feeds long product-attribute lists to an LLM so it can choose the comparison-relevant attributes and highlight meaningful differences such as lower weight or higher camera resolution. For guided discovery, the system uses expert category content from the page, asks one prompt to select the next consideration, maps that consideration to available filters, then generates a user-facing question and answer options. Evaluation combines rule-based checks, traditional precision/recall where applicable, LLM-as-judge prompts, manual review over hundreds of examples, and live A/B tests against business KPIs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Shopper["Shopper"] --> Product_comparison_UI["Product comparison UI"]
  Shopper --> Guided_product_finder_UI["Guided product finder UI"]
  Product_comparison_UI --> Product_attribute_catalog["Product attribute catalog"]
  Product_attribute_catalog --> Comparison_attribute_selector["Comparison attribute selector"]
  Comparison_attribute_selector --> LLM_provider["LLM provider"]
  Comparison_attribute_selector --> Structured_output_parser["Structured output parser"]
  Structured_output_parser --> Product_comparison_UI
  Guided_product_finder_UI --> Next_consideration_prompt["Next-consideration prompt"]
  Expert_category_content["Expert category content"] --> Next_consideration_prompt
  Next_consideration_prompt --> LLM_provider
  Next_consideration_prompt --> Filter_mapping_prompt["Filter mapping prompt"]
  Available_filter_catalog["Available filter catalog"] --> Filter_mapping_prompt
  Filter_mapping_prompt --> LLM_provider
  Filter_mapping_prompt --> Question_and_answer_generator["Question-and-answer generator"]
  Expert_category_content --> Question_and_answer_generator
  Question_and_answer_generator --> LLM_provider
  Question_and_answer_generator --> Structured_output_parser
  Structured_output_parser --> Guided_product_finder_UI
  Evaluation_harness["Evaluation harness"] --> Rule_based_evaluation_checks["Rule-based evaluation checks"]
  Evaluation_harness --> LLM_as_judge_evaluator["LLM-as-judge evaluator"]
  Evaluation_harness --> Manual_review_set["Manual review set"]
  LLM_as_judge_evaluator --> LLM_provider
  Product_attribute_catalog --> Rule_based_evaluation_checks
  Shopper --> A_B_test_measurement["A/B test measurement"]
  Guided_product_finder_UI --> A_B_test_measurement
  A_B_test_measurement --> Evaluation_harness

Components

Data flow

A user views either a product page or a category page. On product pages, idealo passes product attributes into an LLM-backed comparison workflow that extracts relevant attributes and highlights differences. On category pages, expert content is inserted into a next-consideration prompt, the selected consideration is mapped to an available filter, and another prompt generates a question plus answer options. Outputs are expected in structured formats such as JSON or XML so downstream logic can render product tables, filters, or evaluation results. Quality is checked with rule-based assertions, LLM-as-judge evaluations, manual hallucination review, and A/B-test metrics.

Why this architecture

The architecture deliberately narrows LLM responsibility to language-interface tasks where the model is strong: extracting salient attributes, mapping natural language to existing filters, generating structured questions, and judging fuzzy outputs. It avoids using the LLM as an all-knowing shopping database by grounding prompts in idealo's product attributes and expert content, then validating outputs with structured formats and evaluation loops.

Models used: large language models for product comparison · large language models for product finder prompt chain · LLM-as-judge evaluator · GPT-4 · Claude 2 · Claude 2.1 · OpenAI JSON mode

Scale

• idealo operated in six European countries.
  “we're in six European countries Germany is by far the biggest Market”transcript, stated
• idealo had contracts with more than 50,000 shops.
  “we have contracts with over 50,000 shops and we try to help you there”transcript, stated
• The product catalog covered more than 4 million products and more than 500 million offers.
  “over 4 million products over 500 million offers that we map to products a lot with machine learning actually”transcript, stated
• Comparison pages could expose roughly 60 to 70 product attributes.
  “we have a really long list of product attributes down the bottom right and it's but it's like 60 70 attributes”transcript, stated
• The guided product finder was live in an A/B test with a 50% exposure chance for the variant.
  “this is actually live on the web page and this is part of an AB test now so you need to be in a certain variant so if you go on Ido and V talk now you have a 50% chance”transcript, stated
• The team manually reviewed hundreds of examples when checking product-comparison hallucinations.
  “we still manually went through hundreds and hundreds of examples and and in the end we got like a a really high confidence level that there are zero hallucinations”transcript, stated

Alternatives considered (roads not taken)

Build a generic shopping chatbot

Rejected because: User testing showed idealo users did not necessarily want to type or read lots of text, so a chat interface did not address their actual problems.

Treat the LLM as a world-knowledge database

Rejected because: That increases hallucination risk and reduces trust for purchase decisions, so idealo grounded generation in proprietary data and references.

Rule-based product-comparison logic

Rejected because: The team could encode all comparison rules manually, but maintaining rules over many products and attributes would be costly.

Free-form LLM output

Rejected because: Pipeline integration required outputs that downstream systems could parse and use reliably, so the team preferred JSON, XML, or strict schema outputs.

Leave evaluation until after feature development

Rejected because: The team found that evaluation becomes the bottleneck once prototypes scale, so evaluation should be planned up front.

Patterns

rag-retrieval (knowledge-architecture)

idealo grounds product-finder generation in expert category content and uses proprietary product data rather than model world knowledge.

prompt-chaining (workflow-orchestration)

The product finder chains a next-consideration prompt, a filter-mapping prompt, and question/answer generation.

structured-output-contract (quality)

LLM outputs are constrained to structured formats such as JSON or XML so downstream logic can consume them.

eval-driven-development (quality)

idealo treats evaluation as an up-front design concern and combines rule-based checks, LLM-as-judge prompts, manual examples, and A/B-tested business metrics.

human-in-the-loop-gate (governance)

Human user feedback and manual review gate feature direction and hallucination confidence before broader rollout.

Tradeoffs

Product feature selection

Chose focused product comparison and guided discovery features over generic chatbot or shopping assistant.

Rationale: User tests showed idealo users did not want a typing-heavy chat interface, while comparison and filtering were concrete shopping problems.

Grounded language interface

Chose LLMs over idealo data and expert content over ungrounded LLM world knowledge.

Rationale: Grounding preserves the language benefits while reducing hallucination risk for purchase decisions.

LLM comparison logic

Chose LLM-selected comparison attributes and differences over hand-maintained rule-based comparison logic.

Rationale: The rule-based alternative was possible but costly to build and maintain across many product categories and attributes.

Structured integration

Chose JSON/XML or strict schema outputs over free-form text responses.

Rationale: Structured outputs make it possible to integrate LLM outputs into downstream product logic and additional model calls.

Evaluation investment

Chose early evaluation effort comparable to solution-building effort over prototype-first development with evaluation later.

Rationale: The team learned that iteration stalls when evaluation is not fast enough or informative enough.

Gotchas (what bit them)

Hype-driven chatbot prototypes missed the actual user need.

Root cause: The team initially followed the general industry impulse to build a shopping assistant instead of validating whether idealo users wanted conversational interaction.

Resolution: Start from user problems, prototype quickly, and validate with qualitative user tests before productionizing.

LLM world knowledge creates trust and hallucination risk for purchase decisions.

Root cause: A model can answer convincingly from its own learned knowledge even when the answer is false.

Resolution: Use the model as a language interface over proprietary idealo data or retrieved/search-backed sources.

Evaluation becomes the development bottleneck if it is delayed.

Root cause: Generative outputs such as question quality, answer options, and product-finder flow are fuzzy and hard to measure after the fact.

Resolution: Design evaluation alongside the feature using rule checks, traditional metrics, LLM-as-judge prompts, and business A/B metrics.

LLM-as-judge is useful but not a silver bullet.

Root cause: Judge prompts themselves require tuning and can only assess what the evaluation prompt captures.

Resolution: Treat LLM-as-judge as one evaluation technique and spend time tuning the evaluation prompt.

Structured output is necessary for pipeline integration.

Root cause: Free-form generations are hard to use downstream in product logic or further LLM steps.

Resolution: Use JSON, XML, or strict schema outputs and feed those structures into downstream logic or other models.

Evaluation

idealo evaluates LLM product features through several layers: rule-based checks for deterministic attributes, traditional metrics such as precision and recall where appropriate, LLM-as-judge prompts for fuzzy outputs, manual review of hundreds of examples to check hallucinations, and live A/B tests for business impact. The speaker emphasizes evaluation as a first principle rather than an afterthought.

Metrics: rule-based attribute-value correctness checks · precision and recall for named-entity-style tasks · LLM-as-judge quality, relevance, instruction-following, and hallucination checks · manual hallucination review over hundreds of examples · A/B-tested bounce rate · A/B-tested conversion rate

Outcomes

• idealo deployed an LLM-powered product finder as a live A/B test.
• The product comparison workflow extracts six or seven salient attributes from roughly 70 attributes for comparison tables.
• The product finder uses expert content and available filters to ask contextual narrowing questions.
• Manual review over hundreds of examples produced high confidence that the product-comparison approach had no hallucinations in the reviewed set.
• The team reports no guarantee of zero hallucinations and frames the approach as risk reduction rather than proof.

The source is especially useful for separating production LLM product engineering from agentic autonomy. idealo's system is not an autonomous agent; it is a grounded language-interface pipeline at e-commerce scale, backed by product data, structured outputs, evaluation prompts, manual checks, and A/B testing.