Synthetic Consumer Survey Generation Using LLMs with Semantic Similarity Response Mapping

colgate · Other · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Market research and synthetic consumer surveys

PyMC Labs and Colgate needed synthetic consumer research that correlated with real survey behavior, but direct 1-5 rating prompts caused LLMs to hedge toward middle answers and lose distributional realism.

Constraints:

• Synthetic survey outputs had to match both distribution shape and per-item purchase-intent correlations.
• The method needed to reproduce demographic effects such as age and income.
• LLM numeric ratings were biased toward neutral middle responses.
• The system needed to be cheaper and faster than traditional surveys while still validated against real data.
• Prompt development risked overfitting to benchmark cases.

Architecture

The team asked frontier LLMs to answer product-survey prompts in natural language, then mapped those responses onto quantitative purchase-intent scales using embedding similarity to reference responses. This preserved richer language while enabling statistical comparison against real survey ratings.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  synthetic_consumer_persona_prompt["Synthetic consumer persona prompt"] --> frontier_llm_response_generator["Frontier LLM response generator"]
  product_concept_input["Product concept input"] --> frontier_llm_response_generator
  frontier_llm_response_generator --> embedding_similarity_mapper["Embedding similarity mapper"]
  reference_response_embeddings["Reference response embeddings"] --> embedding_similarity_mapper
  embedding_similarity_mapper --> evaluation_metrics["Evaluation metrics"]

Components

Data flow

A synthetic persona with demographic attributes reviews a product concept. GPT-4o or Gemini 2.0 Flash produces a natural text response rather than a numeric score. The text is embedded and compared with embedded reference responses that correspond to the rating scale, producing quantitative values that can be evaluated against real survey distributions.

Why this architecture

The key design was to avoid forcing LLMs into direct numeric ratings, where they hedged, and instead let them express consumer reactions naturally before translating those reactions into numbers through semantic similarity.

Models used: GPT-4o · Gemini 2.0 Flash · Embedding model

Scale

• The method achieved 90% of the maximum achievable correlation with real survey data.
  “This approach achieved 90% of the maximum achievable correlation with real survey data”summary, stated
• The baseline numeric-rating approach reached only 82% of the noise ceiling.
  “the reproduction factor was only 82% of the noise ceiling, which represents the theoretical maximum achievable given the inherent variability in human survey responses.”summary, stated

Alternatives considered (roads not taken)

Direct 1-5 numeric ratings from the LLM

Rejected because: LLMs hedged toward the middle and failed to reproduce realistic response distributions.

Traditional human consumer surveys only

Rejected because: Synthetic surveys were positioned as faster and cheaper, allowing more rapid product-concept iteration.

Patterns

synthetic-data-generation (evaluation data)

The team generated synthetic consumer responses and validated them against real survey data.

eval-driven-development (measurement)

Distribution matching, correlation, and demographic-effect reproduction determined whether the synthetic method was credible.

Tradeoffs

Generate natural text and map it afterward rather than asking for direct ratings.

Chose Natural-language response plus embedding-similarity mapping over Direct 1-5 LLM scoring.

Rationale: Natural language preserved realistic consumer variance, while embedding mapping restored quantitative comparability.

Use capable frontier models for synthetic consumer generation.

Chose GPT-4 and Gemini 2.0 Flash over Less capable tested models.

Rationale: The case reports that other tested models did not perform as well.

Gotchas (what bit them)

Asking an LLM for a rating can collapse variance.

Root cause: The constrained numeric output made the models hedge toward neutral middle values.

Resolution: Collect natural-language reactions and map them semantically to reference responses.

Prompt iteration can overfit the validation setup.

Root cause: Reference prompts for similarity mapping are one of the few tunable parts of the method.

Resolution: Treat prompt development as a validation risk and test generalization beyond the cases used to tune prompts.

Evaluation

The team compared synthetic and real responses using distribution matching, per-item purchase-intent correlation, noise-ceiling reproduction, and demographic effects.

Metrics: Kolmogorov-Smirnov distance. · Mean purchase-intent correlation. · Percentage of noise ceiling reproduced. · Age and income effect reproduction.

Outcomes

• The semantic mapping approach achieved 90% of maximum achievable correlation.
• It reproduced age and income demographic effects.
• It eliminated positivity bias present in human surveys.
• It provided richer qualitative feedback while being faster and cheaper.

The case gives validation percentages rather than production request volume; it has moved into production deployment and is expanding to advertisement evaluation.