AI-Powered Customer Feedback Analysis Using RAG and LLMs in Product Analytics

meta · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Consumer hardware product analytics

Meta Reality Labs needed a self-service way to analyze noisy, biased, unstructured feedback across reviews, bug reports, surveys, social media, and other sources for Quest and Ray-Ban Meta products.

Constraints:

Feedback sources include noise, bias, inauthentic comments, and freeform qualitative text
Mutable feedback can become outdated after product changes
Customer feedback used in the AI tool is deidentified under privacy and data-use policies

Architecture

Meta's Reality Labs tool starts with a deidentified customer-feedback repository spanning internal and external sources, then prepares RAG-ready feedback by gathering raw items, contextualizing and summarizing them, embedding the contextualized feedback, retrieving top-N matches by cosine similarity, and passing retrieved evidence to Llama for responses. The system is used for bug deduplication, internal testing summaries, video-review synthesis, and strategic planning.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  internal-feedback-sources["internal-feedback-sources"]
  external-feedback-sources["external-feedback-sources"]
  feedback-repository["feedback-repository"]
  contextualization-summarization-pipeline["contextualization-summarization-pipeline"]
  embedding-generation["embedding-generation"]
  cosine-similarity-retriever["cosine-similarity-retriever"]
  llama-response-generator["llama-response-generator"]
  manual-verification-references["manual-verification-references"]
  bug-deduplication-workflow["bug-deduplication-workflow"]
  internal-feedback-sources --> feedback-repository
  external-feedback-sources --> feedback-repository
  feedback-repository --> contextualization-summarization-pipeline
  contextualization-summarization-pipeline --> embedding-generation
  embedding-generation --> cosine-similarity-retriever
  cosine-similarity-retriever --> llama-response-generator
  llama-response-generator --> manual-verification-references
  bug-deduplication-workflow --> embedding-generation

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`feedback-repository`	Deidentified store of internal and external customer feedback for Reality Labs products.	`contextualization-summarization-pipeline`
`internal-feedback-sources`	Employee testing feedback, customer-service interactions, bug reports, and other internal signals.	`feedback-repository`
`external-feedback-sources`	Reviews, surveys, social conversations, video reviews, and other public or vendor-sourced feedback.	`feedback-repository`
`contextualization-summarization-pipeline`	Prepares raw feedback by joining related pieces into contextualized feedback and summarizing them.	`embedding-generation`
`embedding-generation`	Creates dense vector representations of contextualized feedback.	`cosine-similarity-retriever`
`cosine-similarity-retriever`	Retrieves top-N semantically similar customer feedback items for a prompt.	`llama-response-generator`
`llama-response-generator`	Llama-powered answer generator that uses retrieved feedback as context for product analytics questions.	`manual-verification-references`
`manual-verification-references`	Supporting references used for manual last-mile checking of generated summaries, especially video-review use cases.	—
`bug-deduplication-workflow`	Uses descriptions and stack traces to cluster related bug reports into higher-level categories.	`embedding-generation`

Data flow

Internal and external customer feedback is deidentified and assembled into a repository. Raw feedback is joined into contextualized feedback units, summarized, embedded into vectors, and stored for similarity search. At runtime, a user prompt is embedded, cosine similarity retrieves the top-N feedback items, and Llama generates an answer grounded in those retrieved examples; manual references support last-mile verification for rich media summaries.

3 supporting quotes

“All customer feedback is assembled in their most granular and raw form.”article, stated

“At runtime, the user’s prompt is matched with relevant customer feedback by comparing its embedding to stored feedback embeddings using cosine similarity”article, stated

“The LLM generates a response to the user’s prompt based on the contextually similar customer feedback retrieved.”article, stated

Why this architecture

Meta emphasizes the data foundation because hallucination risk comes from insufficient or biased feedback, not just model choice. RAG gives Llama relevant customer feedback at answer time, while deidentification and source diversity reduce privacy and bias risk. Additional context such as bug stack traces improves retrieval for operational product analytics use cases.

3 supporting quotes

“The performance of any LLM hinges on the quality and relevance of its data foundation.”article, stated

“The first step to mitigate hallucination risk is to establish an extensive repository of customer feedback”article, stated

“embedding their stack trace, in addition to their freeform description, improves retrieval accuracy”article, stated

Models used: Llama · LLMs · embedding models

Scale

The repository draws from a broad range of data sources.
“ensuring that customer feedback isn’t dominated by a small group of vocal users on a specific platform.”article, stated
All customer feedback used in the AI tool is deidentified for privacy compliance.
“All customer feedback used as a data source within our AI Tool is deidentified in compliance with Meta’s privacy & data-use policies”article, stated
The Quest video-review example summarizes feedback over a seven-month period.
“Summarizing Quest VR use cases based on Youtube product reviews over a seven-month period.”article, stated
Internal testing summaries condense hours of manual effort into minutes.
“This data is being summarized in executive reporting, condensing hours of manual effort into minutes.”article, stated

Alternatives considered (roads not taken)

Prompting Llama without a substantial feedback repository

Rejected because: The article says insufficient data makes even sophisticated LLMs hallucinate or produce misleading answers.

“Even the most sophisticated LLMs can falter, producing “hallucinations” or misleading answers, when they lack sufficient data”article, stated

Relying on one feedback channel such as bug reports

Rejected because: Limited sources create bias, so the repository spans internal and external sources.

“focusing solely on bug reports can create a skewed perspective, emphasizing negative technical issues.”article, stated

Embedding only bug freeform descriptions

Rejected because: The team found that adding stack traces improves retrieval accuracy for bug clustering and classification.

“embedding their stack trace, in addition to their freeform description, improves retrieval accuracy for prompts related to the clustering and classification of common bug traits.”article, stated

Fully trusting generated video-review summaries

Rejected because: The team asks the LLM for supporting references to enable manual last-mile verification.

“To assist with manual “last-mile” verification, we also asked the LLM to provide supporting references.”article, stated

Patterns

`rag-retrieval` (feedback-question-answering)

When users ask product-feedback questions, the system retrieves relevant customer feedback and gives it to Llama as context before generating an answer.

“When a user asks a question, RAG retrieves the most relevant customer feedback tailored to their prompt and augments it into the LLM’s context”article, stated

`retrieval-ranking` (embedding-similarity)

Prompt embeddings are compared with stored feedback embeddings using cosine similarity to retrieve the top-N most relevant matches.

“the user’s prompt is matched with relevant customer feedback by comparing its embedding to stored feedback embeddings using cosine similarity”article, stated

`data-quality-loop` (feedback-repository)

Meta mitigates hallucination and bias by building a large, broad, temporally aware feedback repository rather than relying on one narrow source.

“The first step to mitigate hallucination risk is to establish an extensive repository of customer feedback”article, stated

`privacy-preserving-processing` (customer-feedback-governance)

The feedback data used by the AI tool is deidentified and governed by Meta privacy and data-use policies.

“All customer feedback used as a data source within our AI Tool is deidentified in compliance with Meta’s privacy & data-use policies”article, stated

`confidence-thresholding` (scope-education)

Meta identifies out-of-scope prompts and tracks response usefulness so users learn which questions the tool can answer well.

“Users need to understand what types of questions the LLM can answer. To support this, we’re categorizing the themes of commonly asked questions”article, stated

Tradeoffs

How to validate AI summaries from video reviews.

Chose manual last-mile verification with supporting references over fully trusting generated review summaries.

Rationale: The tool can distill hours of video quickly, but references remain important for human verification of qualitative claims.

“To assist with manual “last-mile” verification, we also asked the LLM to provide supporting references.”article, stated

How to combine feedback with product analytics.

Chose qualitative customer feedback alongside quantitative metrics over a metrics-only product-analytics practice.

Rationale: Meta argues that feedback explains the contextual why behind factual metric movements.

“Combining the strengths of both quantitative (the “factual what”) and qualitative data (the “contextual why”), can deliver a more nuanced understanding”article, stated

Gotchas (what bit them)

Customer feedback is noisy, biased, and unstructured.

Root cause: Anecdotal, inconsistent, inauthentic, limited-source, and freeform data can distort product insights.

Resolution: Meta builds a broad repository and contextualizes feedback before retrieval and summarization.

“The raw data is often hindered by noise, bias, and a lack of structure which can make it difficult to extract meaningful insights from.”article, stated

Old mutable feedback can mislead current product decisions.

Root cause: Discussions about fixed bugs or implemented feature requests can continue to appear in feedback history.

Resolution: Meta incorporates temporal dependence and date-range-aware questions into the data model.

“old discussions about “feature requests” that have since been implemented or “bugs” that are now fixed can provide misleading context”article, stated

LLMs can hallucinate when the data foundation is insufficient.

Root cause: Even sophisticated models lack enough relevant evidence for some customer-feedback questions.

Resolution: The repository prioritizes substantial feedback volume and a broad range of sources before RAG answer generation.

“Even the most sophisticated LLMs can falter, producing “hallucinations” or misleading answers, when they lack sufficient data to answer a question.”article, stated

Bug-report retrieval improves when more than the freeform text is embedded.

Root cause: Descriptions alone may not capture common technical traits needed for clustering and classification.

Resolution: Meta found that embedding stack traces alongside freeform descriptions improves retrieval accuracy for bug prompts.

“embedding their stack trace, in addition to their freeform description, improves retrieval accuracy for prompts related to the clustering and classification of common bug traits.”article, stated

Evaluation

Meta evaluates through response usefulness tracking, out-of-scope prompt themes, supporting references for manual verification, retrieval accuracy improvements such as stack-trace embeddings, and practical product-planning use cases.

Metrics: response usefulness · out-of-scope prompt themes · retrieval accuracy · manual verification support · manual effort reduction

“tracking the usefulness of their LLM response, and using this information to provide educational “suggestions” to new users.”article, stated

“This data is being summarized in executive reporting, condensing hours of manual effort into minutes.”article, stated

Outcomes

Meta uses the tool for bug deduplication and prioritization.
Internal testing and QA feedback can be summarized for executive reporting in minutes.
The tool supports strategic planning for Quest and Ray-Ban Meta products.
Video-review feedback can be distilled into headset use cases with supporting references.

The repository combines internal and external feedback from product reviews, bug reports, surveys, social media, video reviews, and other sources for Reality Labs products.

“Mapping bug reports to higher-level categories is helping prioritize and address the most critical areas reported by our users.”article, stated

“Knowing what people like, dislike, and desire from our Quest & Ray-Ban Meta products today will help us create better devices and experiences for tomorrow.”article, stated