AI-Powered Customer Feedback Analysis at Scale

github · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer-platform customer feedback analytics

GitHub needed to analyze large volumes of support-ticket feedback, discover recurring customer pain points, and communicate trends to product teams without manual classification bottlenecks.

Constraints:

Architecture

GitHub's internal feedback analytics system clusters support-ticket text with BERTopic, BERT embeddings, and HDBSCAN, then uses GPT-4 prompt and parameter tuning to turn topic clusters into human-readable customer pain statements. The delivery layer evolved from an ADX dashboard prototype into a tailored internal web app with business-context filters, visualization choices, and feedback loops for product teams.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support-ticket-feedback["support-ticket-feedback"]
  bertopic-topic-modeler["bertopic-topic-modeler"]
  bert-embedding-model["bert-embedding-model"]
  hdbscan-clustering["hdbscan-clustering"]
  gpt4-summarizer["gpt4-summarizer"]
  prompt-parameter-optimizer["prompt-parameter-optimizer"]
  adx-dashboard-prototype["adx-dashboard-prototype"]
  insight-delivery-application["insight-delivery-application"]
  human-feedback-loop["human-feedback-loop"]
  support-ticket-feedback --> bertopic-topic-modeler
  bertopic-topic-modeler --> bert-embedding-model
  bertopic-topic-modeler --> hdbscan-clustering
  bertopic-topic-modeler --> gpt4-summarizer
  bert-embedding-model --> hdbscan-clustering
  hdbscan-clustering --> gpt4-summarizer
  gpt4-summarizer --> prompt-parameter-optimizer
  gpt4-summarizer --> insight-delivery-application
  prompt-parameter-optimizer --> human-feedback-loop
  adx-dashboard-prototype --> human-feedback-loop
  insight-delivery-application --> human-feedback-loop
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
support-ticket-feedback Raw textual customer feedback from GitHub support tickets. bertopic-topic-modeler
bertopic-topic-modeler Open-source topic modeling framework used to create interpretable topics from feedback text. bert-embedding-model, hdbscan-clustering, gpt4-summarizer
bert-embedding-model Embedding model that represents ticket text using surrounding context and multilingual language understanding. hdbscan-clustering
hdbscan-clustering Density-based clustering layer that groups similar documents before topic words are aggregated. gpt4-summarizer
gpt4-summarizer LLM summarization layer that turns representative cluster words into full customer-pain statements. prompt-parameter-optimizer, insight-delivery-application
prompt-parameter-optimizer Prompt, temperature, max-token, top-p, frequency, and presence-penalty tuning for GPT-4 summaries. human-feedback-loop
adx-dashboard-prototype First-trial visualization surface used to collect internal feedback on useful views. human-feedback-loop
insight-delivery-application Tailored internal web app that presents filtered trends, visuals, and product-specific business context. human-feedback-loop
human-feedback-loop Product-team and internal-user feedback cycle that improves prompts, visuals, filters, and insight presentation. —

Data flow

Support-ticket feedback is collected from GitHub's support portal, embedded and clustered with BERTopic/BERT/HDBSCAN, summarized by GPT-4 into readable pain statements, then surfaced through dashboards and an internal web application. Product and engineering teams consume the output as filtered business-context stories about recurring customer pain points.

3 supporting quotes
“BERTopic combines BERT’s ability to generate high-quality document embeddings with a clustering algorithm, typically Hierarchical Density-Based Spatial Clustering of Applications with Noise”article, stated
“To ensure we could display the customer feedback insights in a more comprehensible form, we decided to summarize them using GPT-4”article, stated
“This marked the beginning of developing our internal web application to communicate insights through visuals.”article, stated

Why this architecture

GitHub combined open-source clustering and GPT-4 summarization because raw ticket volume was too large for manual classification, topic clusters alone were not human-actionable, and generic market tools could not satisfy GitHub's security, privacy, business-metric, and storytelling requirements. The internal app emerged after the dashboard prototype exposed what users actually needed.

“We then began to explore the market for AI-driven analytics solutions, but we quickly realized that we needed a tool that adhered to strict security and privacy regulations”article, stated
“The representative words generated by this model kick-started our project, but there was still a key piece missing—the outputs needed to be easily understandable by humans.”article, stated

Models used: BERTopic · BERT · HDBSCAN · GPT-4

Scale

Alternatives considered (roads not taken)

Manual support-ticket classification

Rejected because: Manual classification was time-consuming, error-prone, fatiguing, and too slow for broad customer feedback analysis.

“Manual data classification and analysis by humans is error-prone and very time-consuming, as it often involves handling vast amounts of data”article, stated

Market AI analytics solutions

Rejected because: The team needed strict security/privacy alignment and GitHub-specific business metrics for product-team storytelling.

“we needed a tool that adhered to strict security and privacy regulations and incorporated tailored business metrics”article, stated

Training models on support-ticket feedback

Rejected because: GitHub applied pre-trained models to analyze ticket text rather than training on customer feedback.

“we don’t train any models with customer feedback from support tickets. Instead, we apply the pre-trained models to analyze the feedback text data”article, stated

Keeping the initial ADX dashboard as the product surface

Rejected because: Internal feedback showed they needed a tailored tool with business-specific context, advanced filters, and links to internal systems.

“It became clear that we needed a tailored tool that incorporated business-specific context into our data.”article, stated

Patterns

retrieval-ranking (feedback-clustering)

BERTopic, BERT embeddings, and clustering group related support-ticket feedback so recurring pain points can be ranked and surfaced.

“Clustering feedback helps us identify recurring problems quicker and address them more efficiently.”article, stated

prompt-chaining (cluster-summarization)

The pipeline first discovers clusters, then uses GPT-4 prompting and parameter settings to turn topic words into readable customer-pain summaries.

“The representative words generated by this model kick-started our project, but there was still a key piece missing—the outputs needed to be easily understandable by humans.”article, stated

prompt-versioning (summarization-control)

GitHub treats prompt crafting and inference parameters as tunable controls for GPT-4 summaries rather than retraining.

“Optimizing GPT-4 without retraining the model involves adjusting the way we use it, such as adjusting prompts and setting parameters for specific tasks.”article, stated

data-quality-loop (dashboard-iteration)

The team shipped an MVP dashboard, gathered internal feedback on visuals, and iterated toward a tailored tool with business context.

“we decided to ship our minimum viable product (MVP) to start collecting feedback and iterating on the visuals most relevant to our product teams.”article, stated

privacy-preserving-processing (support-ticket-analysis)

GitHub applies pre-trained models to analyze ticket text while explicitly not training models on customer support feedback.

“we don’t train any models with customer feedback from support tickets. Instead, we apply the pre-trained models to analyze the feedback text data”article, stated

Tradeoffs

How to use customer-support feedback with models.

Chose apply pre-trained models without training on ticket feedback over training models on customer support tickets.

Rationale: This preserves privacy posture while still extracting themes and summaries from feedback text.

“we don’t train any models with customer feedback from support tickets. Instead, we apply the pre-trained models to analyze the feedback text data”article, stated

How to deliver insights to product teams.

Chose a tailored internal web application with business-specific context over generic dashboards from the first trial.

Rationale: Product teams needed stories such as top pain points for a specific product/feature, not only generic visualizations.

“It became clear that we needed a tailored tool that incorporated business-specific context into our data.”article, stated

Gotchas (what bit them)

Raw topic clusters were not actionable for humans.

Root cause: Representative words from clustering do not communicate a full sentence of customer pain.

Resolution: GitHub added GPT-4 summarization to produce clearer, human-readable insights.

“A group of words is different from a full representative sentence of customer pain.”article, stated

Market AI analytics tools did not fit GitHub's governance and storytelling needs.

Root cause: The team needed strict security/privacy compliance plus business metrics that could persuade product teams.

Resolution: GitHub built an internal AI analytics tool tailored to product areas and business context.

“we quickly realized that we needed a tool that adhered to strict security and privacy regulations and incorporated tailored business metrics”article, stated

Manual classification drained data-science capacity and delayed insight discovery.

Root cause: Humans reviewing large text volumes face fatigue, inconsistency, and high collection/organization overhead.

Resolution: Automated trend identification replaced much of the manual classification workflow.

“data scientists spend about 80% of their time on tasks like data collection and organization, including manual classification”article, stated

Evaluation

GitHub evaluates by human/internal-user feedback, A/B testing, prompt/parameter tuning, visual usefulness, and whether generated insights support product prioritization and self-service opportunities.

Metrics: internal user feedback · A/B testing · visual usefulness · trend identification · self-service opportunity discovery

“Iterative feedback. Continuously improving the model’s performance through human feedback and A/B testing.”article, stated
“This process helped us identify which visualizations our internal users found valuable and which ones were less effective.”article, stated

Outcomes

The source describes countless daily support-portal feedback items and multilingual analysis through pre-trained BERT/BERTopic capabilities.

“Using automated AI-driven trend identification has significantly enhanced our ability to scale our data analysis efforts.”article, stated
“The insights generated enable the identification of self-help opportunities that empower customers to resolve issues on their own more swiftly.”article, stated