LLM-Enhanced Topic Modeling System for Qualitative Text Analysis

qualit · Research & Academia · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Qualitative text analysis and topic modeling

Amazon researchers needed to make large volumes of qualitative free-text data easier to analyze while preserving nuanced, interpretable topics that standard topic-modeling methods often miss.

Constraints:

• Manual analysis of open-ended qualitative responses is labor-intensive at large volume.
• LDA and other standard topic-modeling approaches struggle with contextual nuance and ambiguity in natural language.
• A single document can contain multiple connected themes, so assigning each document to one topic loses information.
• LLM-extracted key phrases require reliability checks so hallucinated or poorly aligned phrases do not pollute clustering.

Architecture

QualIT combines pretrained LLM key-phrase extraction with traditional hierarchical clustering. The LLM extracts multiple salient key phrases from each document, a coherence score filters hallucinated or weakly aligned key phrases, primary clustering groups phrases into high-level themes, and secondary clustering finds granular subtopics inside each theme. Human reviewers and benchmark comparisons validate the resulting topics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Free_text_corpus["Free-text corpus"] --> LLM_key_phrase_extractor["LLM key-phrase extractor"]
  LLM_key_phrase_extractor --> Coherence_scorer["Coherence scorer"]
  Coherence_scorer --> Primary_clustering["Primary clustering"]
  Primary_clustering --> Secondary_clustering["Secondary clustering"]
  Secondary_clustering --> Topic_hierarchy["Topic hierarchy"]
  Topic_hierarchy --> Human_validation["Human validation"]

Components

Data flow

Free-text documents are passed to an LLM for key-phrase extraction. Each extracted phrase receives a coherence score against the source text; low-scoring phrases are removed as possible hallucinations. The remaining key phrases are clustered into primary themes and then subclustered into more specific topics, giving analysts a navigable hierarchy from broad topics to details.

Why this architecture

The architecture uses LLMs for language understanding and phrase generation, but keeps clustering and coherence filtering explicit so the output remains interpretable, less noisy, and benchmarkable against classical topic modeling.

Models used: pretrained LLMs · QualIT · hierarchical clustering · coherence scoring · LDA · BERTopic

Scale

• QualIT was evaluated on the 20 Newsgroups benchmark dataset.
  “We evaluated QualIT on the 20 Newsgroups dataset, a widely used benchmark for topic-modeling research.”article, stated
• QualIT achieved 70% topic coherence compared with 65% and 57% for benchmark methods.
  “QualIT demonstrated substantial improvements in both topic coherence (70% vs. 65% and 57% for the benchmarks)”article, stated
• QualIT achieved 95.5% topic diversity compared with 85% and 72% for benchmark methods.
  “topic diversity (95.5% vs. 85% and 72%).”article, stated
• When at least three of four evaluators agreed, QualIT reached 50% ground-truth overlap versus 25% for LDA and BERTopic.
  “when at least three out of four evaluators agreed on the topic classification, QualIT achieved a 50% overlap with the ground truth, compared to just 25% for LDA and BERTopic.”article, stated

Alternatives considered (roads not taken)

Standard LDA topic modeling

Rejected because: LDA clusters by word co-occurrence and often misses contextual nuance and ambiguity.

Let the LLM directly generate final topics and themes

Rejected because: QualIT explicitly avoids relying only on generated topics and instead uses key phrases, coherence checks, and clustering.

Cluster full documents directly

Rejected because: Clustering validated key phrases reduces noise and focuses on thematic essence.

Patterns

confidence-thresholding (quality-control)

QualIT scores extracted key phrases for coherence and removes phrases below a threshold as possible hallucinations.

eval-driven-development (evaluation)

The researchers compared QualIT against LDA and BERTopic with coherence, diversity, and human ground-truth overlap metrics.

Tradeoffs

Use LLM key phrases plus clustering rather than only LLM-generated topics.

Chose LLM extraction with hierarchical clustering over Direct LLM topic generation.

Rationale: This keeps the LLM's language understanding while preserving an interpretable clustering structure.

Cluster key phrases rather than full documents.

Chose Key-phrase clustering over Full-document clustering.

Rationale: Key phrases reduce noise and let a single document contribute multiple themes.

Gotchas (what bit them)

LLM-extracted phrases can hallucinate or misalign with the text.

Root cause: The LLM generates key phrases and may produce phrases that do not cohere with the actual document.

Resolution: Calculate a coherence score for each key phrase and remove low-scoring phrases.

Single-topic document assumptions miss qualitative complexity.

Root cause: One text can contain multiple connected themes and perspectives.

Resolution: Extract multiple key phrases per document before clustering.

Full-document clustering can be noisy.

Root cause: Irrelevant document content can dominate clustering when full documents are grouped directly.

Resolution: Cluster validated key phrases to focus on thematic essence.

Evaluation

The researchers evaluated QualIT on the 20 Newsgroups benchmark against LDA and BERTopic using topic coherence, topic diversity, and human reviewer ground-truth overlap.

Metrics: topic coherence · topic diversity · human reviewer ground-truth overlap · three of four evaluator agreement

Outcomes

• QualIT outperformed LDA and BERTopic on topic coherence and topic diversity.
• Human reviewers aligned QualIT topics with ground truth more often than baseline methods.
• The article positions QualIT-style analysis for surveys, product feedback, voice-of-customer data, chatbot questions, and customer feedback.

The source reports benchmark results on 20 Newsgroups and four-human-reviewer agreement thresholds, but not a production deployment size.