LLM-based Inappropriate Language Detection in User-Generated Reviews

yelp · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Trust and safety content moderation

Yelp needed to proactively detect egregious inappropriate language in reviews while maintaining high precision so legitimate reviews were not unnecessarily delayed or blocked.

Constraints:

• The classifier had to handle hate speech, lewdness, threats, harassment, personal attacks, sarcasm, metaphors, and other contextual language.
• False positives could delay evaluation of legitimate reviews and harm user experience.
• Real-time traffic has low spam prevalence, so threshold calibration had to preserve precision in production.
• Flagged content still required User Operations review before final moderation decisions.

Architecture

Yelp built an LLM-based binary classification pipeline for egregious inappropriate review detection. The team curated high-quality examples with User Operations, scored severity, augmented data with embedding similarity, used zero-shot and few-shot LLMs to label subcategories for sampling balance, fine-tuned HuggingFace LLMs, evaluated class separation with embeddings and t-SNE, calibrated thresholds with mock traffic at different spam prevalences, and deployed the model through Redshift, S3, MLFlow, MLeap, and a service container.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Historical_review_corpus["Historical review corpus"] --> User_Operations_annotation["User Operations annotation"]
  Historical_review_corpus --> Similarity_augmentation["Similarity augmentation"]
  User_Operations_annotation --> Curated_dataset["Curated dataset"]
  Similarity_augmentation --> Curated_dataset
  Subcategory_sampler["Subcategory sampler"] --> Curated_dataset
  Curated_dataset --> S3_bucket["S3 bucket"]
  S3_bucket --> Fine_tuned_LLM_classifier["Fine-tuned LLM classifier"]
  Fine_tuned_LLM_classifier --> MLFlow_registry["MLFlow registry"]
  MLFlow_registry --> Model_server["Model server"]
  Threshold_calibration["Threshold calibration"] --> Model_server
  Model_server --> User_Operations_review_queue["User Operations review queue"]

Components

Data flow

Historical and reported reviews are selected from Redshift for moderator curation, severity scoring, similarity expansion, and subcategory balancing. Curated training data is stored in S3 and used by a batch training script to fine-tune an LLM classifier. The trained model is registered in MLFlow, loaded into MLeap, served in a model-server container, and used to flag egregious reviews for User Operations review before final action.

Why this architecture

Yelp used LLMs because inappropriate-language detection requires contextual understanding beyond prior approaches, but it wrapped them in human curation, class balancing, threshold calibration, and manual review to protect high-precision moderation.

Models used: Large Language Models · HuggingFace LLMs · sentence embeddings · zero-shot classification · few-shot classification · fine-tuned binary classifier · t-SNE · MLeap

Scale

• More than 26,500 reported reviews were removed in 2022 for threats, lewdness, and hate speech.
  “In 2022, 26,500+ reported reviews were removed from Yelp’s platform”article, stated
• The deployed system prevented more than 23,600 reviews from publishing in 2023.
  “it enabled our moderators to proactively prevent 23,600+ reviews from ever publishing to Yelp in 2023.”article, stated
• The production calibration used mock traffic with different spam prevalence levels.
  “we generated many sets of mock traffic data with different degrees of spam prevalence.”article, stated

Alternatives considered (roads not taken)

Traditional ML or previous automated moderation approaches alone

Rejected because: Prior approaches did not provide the precision-recall balance Yelp wanted, prompting LLM adoption.

Train only on raw historical inappropriate-review labels

Rejected because: Raw labels lacked enough precision and subcategory coverage, so Yelp added moderator curation, severity scoring, similarity augmentation, and sampling.

Ship from balanced-test metrics without real-traffic prevalence simulation

Rejected because: The team needed to assess false positives under low-prevalence real traffic and choose a high-precision threshold.

Patterns

fine-tuning-adaptation (model-adaptation)

Yelp fine-tuned a HuggingFace LLM on a curated binary classification dataset for inappropriate-review detection.

human-in-the-loop-gate (safety)

Human moderators helped curate labels and reviewed model-flagged content before final moderation decisions.

confidence-thresholding (routing)

Yelp simulated real-time traffic at different spam prevalences to choose a threshold that identifies inappropriate reviews within an accepted confidence range.

Tradeoffs

Adopt LLM classification for egregious moderation cases.

Chose Fine-tuned LLMs over Continuing only with prior non-LLM moderation approaches.

Rationale: The team needed better precision-recall behavior for contextual language categories.

Favor high precision in production moderation.

Chose Threshold calibration for high precision over A less stringent recall-maximizing threshold.

Rationale: False positives can delay legitimate reviews, while low prevalence makes threshold choice sensitive.

Keep human review after automated flagging.

Chose Model-assisted moderation with User Operations review over Fully automated removal decisions.

Rationale: The article positions the model as a proactive surfacing system while final decisions remain with moderators.

Gotchas (what bit them)

Balanced-test performance was not enough to ship safely.

Root cause: In real traffic, inappropriate-review prevalence is low, so false positives can dominate if thresholds are set from balanced data alone.

Resolution: Simulate mock traffic at different spam prevalence levels and choose a high-precision confidence threshold.

Historical inappropriate-review data lacked enough category coverage.

Root cause: Many curated reviews did not include subcategory information for the forms of inappropriate content.

Resolution: Use zero-shot and few-shot LLM classification to label subcategories, then under-sample or over-sample where needed.

Language complexity made generic labels insufficient.

Root cause: Metaphors, sarcasm, and figures of speech complicate inappropriate-language detection.

Resolution: Collaborate with User Operations to curate severe examples and define the task precisely.

Evaluation

Yelp evaluated embedding separation with silhouette score and t-SNE, fine-tuned on curated data, checked class-balanced test metrics, and simulated real-time traffic with different spam prevalence levels to choose a confidence threshold.

Metrics: silhouette score · t-SNE separation · balanced test data metrics · false positive rate in simulated real-time traffic · confidence threshold

Outcomes

• The model was deployed to live traffic for egregious inappropriate-language detection.
• The system proactively prevented more than 23,600 reviews from publishing in 2023.
• Yelp expects additional recall improvements based on moderator decisions and retraining.

The source gives concrete moderation volumes: 26,500+ reported reviews removed in 2022 and 23,600+ reviews prevented from publishing in 2023.