Fine-tuned LLM for Message Content Moderation and Trust & Safety

thumbtack · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: trust and safety message moderation

Thumbtack needed to detect subtle policy-violating messages in its home-services marketplace, because keyword rules and the existing CNN model could catch obvious abuse but struggled with nuanced language, sarcasm, implied threats, and costly review volume.

Constraints:

• Most marketplace messages are legitimate, so sending every message to an LLM would be expensive.
• The pipeline needed to filter inappropriate messages before they reached professionals.
• Policy violations included abusive language, job seeking, and partnership requests.
• Prompt engineering with off-the-shelf LLMs produced only 0.56 AUC.
• The deployment needed a centralized LLM service that future Thumbtack teams could reuse.

Architecture

Thumbtack productized a fine-tuned LLM moderation pipeline. Messages first pass rule-based checks and a legacy CNN pre-filter; the CNN threshold screens out most good messages so only about 20% of messages go to the LLM. A centralized LangChain-backed service integrates the fine-tuned model with internal systems, then a resolver aggregates rule/model outputs and routes suspicious messages to manual review while passing safe messages to pros.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  message-review-pipeline["Message review pipeline"] --> rule-based-engine["Rule-based engine"]
  message-review-pipeline --> cnn-pre-filter["CNN pre-filter"]
  cnn-pre-filter --> centralized-langchain-llm-service["Centralized LangChain LLM service"]
  centralized-langchain-llm-service --> fine-tuned-llm-classifier["Fine-tuned LLM classifier"]
  rule-based-engine --> results-resolver["Results resolver"]
  centralized-langchain-llm-service --> results-resolver
  results-resolver --> manual-review-team["Manual review team"]
  results-resolver --> message-review-pipeline

Components

Data flow

A customer message enters the review pipeline, rule-based checks flag obvious policy violations, the adjusted CNN model filters likely-good messages, challenging messages are sent through the centralized LangChain LLM service to the fine-tuned classifier, and the resolver either forwards suspicious cases to manual review or lets legitimate messages reach professionals.

Why this architecture

Thumbtack chose fine-tuning because prompt engineering could not reach production accuracy, then retained the CNN as a pre-filter because LLM inference is GPU-backed and expensive for all messages. LangChain centralized LLM integration so future model deployments across teams would be easier.

Models used: Fine-tuned LLM · CNN model · LangChain · Rule-based engine

Scale

• Thumbtack connects millions of people with local service businesses.
  “helps millions of people take care of their homes by connecting them with local service businesses”article, stated
• Prompt engineering on 1,000 sample messages achieved only 0.56 AUC.
  “We began by validating 1,000 sample messages, where 90% were legitimate, and 10% were suspicious.”article, stated

  “the Area Under the Curve (AUC) for the best prompts in this experiment was only 0.56”article, stated
• Fine-tuning expanded to tens of thousands of samples and reached 0.93 AUC.
  “we expanded our dataset to tens of thousands of samples for further training. As a result, we observed significant enhancements in accuracy, with the AUC increasing to 0.93.”article, stated
• The LLM processes around 20% of messages after CNN pre-filtering.
  “the LLM only processes around 20% of the total messages”article, stated
• The deployed system has processed tens of millions of messages.
  “Since deploying the new system, it has processed tens of millions of messages with robust performance.”article, stated

Alternatives considered (roads not taken)

Keyword rules and the existing CNN model alone.

Rejected because: They struggled with nuanced language, sarcasm, and implied threats.

Prompt engineering off-the-shelf LLMs.

Rejected because: The best prompts only reached 0.56 AUC, below production needs.

Send every message to the LLM.

Rejected because: Most messages were legitimate and GPU-backed LLM review of every message would be inefficient and costly.

Build separate LLM integrations per team.

Rejected because: Thumbtack needed consistency and reuse across teams through a centralized service.

Patterns

fine-tuning-adaptation (trust and safety classification)

Thumbtack fine-tuned an LLM on tens of thousands of labeled marketplace messages to detect nuanced policy violations.

cost-aware-routing (LLM pre-filtering)

The legacy CNN was repurposed as a pre-filter so only harder suspicious cases reach the LLM.

safety-policy-enforcement (marketplace message moderation)

The pipeline detects abusive language, job seeking, partnership requests, and other policy violations before delivery.

tool-use-orchestration (centralized LLM service)

LangChain bridges internal systems to LLM models through one centralized service reusable across teams.

Tradeoffs

Fine-tune a classifier rather than rely on prompts.

Chose Fine-tuned LLM over Prompt-engineered off-the-shelf LLM.

Rationale: Prompt engineering reached only 0.56 AUC, while fine-tuning achieved 0.93 AUC.

Use the CNN as a routing filter instead of retiring it.

Chose CNN pre-filter plus LLM for harder cases over LLM-only moderation.

Rationale: The CNN could cheaply pass through most good messages, reducing LLM cost while preserving quality.

Centralize LLM integration through LangChain.

Chose Centralized LangChain-backed service over Ad hoc LLM integrations.

Rationale: A single service improves consistency and makes future LLM deployments faster and manageable across teams.

Gotchas (what bit them)

Prompt engineering was far below production quality.

Root cause: Off-the-shelf prompts could not reliably distinguish legitimate from suspicious marketplace messages.

Resolution: Thumbtack shifted to fine-tuning with thousands and then tens of thousands of labeled samples.

LLM review of every message would be too expensive.

Root cause: The LLM runs on GPU and most platform messages are legitimate.

Resolution: Thumbtack adjusted the CNN threshold to route only about 20% of messages to the LLM.

Existing CNN moderation missed subtle violations.

Root cause: Nuance, sarcasm, and implied threats require more context than the CNN handled well.

Resolution: Thumbtack used a fine-tuned LLM for more difficult message review cases.

Evaluation

Thumbtack evaluated prompt-engineered models on a 1,000-message sample, then fine-tuned on thousands to tens of thousands of examples and measured AUC, precision, and recall after deployment.

Metrics: Prompt-engineering AUC 0.56 · Fine-tuned AUC 0.93 · LLM processes around 20% of messages · Precision improved by 3.7x · Recall improved by 1.5x · Tens of millions of messages processed

Outcomes

• Fine-tuning raised AUC to 0.93 for production deployment.
• The CNN pre-filter reduced LLM processing to about 20% of messages.
• The deployed system processed tens of millions of messages, with 3.7x precision and 1.5x recall improvement.

Thumbtack's marketplace serves millions of people; the moderation system has processed tens of millions of messages and routes only about 20% to the LLM.