LLM-Enhanced Trust and Safety Platform for E-commerce Content Moderation

whatnot · E-commerce · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketplace trust and safety

Whatnot needed to detect scams, harassment, and other trust-and-safety violations that unfold across conversations rather than as isolated messages.

Constraints:

• The existing rule engine operated on scalar values and worked well for structured enforcement, but not ambiguous conversational context.
• Traditional moderation models assessed individual messages in isolation and missed multi-message social-engineering patterns.
• Scammers targeted new users and tried to move conversations off-platform after benign initial exchanges.
• The system needed to feed enforcement decisions into an existing rule engine and ops process rather than making the LLM the sole decision maker.

Architecture

Whatnot upgraded its rule-engine foundation into a Rule Engine++ pattern: user signals such as messaging patterns and account age qualify which cases go to an LLM, conversation history and known scam patterns are passed in a structured prompt, the LLM returns JSON likelihood and explanation fields, and the existing rule engine combines those scalar outputs with account and behavior signals to close, act, or escalate. The broader T&S stack follows gather, evaluate, and enforce phases with Kafka-backed product-access updates after confirmed action.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Signal_qualifiers["Signal qualifiers"] --> LLM_evaluation_layer["LLM evaluation layer"]
  Conversation_history["Conversation history"] --> LLM_evaluation_layer
  Known_scam_patterns["Known scam patterns"] --> LLM_evaluation_layer
  OCR_attachment_input["OCR attachment input"] --> Conversation_history
  LLM_evaluation_layer --> Centralized_rule_engine["Centralized rule engine"]
  Centralized_rule_engine --> Enforcement_matrix["Enforcement matrix"]
  Enforcement_matrix --> Ops_team["Ops team"]
  Centralized_rule_engine --> Kafka_product_access_updates["Kafka product-access updates"]

Components

Data flow

The system gathers event data, user data, order history, ML outputs, logs, and conversation histories. Qualifying accounts are sent to the LLM with direct messages, interaction history, and known scam patterns. The LLM returns scam_likelihood and explanation JSON, which is passed with additional signals into the rule engine. The rule engine recommends close, act, or escalate; confirmed actions notify users and update product access through Kafka.

Why this architecture

Whatnot preserves the rule engine's efficient structured enforcement while adding LLM context understanding where fraud patterns span multiple messages. The LLM produces scalar likelihoods and explanations that fit back into the existing enforcement matrix.

Models used: LLMs · GPT-4 · zero-shot learning LLM predictions · few-shot learning LLM predictions · traditional ML content moderation models · OCR

Scale

• Whatnot describes itself as one of the fastest-growing marketplaces in the world.
  “As one of the fastest-growing marketplaces in the world”article, stated
• The scam detector proactively detects over 95% of scam attempts within a few minutes.
  “Using automated detection we are able to proactively detect over 95% of scam attempts on the platform within a few minutes.”article, stated
• Whatnot reports 96% precision and high recall from the LLM output.
  “We have seen 96% precision and high recall with the LLM output.”article, stated
• The system uses a scam_likelihood threshold above 0.6 in the example rule.
  “scam_likelihood > 0.6 and account_age < X days and message_frequency > Y and lifetime_orders < Z”article, stated

Alternatives considered (roads not taken)

Use only the centralized rule engine

Rejected because: Scalar rules could not handle ambiguous conversational context.

Moderate each message in isolation

Rejected because: Individual scam messages can appear benign, while the full exchange reveals the pattern.

Let the LLM be the enforcement decision maker

Rejected because: Whatnot frames GenAI as a cognitive partner and routes LLM outputs through rule engine logic and human judgment.

Patterns

safety-policy-enforcement (safety)

LLM scam likelihoods and explanations are transformed into rule-engine signals for warnings, suspensions, escalations, and product-access changes.

confidence-thresholding (routing)

The example enforcement rule combines a scam_likelihood threshold with account age, message frequency, and lifetime orders before action.

human-in-the-loop-gate (governance)

The platform escalates uncertain cases and passes LLM output to ops for investigation or action.

Tradeoffs

Augment the rule engine instead of replacing it.

Chose LLM-enhanced Rule Engine++ over A standalone LLM moderation system.

Rationale: The rule engine already handled structured data and enforcement, while LLMs added conversation-level context.

Use full conversation context for scam detection.

Chose Conversation-level LLM analysis over Single-message scam scoring.

Rationale: Scam intent emerges from the exchange pattern rather than any one message.

Keep humans in the safety loop.

Chose Cognitive partner with human review/escalation over LLM as final decision maker.

Rationale: Trust and safety enforcement needs judgment, escalation, and confirmed actions rather than raw model decisions.

Gotchas (what bit them)

Message-level moderation missed scam trajectories.

Root cause: Fraudsters build trust across multiple benign-looking messages before moving users off-platform.

Resolution: Analyze the full exchange, account signals, attachments, and engagement history with an LLM before enforcement.

Structured rules could not reason over ambiguous context.

Root cause: The rule engine was built around distinct scalar values.

Resolution: Convert LLM contextual analysis into scalar likelihood and explanation outputs that the rule engine can consume.

Attackers moved policy-violating text into images.

Root cause: Fraud tactics evolved to embed text in message attachments rather than normal text messages.

Resolution: Add OCR over message attachments and feed the extracted text as additional LLM input.

Evaluation

Whatnot evaluates the scam detector using scam attempt detection rate, time-to-detection, precision, recall, threshold-based rules, and human ops investigation for uncertain or actionable cases.

Metrics: over 95% scam attempt detection · within a few minutes · 96% precision · high recall · scam_likelihood > 0.6 example threshold

Outcomes

• The LLM-enhanced flow detects more than 95% of scam attempts within minutes.
• Whatnot reports 96% precision and high recall from LLM output.
• The flow expanded beyond scams to off-platform transactions and harassment.
• OCR was added to handle text embedded in images.

The source does not provide total message volume, but it gives operational metrics for scam detection rate, precision, and response time.