Auto-Moderating Car Dealer Reviews with GenAI

edmunds · Automotive · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Dealer review moderation

Edmunds needed to publish dealer service reviews quickly while still enforcing nuanced moderation rules that were difficult to capture with manual review or fine-tuned off-the-shelf models.

Constraints:

• over 300 reviews arrived daily
• manual moderation could take up to 72 hours before publication
• reviews needed classification specifically for dealer quality of service
• ambiguous reviews and edge cases made rules difficult to encode
• the moderation system also needed better data governance, permissions, lineage, and access control

Architecture

Edmunds moved dealer review moderation to GPT-4 accessed through Databricks Model Serving endpoints with custom prompts that decide whether to accept or reject each review. Databricks Model Serving lets the team compare commercial models and custom-served models in one environment, manage permissions, and set rate limits. In parallel, Edmunds uses the Databricks Data Intelligence Platform and Unity Catalog-oriented governance to improve data lineage, security, and access control around the review moderation pipeline.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  dealer-review-submissions["dealer-review-submissions"] --> moderation-pipeline["moderation-pipeline"]
  moderation-pipeline["moderation-pipeline"] --> databricks-model-serving["databricks-model-serving"]
  databricks-model-serving["databricks-model-serving"] --> gpt-4-moderation-model["gpt-4-moderation-model"]
  custom-moderation-prompts["custom-moderation-prompts"] --> gpt-4-moderation-model["gpt-4-moderation-model"]
  databricks-model-serving["databricks-model-serving"] --> model-comparison-workbench["model-comparison-workbench"]
  databricks-model-serving["databricks-model-serving"] --> data-governance-layer["data-governance-layer"]
  moderation-pipeline["moderation-pipeline"] --> human-moderators["human-moderators"]
  moderation-pipeline["moderation-pipeline"] --> review-publication-system["review-publication-system"]
  moderation-pipeline["moderation-pipeline"] --> data-governance-layer["data-governance-layer"]

Components

Data flow

Dealer reviews enter the moderation pipeline. Databricks Model Serving calls GPT-4 with custom moderation prompts that encode the dealer-quality rules and output an accept/reject decision within seconds. Moderators oversee the process with far less manual triage, while the Databricks governance layer manages permissions, rate limits, and review data controls. Accepted reviews can be published in minutes rather than after a multi-day manual queue.

Why this architecture

Fine-tuned off-the-shelf models did not capture Edmunds’ nuanced moderation rules, so the team moved to GPT-4 with detailed custom prompts. Databricks Model Serving gave them a unified way to compare models and manage permissions/rate limits, while governance improvements addressed the data-access overhead around moderation pipelines.

Models used: GPT-4 · Databricks Model Serving · custom prompts · Databricks Data Intelligence Platform

Scale

• Edmunds received over 300 daily reviews.
  “With over 300 daily reviews submitted on both the quality of new and used cars and the dealers selling them”article, stated
• Manual moderation could take up to 72 hours.
  “sometimes requiring up to 72 hours of turnaround time to publish vetted reviews.”article, stated
• The GenAI solution makes accept/reject decisions in seconds.
  “in a matter of seconds, not hours.”article, stated
• The team saves three to five hours per week.
  “the team saves three to five hours per week.”article, stated
• The moderation team now needs only two moderators.
  “they now need only two moderators, freeing up these resources for more valuable tasks.”article, stated

Alternatives considered (roads not taken)

Manual moderation

Rejected because: Manual review took days and delayed timely car-buyer access to dealer information.

Fine-tuned off-the-shelf model

Rejected because: Fine-tuning did not handle the complex moderation rules and edge cases well enough.

Prompt-engineered off-the-shelf models outside Databricks

Rejected because: It was hard to compare outputs across models until Databricks Model Serving consolidated model options.

Patterns

structured-output-contract (classification)

The moderation prompt directs GPT-4 to produce accept/reject decisions for reviews.

model-routing (model-ops)

Databricks Model Serving lets the team switch between commercial models and custom-served models to compare performance.

prompt-versioning (prompting)

Rules that fine-tuning could not absorb are encoded in custom moderation prompts.

human-in-the-loop-gate (operations)

Automation reduced but did not eliminate the moderation team; two moderators remain in the process.

Tradeoffs

Use GPT-4 with custom prompts

Chose prompted GPT-4 via Databricks Model Serving over fine-tuned off-the-shelf moderation model.

Rationale: GPT-4 with custom prompt rules handled complex and ambiguous moderation cases better than the fine-tuned model.

Automate moderation while keeping a small team

Chose AI-assisted moderation with two moderators over fully manual moderation or full human replacement.

Rationale: Automation reduced workload and turnaround while keeping human operational oversight.

Gotchas (what bit them)

Fine-tuning did not solve nuanced review moderation.

Root cause: The rules were complex and edge cases needed flexible prompt guidance.

Resolution: Use GPT-4 through Databricks Model Serving with custom prompts that encode the moderation criteria.

Comparing off-the-shelf model outputs was hard without a unified serving layer.

Root cause: Different model providers and custom models were not easily compared in one environment.

Resolution: Use Databricks Model Serving to consolidate third-party LLMs and custom-served models in one view.

Evaluation

Edmunds evaluated model alternatives by comparing outputs through Databricks Model Serving and judged the production workflow through turnaround time, team size, and weekly staff time saved.

Metrics: turnaround time from up to 72 hours to seconds/minutes · three to five hours saved per week · two moderators needed after automation · daily review volume over 300 · model output comparison

Outcomes

• dealer service review moderation moved from manual days-long review to AI-assisted decisions in seconds
• new reviews can be analyzed and published in minutes
• the team saves three to five hours per week
• only two moderators are needed for the workflow
• Edmunds plans to expand the approach across all reviews

The source reports over 300 daily reviews, up to 72-hour manual turnaround before automation, seconds-level model decisions, and three to five staff hours saved weekly.