Building AI Assist: LLM Integration for E-commerce Product Listings

mercari · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Consumer marketplace listing assistance

Mercari wanted to use LLMs to improve seller listing titles while handling user-generated content, substantial live traffic, model cost/latency, inconsistent outputs, hallucination risk, and third-party API behavior.

Constraints:

• The feature operates on user-generated listing titles and descriptions.
• Real-time generation must balance quality, latency, and cost.
• Commercial LLM APIs can return inconsistent formats at scale.
• The marketplace already has moderation rules that may differ from third-party model provider policies.

Architecture

Mercari AI Assist separates title suggestion into an offline GPT-4 attribute extraction path and a real-time GPT-3.5-turbo suggestion path. The team evaluates prompts offline and online, stores key title attributes in a database, preprocesses user-generated prompts, and post-processes model responses before returning output to the client.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  seller[Seller] --> mercari-app[Mercari App]
  mercari-app --> ai-assist-service[AI Assist Service]
  gpt-4-offline-extractor[GPT-4 Offline Extractor] --> title-attribute-database[Title Attribute Database]
  gpt-4-offline-extractor --> offline-evaluation[Offline Evaluation]
  title-attribute-database --> ai-assist-service
  ai-assist-service --> prompt-preprocessor[Prompt Preprocessor]
  prompt-preprocessor --> gpt-3-5-turbo-runtime[GPT-3.5 Turbo Runtime]
  gpt-3-5-turbo-runtime --> response-postprocessor[Response Postprocessor]
  response-postprocessor --> ai-assist-service
  offline-evaluation --> online-evaluation[Online Evaluation]

Components

Data flow

Domain experts define what makes a good listing title, GPT-4 distills key title attributes from existing data for storage, the live listing flow sends new user-generated content to GPT-3.5-turbo to identify those attributes and generate title suggestions, and pre/post-processing protects the client from bad prompts and malformed responses.

Why this architecture

The split architecture uses GPT-4 where quality matters and latency is less critical, while using GPT-3.5-turbo for real-time suggestions to balance cost and performance.

Models used: GPT-4 · GPT-3.5-turbo · commercial LLM APIs

Scale

• The live feature had to operate with substantial real-time marketplace traffic.
  “this is particularly significant because we are dealing with user-generated content and substantial traffic in real-time.”article, stated
• Japanese prompt and generated token costs can exceed English by more than double in cited ChatGPT experiments.
  “using ChatGPT, the average cost of prompt and generated tokens in Japanese can exceed that of English by more than double.”article, stated
• Mercari used a preliminary online test with a small segment of AI Assist before full customer release.
  “We conducted a partial release, only implementing a small segment of Mercari AI Assist that calls the LLM API, to assess performance and confirm that the complete feature is ready for our customer base.”article, stated

Alternatives considered (roads not taken)

Use GPT-4 for all real-time operations.

Rejected because: GPT-4 had better quality but higher latency and cost, so Mercari limited it to offline attribute extraction.

Release the complete feature without a preliminary online LLM test.

Rejected because: The team needed to observe live API behavior and confirm readiness under user-generated traffic.

Trust raw LLM output formatting.

Rejected because: Inconsistent response formats increased with request volume, requiring post-processing before client delivery.

Patterns

model-routing (model-selection)

Mercari routes quality-sensitive offline extraction to GPT-4 and real-time title suggestion to GPT-3.5-turbo.

cost-aware-routing (model-selection)

The architecture explicitly balances quality, cost, latency, and token usage before release.

eval-driven-development (evaluation)

Offline and online evaluations gate prompt selection, cost estimation, and live readiness.

structured-output-contract (reliability)

The team tests and enforces simple output contracts such as YES/NO extraction, then adds post-processing when the contract fails.

guardrails-io-filtering (safety)

Mercari uses prompt preprocessing and response post-processing around user-generated content and malformed model output.

Tradeoffs

Model split

Chose GPT-4 offline plus GPT-3.5-turbo online over One model for every step.

Rationale: The split preserves GPT-4 quality where latency matters less and uses GPT-3.5-turbo for the live path.

Feature rollout

Chose Partial preliminary release of the LLM API path over Full release without live API validation.

Rationale: The partial release exposed response-format inconsistency before broad release.

Output handling

Chose Preprocessing and post-processing around LLM calls over Directly trusting prompts and model responses.

Rationale: User-generated content and output inconsistencies make raw model interaction unsafe for a client-facing feature.

Whether to use LLMs at all

Chose Use LLMs for unstructured listing text understanding over Applying LLMs indiscriminately to every task.

Rationale: Mercari explicitly frames LLM use as task-dependent based on cost, objectives, and expertise.

Gotchas (what bit them)

Output format drift increased with request volume.

Root cause: Commercial LLMs did not reliably follow the simple YES/NO contract under live traffic.

Resolution: Detect inconsistencies and enforce expected formats with post-processing before sending results to the client.

Hallucinations become harder to detect as output complexity increases.

Root cause: More complex LLM outputs make both inconsistency and hallucination detection harder than simple format checks.

Resolution: Use preprocessing, post-processing, and evaluation as outputs become more complex.

Third-party content moderation can conflict with the product's own moderation system.

Root cause: Provider policy filters may trigger differently from Mercari's internal moderation rules.

Resolution: Understand provider-specific error modes and prepare prompts to avoid undesired outcomes.

Token cost depends on language.

Root cause: Japanese prompts and outputs can consume more costly tokenization than English.

Resolution: Include language-dependent token cost in model and prompt design decisions.

Evaluation

Mercari combines offline evaluation for prompts, token usage, response quality, manual/automated review, and cost estimation with online partial-release evaluation under live user-generated traffic.

Metrics: Token usage · Response quality · Estimated cost for all users · Live output-format consistency · API performance under traffic

Outcomes

• Mercari shipped a title-suggestion architecture for AI Assist using commercial LLMs.
• The team discovered live output-format inconsistencies and elevated pre/post-processing as a production requirement.
• The case records practical operating lessons around model choice, cost/performance, stability, hallucination, and moderation.

The article emphasizes substantial real-time traffic, all-user cost estimation, and Japanese token cost potentially more than double English.