Building a Production LLM Platform for Live Shopping and Trust & Safety

whatnot · E-commerce · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM platform for live-shopping operations, trust and safety, support, and seller workflows

Whatnot needed an internal LLM platform that let product, operations, policy, and engineering teams ship trustworthy production LLM workflows quickly, despite non-deterministic outputs, weak ground truth, tool-access complexity, provider reliability concerns, and the need for domain experts to improve prompts and evaluations without waiting on engineering for every change.

Constraints:

• the model API call is a small part of the production system around it
• LLM outputs are non-deterministic, harder to constrain, and easier for users to push in unintended directions
• high-leverage improvements often come from prompts, examples, tools, evaluation, and context rather than model rebuilding
• self-serve experimentation needed guardrails rather than ungoverned prompt changes
• production workflows required high availability, predictable performance, observability, caching, rate limiting, guardrails, and provider fallback
• Whatnot's approach depended on existing platform foundations, a modern data stack, shared tooling, logging patterns, analytics sinks, and consolidated business logic

Architecture

Whatnot organized its LLM Platform around velocity, trust, and reliability. The platform provides a UI for self-serve prompt experimentation, post-exposure logging that measures only cases where prompt variants differ, a Python-defined tool registry that exposes reusable functions through a shared catalog, LLM-as-judge calibration workflows owned by domain experts, and reliability primitives such as multi-provider routing, fallback behavior, observability, caching, rate limiting, and guardrails. Existing data/platform foundations and a consolidated codebase made it easier to expose company capabilities through the platform.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  platform-ui[Platform UI] --> prompt-experimentation-framework[Prompt experimentation framework]
  platform-ui --> tool-registry[Tool registry]
  platform-ui --> llm-judge-calibration-workflow[LLM judge calibration workflow]
  prompt-experimentation-framework --> core-llm-service[Core LLM service]
  tool-registry --> core-llm-service
  core-llm-service --> shared-middleware[Shared middleware]
  shared-middleware --> provider-routing[Provider routing]
  shared-middleware --> observability-layer[Observability layer]
  tool-registry --> company-tooling-ecosystem[Company tooling ecosystem]
  prompt-experimentation-framework --> feedback-and-analytics-data[Feedback and analytics data]
  llm-judge-calibration-workflow --> feedback-and-analytics-data
  feedback-and-analytics-data --> observability-layer
  core-llm-service --> production-workflows[Production workflows]
  production-workflows --> observability-layer

Components

Data flow

A team builds or improves an LLM workflow in the platform UI by editing prompts, selecting reusable tools from the tool catalog, and running guarded experiments through the product experimentation framework. Outputs and production data feed trust workflows where deterministic checks and calibrated LLM judges evaluate quality. Production use routes through the core LLM service, shared middleware, provider routing, observability, caching, rate limits, and guardrails so teams inherit platform reliability rather than rebuilding it.

Why this architecture

Whatnot built a platform rather than feature-specific LLM plumbing because most production value came from iteration loops, tool access, evaluation, and reliability around the model. The platform makes domain experts first-class contributors, gives prompt authors reusable tools without custom wiring, uses calibrated judges to measure open-ended quality, and centralizes reliability so teams inherit fallbacks, observability, caching, rate limits, and guardrails.

Models used: LLM provider models · LLM-as-a-judge evaluators · multi-provider LLMs

Scale

• Post-exposure logging made prompt iteration more than 10x faster.
  “That lets us focus measurement on the cases where the experiment could actually matter and makes prompt iteration meaningfully faster (10x+ faster).”article, stated
• The article gives an example where prompt variants produce identical outputs 80% of the time and differ only 20% of the time.
  “Both prompts produce the same output 80% of the time. Post-exposure logging discards those identical cases and only measures the 20% where outputs actually differ”article, stated
• One cited trust-and-safety workflow helps process harassment reports in minutes instead of hours.
  “the platform this way because the problems we are solving with LLMs are real and specific: helping a trust reviewer process a harassment report in minutes instead of hours, giving a support agent the right context to resolve a buyer’s issue on the first try, making it possible for a seller to”article, stated

Alternatives considered (roads not taken)

Treat prompt experiments like ordinary A/B exposures

Rejected because: Most LLM prompt variants produce identical outputs, diluting signal when standard A/B frameworks count every request.

Attach tools as one-off code to individual LLM features

Rejected because: Reusable business functions needed to be discoverable, schema-inspectable, and safe to test across prompts and teams.

Rely only on deterministic checks for quality

Rejected because: Mechanical checks show whether required fields/tool calls exist, not whether the answer did the right thing for the user.

Let each team rebuild reliability primitives

Rejected because: Provider failures, availability, performance, caching, rate limits, observability, and guardrails are platform concerns.

Copy the architecture without similar platform foundations

Rejected because: The approach depends on prior investments in data stack, logging, analytics, tooling, and consolidated business logic.

Patterns

prompt-versioning (experimentation)

The platform lets non-engineers test prompt variants through guarded product experimentation flows and measures only output-changing exposures.

tool-use-orchestration (tool-platform)

Engineers register reusable Python tools once, and prompt authors attach them to prompts through a discoverable shared catalog.

llm-as-judge (evaluation)

Domain experts annotate examples, define rubrics, calibrate LLM judges, and reuse them in backtesting or online evaluation.

fallback-degradation (reliability)

The platform adds multi-provider support and fallback behavior so provider issues do not immediately take production workflows down.

observability-tracing (platform-reliability)

Observability is a shared reliability primitive that lets teams monitor performance, agreement, degradation, and product drift over time.

safety-policy-enforcement (trust-and-safety)

Guardrails and evaluation support real operational workflows such as trust-review harassment reports, support resolution, seller policy answers, and ticket updates.

Tradeoffs

Build for domain-expert participation

Chose self-serve prompt experiments, tool attachment, and judge calibration for PMs, Ops leads, and policy experts over engineering-only LLM iteration workflows.

Rationale: The people closest to the user experience often know how the answer should behave and can critique outputs better than generic platform engineers.

Use post-exposure logging for LLM experiments

Chose count only output-changing prompt-variant exposures over counting every request assigned to an A/B variant.

Rationale: Experiment power improves when measurement focuses on requests where the user experience actually changed.

Centralize reusable tools

Chose shared Python tool registry and catalog over custom agent loops and one-off feature code.

Rationale: Reusable tools reduce wiring, access-control duplication, and implementation-detail exposure for prompt authors.

Use calibrated LLM judges for open-ended quality

Chose domain-expert-trained LLM judges over brittle rules, similarity metrics, and manual spot checking alone.

Rationale: Open-ended LLM outputs need rubrics closer to human review, and manual spot checks do not scale.

Make reliability a shared platform concern

Chose multi-provider support, fallbacks, observability, caching, rate limiting, and guardrails in the platform over each product team assembling reliability separately.

Rationale: Production workflows should inherit availability and operational safety by default.

Gotchas (what bit them)

Calling the model was the smallest part of production LLM work.

Root cause: LLM products need iteration, trust, reliability, constraints, feedback loops, and operational infrastructure around non-deterministic outputs.

Resolution: Whatnot built a platform with velocity, trust, and reliability pillars instead of treating model API calls as the product.

Self-serve experimentation can become unsafe if it bypasses product guardrails.

Root cause: Prompt changes are easy for non-engineers to make, but production systems still need rigor.

Resolution: Self-serve prompt experiments run through the same experimentation framework and guardrails as app product flows.

Standard A/B exposure counting diluted prompt experiments.

Root cause: Two prompt variants often produce identical outputs, so many counted exposures do not represent a changed user experience.

Resolution: Whatnot logs exposure only when variants produce different outputs.

Fluent examples can create false confidence.

Root cause: LLMs can produce convincing answers even when the underlying behavior is weak, and deterministic checks miss human-quality failures.

Resolution: Whatnot uses inspection, measurement, repeated review, calibrated LLM judges, and evolving human-review examples.

Fragmented architectures make LLM platform integration harder.

Root cause: Many languages, microservices, and ownership boundaries complicate exposing business logic as reusable tools.

Resolution: Whatnot benefited from consolidated business logic and prior investments in platform foundations.

Evaluation

Whatnot evaluates LLM workflows through guarded experiments, post-exposure logging, deterministic checks, calibrated LLM-as-judge rubrics, domain-expert annotations, backtesting, online evaluation, agreement monitoring, product-drift detection, and data-mined human-review samples.

Metrics: output-changing prompt exposure rate · deterministic validity checks such as JSON, required fields, and tool calls · LLM judge rubric scores · judge agreement over time · production degradation and product drift · rare/high-value examples surfaced for human review · prompt iteration speed

Outcomes

• Whatnot built an LLM platform organized around velocity, reliability, and trust.
• Prompt experimentation became self-serve while still using product-flow guardrails.
• Post-exposure logging made prompt iteration more than 10x faster in relevant cases.
• A shared tool catalog made live-data and action functions reusable by prompt authors.
• Calibrated LLM-as-judge workflows let domain experts define and monitor open-ended quality.
• Reliability primitives such as multi-provider support, fallback behavior, observability, caching, rate limiting, and guardrails became shared platform capabilities.
• The platform supports workflows such as harassment-report review, buyer support context, and seller policy answers.

The source emphasizes platform scale qualitatively rather than volume metrics: non-engineer prompt authors, reusable company tools, production workflows, 10x+ faster prompt iteration, minutes-vs-hours trust review, and inherited reliability for teams shipping on the platform.