Democratizing Prompt Engineering Through Platform Architecture and Employee Empowerment

pinterest · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise prompt platform and employee enablement

Pinterest needed to turn prompt experimentation into production capability across the company, letting employees and teams use multiple LLM vendors safely while scaling labeling, prompt evaluation, and prompt optimization.

Constraints:

• Pinterest serves about 570 million users.
• Teams wanted rapid access to multiple models and unreleased models.
• Central operations needed rate limits, logging, vendor integration, access control, and guardrails.
• Non-engineers needed ways to build and test prompts without writing code.
• Batch labeling had to scale to millions of labels while controlling cost, accuracy, and safety.

Architecture

Pinterest built a layered LLMOps platform: multimodal/multi-vendor model onboarding, centralized proxy services for vendor integration, logging, rate limiting, and access control, common employee-facing tools, centralized guardrails, batch labeling, prompt leaderboards, and AutoPrompter-style prompt generation and critique. The organizational layer included Prompt Doctors, company-wide classes, hackathons, bounties, and no-code tools.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  multi-vendor-model-layer["Multi-vendor model layer"] --> central-proxy["Central proxy"]
  central-proxy --> employee-tools["Employee tools"]
  central-proxy --> central-guardrails["Central guardrails"]
  employee-tools --> batch-labeling-system["Batch labeling system"]
  batch-labeling-system --> prompt-leaderboard["Prompt leaderboard"]
  employee-tools --> prompt-leaderboard
  prompt-leaderboard --> autoprompter["AutoPrompter"]
  prompt-doctors["Prompt Doctors"] --> employee-tools
  central-guardrails --> batch-labeling-system

Components

Data flow

A team chooses a model through the multi-vendor layer. Requests pass through the proxy for rate limits, logging, vendor integration, and access policy. Employees use prompt tools, notebooks, or prompt hub/leaderboards to develop and test prompts. Batch labeling scales successful prompts to large data tables, and AutoPrompter generates, critiques, and refines prompts for candidate tasks before production use.

Why this architecture

Pinterest separated model access, proxy controls, employee tools, and guardrails so teams could move quickly while centralizing operational controls. The education/hackathon layer made prompt engineering a company-wide capability instead of a single specialist team's bottleneck.

Models used: Multiple vendor LLMs · Open-source models · LLMs used for AutoPrompter

Scale

• Pinterest had about 570 million users.
  “Pinterest is a visual discovery platform uh which I hope you have seen before heard before. Uh we have about 570 million users.”transcript, stated
• One training class drew about one-third of the company organically.
  “We had onethird of the company join in and the interesting part is one-third of the company. Uh this is not like mandated from the top.”transcript, stated
• The batch labeling system became the fastest adopted platform at Pinterest.
  “just because of how quickly it got adopted. It became the fastest adopted platform at Pinterest.”transcript, stated
• The system can scale to labeling millions of items with one button.
  “we have a single button that if you think you are ready you press that button uh and it will scale up and label millions of items”transcript, stated

Alternatives considered (roads not taken)

Let one central team build all GenAI use cases

Rejected because: The talk's operating principle is to enable employees across the company to build and discover use cases.

Expose models directly without a proxy

Rejected because: The proxy centralizes rate limiting, vendor integration, logging, and access control.

Require engineering work for every labeling task

Rejected because: No-code notebooks and batch labeling let teams bring a table and prompt instead.

Patterns

model-routing (platform)

The platform exposes multiple models through a central layer so teams can onboard or access different vendors/models.

prompt-versioning (prompting)

Prompt leaderboards, secret test sets, starter prompts, and cost/accuracy comparisons make prompt iteration measurable.

batch-offline-processing (data)

The batch labeling platform scales prompts over large datasets at high throughput.

guardrails-io-filtering (safety)

Central guardrails include empathetic AI, safety, content quality, and toxicity-style checks.

llm-as-judge (optimization)

AutoPrompter uses LLM critique/refinement loops to generate better prompts.

Tradeoffs

Centralize model access through a proxy

Chose shared controls for vendors, logging, rate limits, and access over direct team-by-team model integrations.

Rationale: Central controls let Pinterest expose models broadly while administering unreleased and general models differently.

Open prompt contests to all employees

Chose company-wide leaderboard participation over only professional prompt engineers.

Rationale: Non-engineering employees produced competitive solutions and increased ownership.

Gotchas (what bit them)

Employee adoption requires enablement, not just infrastructure.

Root cause: Many employees did not yet know what GenAI is good or bad at.

Resolution: Use approachable teaching, prompt doctors, hackathons, and no-code tools.

Jupyter notebooks are useful for discovery but not production scale.

Root cause: A self-serve notebook can prove use cases but cannot handle large-scale labeling needs.

Resolution: Use initial meetings/notebooks to gather use cases, then fund a production batch labeling system.

Evaluation

Pinterest evaluates prompts and labeling through leaderboards, secret test sets, cost per million labels, accuracy, sample review, trust/safety toxicity checks, and production-readiness buttons.

Metrics: Accuracy · Cost per million labels · Secret test set score · Toxicity score · Prompt leaderboard rank · Label throughput · Adoption

Outcomes

• The batch labeling platform became Pinterest's fastest adopted platform.
• Pinterest trained roughly one-third of the company through an organically attended class.
• Hackathon participants beat professionally curated prompts in about two weeks.
• The platform could scale prompts to millions of item labels.
• AutoPrompter demonstrated AI-generated prompt competition against humans.

Pinterest's platform supported a 570-million-user product and company-wide prompt enablement.