Expert-in-the-Loop Generative AI for Creative Content at Scale

stitch_fix · E-commerce · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce creative content generation

Stitch Fix needed to generate ad headlines and product descriptions at e-commerce scale without losing brand tone, product truthfulness, nuanced language, or expert quality control.

Constraints:

• Ad headlines must be engaging because they are often a first interaction with potential clients.
• Traditional copywriting for every new ad asset is time-consuming and costly.
• Hundreds of thousands of product styles require descriptions that are accurate, detailed, and brand-aligned.
• Few-shot prompting that worked for ad headlines produced generic, limited-quality product descriptions.
• Client-facing fashion language requires human judgment about tone, sentiment, nuance, and brand fit.
• Outputs needed to be original, unique, natural, truthful, and aligned with Stitch Fix brand guidelines.

Architecture

Stitch Fix built an expert-in-the-loop content pipeline with two paths. For ad headlines, outfit images and style keywords are mapped into latent style space, nearby style terms are selected, GPT-3 generates brand-toned headlines, and copywriters review and edit the results. For product descriptions, human experts write several hundred examples from product attributes, a base language model is fine-tuned on that task-specific dataset, and experts continue to evaluate and improve generated copy through quality assurance.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ad-asset-intake["Ad asset intake"] --> latent-style-matcher["Latent style matcher"]
  latent-style-matcher --> gpt-3-headline-generator["GPT-3 headline generator"]
  gpt-3-headline-generator --> copywriter-review-loop["Copywriter review loop"]
  product-attribute-prompts["Product attribute prompts"] --> expert-written-training-set["Expert-written training set"]
  expert-written-training-set --> fine-tuning-pipeline["Fine-tuning pipeline"]
  fine-tuning-pipeline --> fine-tuned-description-model["Fine-tuned description model"]
  product-attribute-prompts --> fine-tuned-description-model
  fine-tuned-description-model --> copywriter-review-loop
  fine-tuned-description-model --> blind-evaluation["Blind evaluation"]
  blind-evaluation --> quality-assurance-feedback["Quality assurance feedback"]
  copywriter-review-loop --> quality-assurance-feedback
  quality-assurance-feedback --> fine-tuning-pipeline

Components

Data flow

Ad assets flow from outfit images and style keywords into latent style matching, GPT-3 headline generation, and copywriter review/editing. Product descriptions flow from product attributes into expert-written completions, fine-tuning of a pre-trained language model, generated PDP descriptions, blind evaluation, and continuing expert QA feedback.

Why this architecture

Stitch Fix used two different modeling strategies because headline generation and product descriptions had different scale and quality needs. Few-shot GPT-3 was enough to draft many brand-toned headlines when experts reviewed them, but product descriptions required fine-tuning on expert-written examples because generic few-shot descriptions were too limited for hundreds of thousands of inventory styles.

Models used: GPT-3 · Few-shot learning · Latent style understanding · Word embeddings · Fine-tuned pre-trained language model · Task-specific product-description dataset

Scale

• Stitch Fix used the expert-in-the-loop system for all ad headlines for Facebook and Instagram campaigns.
  “We have since used the expert-in-the-loop approach for generating all ad headlines for Facebook and Instagram campaigns”article, stated
• Product-description generation targeted hundreds of thousands of styles in inventory.
  “Writing descriptions for hundreds of thousands of styles in inventory is a daunting task for human copywriters alone”article, stated
• The product-description training set contained several hundred expert-written examples.
  “we gathered a task-specific dataset by having our human experts write several hundred high-quality product descriptions”article, stated

Alternatives considered (roads not taken)

Manual copywriting for every ad asset and product description.

Rejected because: Manual creation was too time-consuming and costly for repeated ad assets and inventory-scale product descriptions.

Use the ad-headline few-shot approach for product descriptions.

Rejected because: Few-shot prompting produced generic and limited-quality descriptions for PDP use.

Purely algorithmic client-facing text generation.

Rejected because: Algorithms miss subtle tone, sentiment, and appropriateness judgments that human fashion experts can make.

Patterns

human-in-the-loop-gate (creative quality control)

Copywriters review and edit AI-generated headlines and provide feedback for ongoing quality improvement.

fine-tuning-adaptation (brand voice adaptation)

Stitch Fix fine-tuned a base model on expert-written product descriptions to adapt it to the company's language, style, and template.

eval-driven-development (content quality measurement)

The team used blind evaluation and regular QA checks to compare generated descriptions and improve the model.

data-quality-loop (expert feedback)

Expert feedback about brand alignment and expression quality is fed back into algorithm improvement and QA.

Tradeoffs

How to scale ad-headline creation.

Chose GPT-3 headline candidates reviewed by copywriters over Copywriters writing every headline from scratch.

Rationale: The hybrid path saved copywriter time while preserving human oversight of style and brand fit.

How to handle product-description quality at inventory scale.

Chose Fine-tuned model trained on expert-written examples over Generic few-shot LLM generation.

Rationale: Fine-tuning learned the specific language, style, and requirements needed for Stitch Fix product descriptions.

How much human oversight to keep in a scaled content pipeline.

Chose Expert-in-the-loop review and improvement over Fully automated text generation.

Rationale: Human expertise protects client-facing quality, appropriateness, truthfulness, and brand alignment.

Gotchas (what bit them)

Few-shot headline prompting did not transfer cleanly to product descriptions.

Root cause: PDP descriptions require richer accuracy, product detail, and brand voice than short ad headlines.

Resolution: Fine-tune a base model on expert-written product-description examples.

Pure algorithmic generation can miss language nuance.

Root cause: Fashion copy depends on subtle tone, sentiment, and expression choices.

Resolution: Keep human experts in the loop to define good output and distinguish nuanced language choices.

Generated text can drift from brand messaging even when it is creative.

Root cause: Some fashion-forward wording may not fit Stitch Fix's desired brand expression.

Resolution: Use copywriter review and regular QA checks to catch wording that does not align with brand messaging.

Evaluation

Stitch Fix evaluated creative content through copywriter review, regular quality assurance checks, and blind comparisons of AI-generated product descriptions against written descriptions.

Metrics: Blind evaluation quality scores · Copywriter time savings · Brand tone and messaging alignment · Truthfulness of product statements · Quality-assurance feedback

Outcomes

• Stitch Fix used the expert-in-the-loop system for all Facebook and Instagram ad headlines.
• Copywriters saved significant time by reviewing and editing generated content instead of starting from scratch.
• Fine-tuned product descriptions achieved higher quality scores than written descriptions in a blind evaluation.
• The workflow created a continuous improvement loop between experts and algorithms.

The product-description use case targets hundreds of thousands of inventory styles, while the ad-headline system is used across all Facebook and Instagram campaign headlines.