Automating Search Engine Marketing Ad Generation with Multi-Stage LLM Pipeline

thumbtack · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Search engine marketing ad generation

Thumbtack needed to generate personalized Google Responsive Search Ad assets at SEM scale without relying on generic templates or months of manual copy review.

Constraints:

• Google RSAs can include up to 15 headlines and 4 descriptions
• headlines are limited to 30 characters and descriptions to 90 characters
• the old template made 80% of assets generic
• LLM outputs need validation for character limits, best practices, and legal/compliance quality
• evaluation metrics are volatile and require robust experimentation

Architecture

Thumbtack built a multi-stage LLM pipeline for SEM ad generation. Stage one generates ad assets from ad group names, keywords, best practices, character limits, and optional custom inputs. Stage two uses an LLM as a judge to review and select preferred assets from the generated pool. Stage three separates selected assets into distinct ads to avoid repetition across ad variants. Automated validation enforces character limits and checks that review/selection steps only refine existing generated assets, after which expert reviewers approve final copy for Google upload.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ad-group-input-set[Ad group input set] --> llm-generation-stage[LLM generation stage]
  llm-generation-stage --> automated-validation-layer[Automated validation layer]
  automated-validation-layer --> llm-review-stage[LLM review stage]
  llm-review-stage --> asset-grouping-stage[Asset grouping stage]
  asset-grouping-stage --> expert-review-gate[Expert review gate]
  expert-review-gate --> google-experiments[Google experiments]

Components

Data flow

Ad group metadata, target keywords, value propositions, best practices, and character limits feed the generation prompt. Generated assets are parsed and filtered, a second LLM selects/refines the best options, grouping logic separates assets into distinct ads, automated checks enforce constraints, and expert reviewers make final selections before Google upload and experiment measurement.

Why this architecture

A multi-stage pipeline fit the ad-generation problem because output quality depends on several distinct constraints: generation must be keyword-specific, review must filter low-quality copy, grouping must avoid ad-variant overlap, and humans must approve final copy for legal and quality reasons. Measurement moved from synthetic controls to Google Drafts and Experiments because later phases needed more robust impact estimation.

Models used: large language models for generation · LLM-as-judge review model

Scale

• Each RSA can have up to 15 headlines and 4 description lines.
  “Each RSA can have up to 15 headlines and 4 description lines, from which Google selects the most relevant combination of headlines and descriptions for a given search.”article, stated
• Headlines are 30 characters and descriptions are 90 characters.
  “A headline should be only 30 characters long and a description can be 90 characters long. The headlines and descriptions within a RSA are referred to as ad assets.”article, stated
• Before automation, 80% of assets were generic across ad groups.
  “With the templated approach above, all the ad groups had the same generic content for 80% of the assets, regardless of what the ad group represents.”article, stated
• Eleven of fifteen headlines were identical across all ad groups.
  “Also, 11 of the 15 headlines in the RSA were the same across all the ad groups. This templated approach meant that ad copies were very generic and often not specific to the actual search ad group.”article, stated
• The PoC saw a 20% traffic lift and 10% conversion lift.
  “Synthetic control method was used to estimate the impact of PoC, and we observed a 20% lift in traffic and a 10% lift in conversions.”article, stated

Alternatives considered (roads not taken)

Manual template-based RSA creation

Rejected because: Templates produced generic assets and manual keyword personalization was infeasible at scale.

Generation-only LLM pipeline

Rejected because: Later phases added automated selection, review, and separation to reduce manual review and repeated assets.

Synthetic-control measurement for all phases

Rejected because: Volatile metrics made synthetic controls hard for phases 2 and 3, so Thumbtack used Google D&E in phase 4.

Patterns

prompt-chaining (workflow)

Generation, automated review/selection, and grouping are separate LLM pipeline stages.

llm-as-judge (evaluation)

A second LLM judges and selects a refined subset of ad assets.

guardrails-io-filtering (reliability)

Validation filters irrelevant content, character-limit failures, and unvalidated new content in review steps.

human-in-the-loop-gate (workflow)

SEM experts review final copy before upload to Google.

eval-driven-development (evaluation)

Thumbtack used PoC lift, synthetic controls, and Google Drafts and Experiments to validate rollout phases.

Tradeoffs

Keep expert review for final upload

Chose automated filtering plus human final selection over fully automated Google upload.

Rationale: Legal compliance and quality control still required human oversight.

Use multiple LLM stages

Chose generation plus judge plus grouping over one-shot ad generation.

Rationale: Separate stages handle different risks: personalization, quality filtering, and ad-variant diversity.

Gotchas (what bit them)

Manual templates waste ad slots with repeated generic copy.

Root cause: Most ad assets did not include service-specific keywords or value propositions.

Resolution: The LLM generation stage inserts top keywords and ad-group-specific context.

LLM review stages can introduce unvalidated content.

Root cause: A review/selection model might generate new copy instead of selecting from the original generated pool.

Resolution: Checks flag if LLMs generate new unvalidated content during review or separation.

SEM metrics can be volatile.

Root cause: The team found synthetic controls hard to use in phases 2 and 3.

Resolution: Phase 4 used Google Drafts and Experiments for more robust measurement.

Evaluation

Thumbtack evaluated through a phase-1 synthetic-control PoC and a later Google Drafts and Experiments phase-4 test.

Metrics: traffic lift · conversion lift · CTR · conversion value per request · impressions · ROAS

Outcomes

• The PoC saw 20% traffic lift and 10% conversion lift.
• The latest rollout showed statistically significant improvements in impressions, CTR, and conversions.
• Phase 4 showed higher CTR and conversion value per request for LLM ads.
• The approach saved several months of review and enabled personalization that had been nearly impossible manually.

The pipeline generated personalized assets across SEM ad groups under Google RSA constraints of 15 headlines, 4 descriptions, 30-character headlines, and 90-character descriptions.