Optimizing Email Engagement Using LLMs and Rejection Sampling
nextdoor · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)
Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.
Problem
Domain: Email subject-line optimization
Nextdoor needed AI-generated email subject lines to outperform user-generated baselines rather than simply produce fluent but lower-engagement copy.
Constraints:
- Initial ChatGPT API subjects generated only 56% of baseline clicks.
- The system needed daily monitoring, caching, retries, and fallbacks.
- AI subjects should be accepted only when predicted to outperform alternatives.
Architecture
Nextdoor built a generator plus fine-tuned reward model, using rejection sampling to accept generated subject lines only when the reward model predicted higher engagement.
Diagram confidence: reconstructed (reconstructed from prose).
flowchart LR subject-generator["Subject Generator"] reward-model["Reward Model"] rejection-sampler["Rejection Sampler"] monitoring-and-caching["Monitoring and Caching"] email-delivery["Email Delivery"] subject-generator --> reward-model reward-model --> rejection-sampler rejection-sampler --> email-delivery monitoring-and-caching --> subject-generator
Components
| Component | Role | Interacts with |
|---|---|---|
Subject Generator |
Generates candidate email subject lines from content. | Reward Model |
Reward Model |
Predicts which subject will generate more engagement. | Rejection Sampler |
Rejection Sampler |
Accepts generated subjects only when predicted to beat baseline. | Email Delivery |
Monitoring and Caching |
Controls daily quality, cost, retries, and fallbacks. | Subject Generator |
Data flow
A subject generator creates candidates from post content, a fine-tuned reward model predicts engagement against user-generated alternatives, rejection sampling filters candidates, and production monitoring/caching/fallbacks control reliability and cost.
Why this architecture
Generation alone underperformed; reward-model filtering made AI output selective enough to beat user-generated subjects.
Models used: OpenAI API · Fine-tuned reward model
Scale
- Initial AI subjects got only 56% of baseline clicks.“only 56% of the clicks compared to baseline.”summary, stated
- Caching reduced costs by 600x.“caching (reducing costs by 600x)”summary, stated
- A/B testing showed 1% sessions lift, 0.4% WAU lift, and 1% ad revenue lift.“1% lift in sessions, 0.4% increase in Weekly Active Users, and 1% increase in ad revenue”summary, stated
Alternatives considered (roads not taken)
Use generated subject lines directly
Rejected because: Initial generated subjects underperformed user-generated baselines.
Patterns
eval-driven-development (optimization)
A reward model and A/B tests optimized subject-line engagement.
cost-aware-routing (operations)
Caching cut generation cost by 600x.
Tradeoffs
Gate generated subjects
Chose Reward-model rejection sampling over Always sending generated subjects.
Rationale: Only candidates predicted to outperform user-generated alternatives are accepted.
Gotchas (what bit them)
Fluent generated copy can still underperform human baselines.
Root cause: The generator optimized text plausibility, not engagement.
Resolution: Add a reward model and rejection sampling gate.
Evaluation
Nextdoor used reward-model predictions, daily monitoring, and A/B tests on sessions, WAU, and ad revenue.
Metrics: Clicks versus baseline · Sessions · Weekly Active Users · Ad revenue · Cost reduction
Outcomes
- Caching reduced costs by 600x.
- Sessions increased 1%.
- Weekly Active Users increased 0.4%.
- Ad revenue increased 1%.
The reported lifts come from A/B testing against user-generated subject lines.