Self-Improving Agentic Systems Using DSPy for Production Email Generation

relevance_ai · Tech · 2025 · source Agentic: yes · promotional bias: high · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: outbound sales email generation

Relevance AI wanted production email-generation agents that improve from real feedback instead of relying on static manually tuned prompts.

Constraints:

• The original article source was unavailable, so details are extracted from the ZenML summary.
• Sales emails require brand voice, quality control, and human oversight before delivery.
• Continuous learning depends on collecting high-quality approved examples.
• Optimization has to be separated from fast inference so production response time stays practical.

Architecture

The described system integrates DSPy optimization with a human-in-the-loop approval workflow: agents draft outbound sales emails after CRM data gathering and prospect research, pause for human approval at critical checkpoints, store approved or corrected outputs as training data, optimize DSPy programs with the accumulated examples, cache optimized programs, and use them for future inference.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["CRM and prospect research context"] --> B["Email generation agent"]
  B --> C["Human approval checkpoint"]
  C --> D["Training example collector"]
  D --> E["DSPy optimization pipeline"]
  E --> F["Cached DSPy programs"]
  F --> B
  E --> G["Evaluation framework"]

Components

Data flow

CRM and prospect-research context enters the email-generation agent; the agent drafts an email; an approval checkpoint lets a human approve or correct the output; the feedback is stored as training data by Prompt Optimizer tooling; DSPy optimizers select examples and/or prompt instructions; optimized programs are cached in the knowledge base; future email generations use the cached program at inference time.

Why this architecture

The architecture turns required human approval from a pure bottleneck into a learning signal: approved and corrected outputs become training examples, DSPy optimizes programs offline, and cached optimized programs keep inference fast.

Models used: DSPy-optimized LLM programs · LLM-based semanticF1 evaluator

Scale

• The reported system matched human-written email quality 80% of the time and exceeded it in 6% of cases.
  “generated emails matching human-written quality 80% of the time, with 6% of cases exceeding human performance”summary, stated
• The system reportedly reduced production agent building time by 50%.
  “cutting production agent building time by 50%.”summary, stated
• Response times are reported at one to two seconds under heavy load.
  “maintaining the reported one to two second response times.”summary, stated

Alternatives considered (roads not taken)

Manual prompt tuning

Rejected because: The case study frames DSPy optimization as reducing production agent build time by eliminating manual prompt tuning.

Static agent prompts without feedback

Rejected because: The system uses human-approved examples as a continuous training signal for future optimization.

One optimizer for all data sizes

Rejected because: The summary describes different DSPy optimizers for fewer than 20, around 50, and 200+ samples.

Patterns

human-in-the-loop-gate (quality-control)

Humans approve or correct generated emails at checkpoints, and those corrections become training examples.

eval-driven-development (optimization)

DSPy optimization and semanticF1 comparisons guide program improvement against control agents.

Tradeoffs

Use DSPy optimization.

Chose learned prompts/programs from approved examples over manual prompt tuning.

Rationale: Optimization reduced build time and improved the system through real feedback.

Keep human approval in the loop.

Chose approval checkpoints before email delivery over fully autonomous email sending.

Rationale: Human oversight preserves quality and creates the training signal for improvement.

Gotchas (what bit them)

Training data quality dominates the optimization pipeline.

Root cause: Bad or inconsistent approved examples propagate into DSPy programs.

Resolution: Invest in refined gold sets and human approval quality before relying on optimization.

The reported performance metrics lack full methodology detail.

Root cause: The summary notes ambiguity around how matching or exceeding human quality was operationalized.

Resolution: Treat the performance claims as promotional and require clearer evaluation documentation before copying the metric design.

Human approval can become a throughput bottleneck.

Root cause: The self-improvement loop depends on ongoing human review and correction quality.

Resolution: Use approval checkpoints for quality-sensitive outputs while monitoring review volume and consistency.

Evaluation

The system reportedly compared DSPy agents to non-DSPy control agents using identical workflows and semanticF1, while also reporting human-quality match/exceed rates.

Metrics: semanticF1 · human-written quality match rate · human performance exceed rate · agent development time reduction · response time

Outcomes

• Reported emails matched human-written quality 80% of the time and exceeded it in 6% of cases.
• Reported production-agent build time fell 50%.
• The system continuously collected approved examples for DSPy optimization.

The source is unavailable, so these outcomes are summary-backed and should be discounted relative to source-backed cases.