LLM-as-Judge Framework for Production LLM Evaluation and Improvement

segment · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer data platform audience generation

Twilio Segment needed a way to evaluate natural-language audience generation where outputs are ASTs with many semantically valid forms, making exact-match evaluation unusable for production iteration.

Constraints:

• Audience definitions compile into ASTs where multiple syntactic structures can express the same customer segment.
• The team had real ASTs from the UI but not the corresponding natural-language prompts needed for evaluation pairs.
• Evaluation had to guide model selection, prompt changes, RAG or memory additions, and single-stage versus multi-stage architecture choices.
• The judge needed enough alignment with human evaluation to be trusted as a development scoreboard.

Architecture

Segment built an LLM-as-judge evaluation pipeline around real customer ASTs. A question-generator agent reverse-generates synthetic prompts from ground-truth ASTs, an AST-generator agent produces candidate ASTs from those prompts, and a judge agent scores the generated AST against the original AST. The resulting scores become the evaluation scoreboard for model, prompt, memory, and architecture changes in CustomerAI audiences.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Audience_Builder_UI["Audience Builder UI"] --> Real_World_AST_Input["Real World AST Input"]
  Real_World_AST_Input --> LLM_Question_Generator_Agent["LLM Question Generator Agent"]
  LLM_Question_Generator_Agent --> User_Prompt["User Prompt"]
  User_Prompt --> LLM_AST_Generator_Agent["LLM AST Generator Agent"]
  LLM_AST_Generator_Agent --> Generated_AST["Generated AST"]
  Generated_AST --> LLM_Judge_Agent["LLM Judge Agent"]
  Real_World_AST_Input --> LLM_Judge_Agent
  LLM_Judge_Agent --> Evaluation_scoreboard["Evaluation scoreboard"]

Components

Data flow

Customer-created ASTs from the Audience Builder become ground truth. The LLM question generator derives likely natural-language prompts from those ASTs, the AST generator produces new ASTs from the synthetic prompts, and the judge compares generated ASTs with real-world ASTs. Engineers use the judge score and chain-of-thought reasoning to compare model families, prompt variants, context-window sizes, and future RAG or memory additions.

Why this architecture

The architecture turns ambiguous semantic-equivalence evaluation into a repeatable scoreboard. Synthetic prompt generation solves the missing-input problem, and a judge model handles cases where strict heuristics would reject valid AST variants.

Models used: LLM question generator · LLM AST generator · LLM judge · GPT-4 · gpt-4-32k-0613 · Claude 3 Opus

Scale

• CustomerAI audiences produced a 3x median improvement in time-to-audience creation.
  “customers are experiencing a 3x improvement in median time-to-audience creation”article, stated
• Feature retention reached 95% when the generated audience worked on the first attempt.
  “a whopping 95% feature retention rate IF the audience generation works on the first attempt.”article, stated
• The judge system achieved over 90% alignment with human evaluation.
  “Our LLM Judge Evaluation system has achieved over 90% alignment with human evaluation for ASTs.”article, stated
• Chain-of-thought judging improved human alignment from roughly 89% to 92%.
  “By using CoT, the alignment with human increase from roughly 89% to 92%.”article, stated

Alternatives considered (roads not taken)

Exact-match or rule-based AST evaluation

Rejected because: Multiple ASTs can represent the same audience meaning, so strict matching would penalize valid generations.

Continuous 0-100 judge scores

Rejected because: The team observed that LLMs tended to collapse continuous scores into extremes, so they used a discrete 1-5 scale.

Evaluate only a single model or context-window setup

Rejected because: The scoreboard was used to compare Claude, GPT-4 variants, and context-window sizes.

Patterns

llm-as-judge (evaluation)

A separate judge model scored generated ASTs against ground-truth ASTs and produced reasoning used by engineers to compare models and architectures.

synthetic-data-generation (data)

Segment generated synthetic prompts from real ASTs to create evaluation examples even though the original UI-generated ASTs had no prompt text.

eval-driven-development (evaluation)

The judge score served as the scoreboard for model selection, prompt optimization, persistent memory/RAG experiments, and multi-stage architecture decisions.

Tradeoffs

Use an LLM judge rather than strict heuristics for AST quality.

Chose LLM-as-judge semantic scoring over Exact-match or rule-based comparison.

Rationale: The product needed to accept semantically equivalent ASTs whose syntax differed from the ground truth.

Generate synthetic prompts from ASTs rather than wait for natural prompt logs.

Chose Reverse prompt generation from ground-truth ASTs over Using only naturally collected prompt-AST pairs.

Rationale: The available production data contained ASTs from UI interactions, so synthetic prompt generation made evaluation possible immediately.

Use discrete scoring with reasoning rather than raw continuous scores.

Chose 1-5 judge score plus chain-of-thought over 0-100 continuous scoring without reasoning.

Rationale: Discrete scoring avoided collapsed continuous outputs, while reasoning made judgments easier for engineers to inspect.

Gotchas (what bit them)

Semantic equivalence makes ordinary automated scoring misleading.

Root cause: The AST representation allows many structures to express identical audience logic.

Resolution: Use an LLM judge to compare meaning against ground truth rather than relying on exact syntax.

The evaluation dataset had outputs but lacked the natural-language inputs.

Root cause: Ground-truth ASTs came from the existing UI, not prompt-based generation.

Resolution: Reverse-generate prompts from ground-truth ASTs with a question-generator agent.

Judge scoring needed calibration before engineers could trust it.

Root cause: Continuous scoring and unexplained judgments were not reliable enough for human comparison.

Resolution: Use discrete 1-5 scores and chain-of-thought reasoning to improve interpretability and human alignment.

Evaluation

Segment compared judge scores with human evaluation, experimented across model families and context sizes, and used the resulting scores as baselines for future optimization.

Metrics: over 90% alignment with human evaluation · gpt-4-32k-0613 score of 4.55 out of 5.0 · Claude model score of 4.02 · roughly 89% to 92% alignment improvement from CoT

Outcomes

• The system achieved more than 90% alignment with human AST evaluation.
• The framework enabled Segment to compare models, prompts, memory/RAG additions, and single-stage versus multi-stage architectures.
• CustomerAI audiences reached 3x faster median audience creation and 95% feature retention when first-attempt generation succeeded.

The article describes thousands of marketers using Segment and a large AST dataset from the existing UI, but does not provide exact dataset row counts.