Building Evaluation Frameworks for AI Product Managers: A Workshop on Production LLM Testing

arize · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI product management, LLM evaluation, and production observability

Arize's workshop addresses how product managers can specify, evaluate, and iterate on LLM and agentic products when traditional deterministic testing is insufficient. The central problem is that LLM agents hallucinate, take multiple paths through tool and agent workflows, depend on product data, and require cross-functional confidence before production release. The case demonstrates a workflow that turns traces from a multi-agent trip-planner demo into prompt experiments, datasets, LLM-as-judge evaluations, human labels, and iterative eval improvement.

Constraints:

• LLM agents are non-deterministic, so unit-test assumptions do not transfer cleanly.
• A production AI product must evaluate the whole system, including prompts, data, tools, and subcomponents, not only final text.
• Product managers need enough prompt, dataset, and evaluation control to express product requirements without relying only on engineering handoff.
• LLM-as-judge evaluators themselves can be wrong, so they need human labels and repeated calibration.
• The demonstrated workflow is a workshop and product demo, so reported outcomes are educational and workflow-level rather than deployed customer impact.

Architecture

The workshop uses a multi-agent AI trip planner as the running system under evaluation. A form captures travel parameters, then parallel budget, local-experience, and research agents feed an itinerary agent. Arize/Phoenix instrumentation captures traces and spans from that agent graph, including agent, tool, and LLM spans. A prompt playground pulls a traced prompt plus its variables into an editable UI, where product managers can change prompts, models, and inputs. Selected traces become tabular datasets; experiments run prompt variants across the dataset; LLM-as-judge evaluators classify outputs with text labels; human annotation supplies ground truth; agreement checks expose weak judges; and the resulting evals can become acceptance criteria for engineering.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ai-product-manager[AI product manager] --> prompt-playground[Prompt playground]
  ai-product-manager --> dataset-builder[Dataset builder]
  ai-product-manager --> human-labeling-queue[Human labeling queue]
  ai-trip-planner-form[AI trip planner form] --> multi-agent-trip-planner[Multi-agent trip planner]
  multi-agent-trip-planner --> research-agent[Research agent]
  multi-agent-trip-planner --> budget-agent[Budget agent]
  multi-agent-trip-planner --> local-experiences-agent[Local experiences agent]
  multi-agent-trip-planner --> tools-and-rag-layer[Tools and RAG layer]
  research-agent --> itinerary-agent[Itinerary agent]
  budget-agent --> itinerary-agent
  local-experiences-agent --> itinerary-agent
  tools-and-rag-layer --> trace-span-store[Trace/span store]
  itinerary-agent --> trace-span-store
  arize-phoenix-tracing[Arize/Phoenix tracing] --> trace-span-store
  arize-phoenix-tracing --> agent-visualization[Agent visualization]
  trace-span-store --> agent-visualization
  trace-span-store --> prompt-playground
  trace-span-store --> dataset-builder
  prompt-playground --> prompt-and-model-variants[Prompt and model variants]
  prompt-and-model-variants --> experiment-runner[Experiment runner]
  dataset-builder --> experiment-runner
  experiment-runner --> llm-as-judge-evaluator[LLM-as-judge evaluator]
  llm-as-judge-evaluator --> eval-agreement-checker[Eval-agreement checker]
  human-labeling-queue --> eval-agreement-checker
  eval-agreement-checker --> prompt-playground
  eval-agreement-checker --> dataset-builder
  ai-product-manager --> engineering-team[Engineering team]

Components

Data flow

A product manager starts from a trip-planner form request, which becomes a multi-agent LangGraph/Crew-style workflow. Research, budget, and local-experience agents run in parallel and feed an itinerary agent. Arize/Phoenix tracing records input, output, metadata, traces, and spans, then visualizes the agent graph. The PM pulls a traced prompt and its variables into a prompt playground, creates prompt or model variants, adds traced examples into a dataset, and runs experiments across that dataset. LLM-as-judge evaluators label outputs such as tone, discount inclusion, hallucination, or correctness. Human labels are then compared with judge labels, and disagreement drives prompt/eval iteration before those evals are used as acceptance criteria.

Why this architecture

The architecture separates observability, prompt iteration, dataset construction, automated judging, and human validation because single-example prompt tweaking does not scale to production. Traces make the actual agent graph inspectable, the prompt playground gives PMs a controlled way to change product behavior, datasets turn individual examples into repeatable experiments, and human labels keep the judge from becoming an unverified authority.

Models used: LLMs · LLM-as-judge · GPT-4.1 mini · GPT-4o · LangGraph · Crew AI · OpenTelemetry · Arize Phoenix · Cursor · BERT or ALBERT custom endpoints

Scale

• The workshop demo used a seven-day Tokyo trip with a $1,000 budget.
  “Maybe we want to do Tokyo for seven days. And assuming the internet works, um, we'll see if it does. We're going to put a budget of $1,000.”transcript, stated
• The demo prompt was modified to produce outputs of 500 characters or less.
  “Keep it to 500 characters or less.”transcript, stated
• The demo added a 10% discount-code instruction.
  “Email me to get a 10% discount code.”transcript, stated
• The synthetic demo dataset was bootstrapped with 15 itineraries.
  “I literally just asked Cursor to make me like synthetic data. Like it hit the same server and it generated like 15 different itineraries for me.”transcript, stated
• The dataset workflow is presented as scaling from 10 examples toward 100 examples.
  “build out a data set with 10 or more, maybe even a hundred examples”transcript, stated
• A live experiment ran on 12 examples instead of a single prompt playground row.
  “now instead of running on one example I'm actually running on 12 examples here.”transcript, stated
• One experiment took about 32 seconds on average.
  “This one is taking an average of like 32 seconds”transcript, stated
• The changed prompt caused 100% of examples in that experiment to include a discount offer.
  “100% of our examples when the when we made this change actually have an offer of a discount.”transcript, stated

Alternatives considered (roads not taken)

Use traditional deterministic software tests as the main confidence mechanism.

Rejected because: The speaker argues that LLM agents are non-deterministic, can take multiple paths, and rely on data, so deterministic unit-test assumptions are not enough.

Ask an LLM judge for a numeric score directly.

Rejected because: The recommended design is to ask for a text label and map that to a score, because LLMs are unreliable at numeric scoring.

Judge prompt changes from one example in a playground.

Rejected because: The speaker explicitly calls the one-row demo vibe coding and moves to datasets and experiments because production workloads require many examples.

Trust LLM-as-judge labels without human calibration.

Rejected because: The workshop shows LLM judges need human labels, agreement checks, and iteration because a judge can disagree with human judgment.

Keep PMs limited to PRDs while engineers own prompts and eval implementation.

Rejected because: The speaker frames eval datasets and evals themselves as new requirement artifacts and argues PMs need control over prompts because they own the product outcome.

Patterns

eval-driven-development (evaluation)

The workshop reframes product requirements as eval datasets and acceptance criteria: PMs build datasets, run experiments, calibrate evals, and hand engineering concrete evaluation artifacts rather than only prose PRDs.

llm-as-judge (evaluation)

Arize demonstrates judge prompts with role, context, goal, terminology, and text labels, then runs them across datasets to classify outputs such as friendly/robotic, discount/no-discount, hallucination, and correctness.

observability-tracing (operations)

The demo instruments the agent workflow so product and engineering teams can inspect traces, spans, metadata, agent/tool/LLM steps, and a visual graph of the agent system.

multi-agent-orchestration (orchestration)

The trip-planner demo uses separate budget, local-experience, and research agents in parallel, then routes their results into an itinerary agent that synthesizes the final plan.

prompt-versioning (prompt-operations)

The workflow saves prompt variants, duplicates prompts, changes one variable at a time for experiments, and uses prompt hub/versioning concepts so prompts can move from playground iteration into code.

human-in-the-loop-gate (governance)

Human and subject-matter-expert labels are used to validate LLM judge outputs, identify disagreement, and produce hard examples for further evaluation work.

data-quality-loop (data)

Production traces and synthetic examples are curated into datasets, used for experiments, augmented with human labels, and continuously updated with hard examples discovered in production.

tool-use-orchestration (orchestration)

The agentic demo explicitly includes tool-calling and possible RAG/retrieval under the hood, and the tracing layer distinguishes tool spans from agent and LLM spans.

Tradeoffs

Evaluation output format

Chose text labels mapped to scores over direct numeric scores from the LLM judge.

Rationale: Text labels are more reliable because the speaker says LLMs remain weak at numbers, while labels can still be mapped to scores downstream.

Prompt iteration method

Chose dataset-backed experiments over one-off vibe checks in a prompt playground.

Rationale: The demo starts with a single example for iteration but explicitly moves to datasets because production confidence needs many examples.

Automation level for judging

Chose LLM-as-judge with human-label calibration over manual row-by-row grading or unvalidated judge labels.

Rationale: The speaker positions LLM-as-judge as scalable, but still requires human labels to validate the judge and expose disagreement.

System structure for the demo app

Chose a multi-agent workflow with traces and specialized spans over a simple chatbot or opaque prompt-only app.

Rationale: The workshop deliberately moves beyond chatbot demos to show form inputs, parallel specialized agents, tools/RAG, and trace-based visualization of what the agent actually did.

Instrumentation standard

Chose OpenTelemetry/OpenInference-based tracing over proprietary trace capture that locks teams into one vendor.

Rationale: The Q&A emphasizes that Arize builds on open tracing conventions so teams can keep logs and avoid vendor lock-in.

Dataset evolution strategy

Chose continuous production sampling and hard-example curation over a fixed once-created evaluation dataset.

Rationale: The speaker says production data shifts, so the dataset should keep accumulating examples where human and judge labels disagree or where confidence is low.

Gotchas (what bit them)

Numeric LLM judge scores are unreliable.

Root cause: The speaker says even strong LLMs are bad at numbers because of how tokens are represented.

Resolution: Ask the judge for text labels such as friendly/robotic or toxic/not-toxic, then map those labels to scores programmatically if a score is needed.

A single prompt-playground example can create false confidence.

Root cause: One example is vibe coding; it cannot represent the range of requests from hundreds or thousands of users.

Resolution: Move the trace into a dataset and run experiments across many examples before deciding whether a prompt/model change is good.

LLM judges need their own validation loop.

Root cause: A judge is itself an agent and can disagree with human judgment, especially on subjective criteria such as friendliness.

Resolution: Collect human labels, compare judge labels with human labels, add few-shot examples or stricter criteria, and iterate until judge outputs align better.

Evaluating only the final LLM text misses system-level behavior.

Root cause: Agentic applications include routers, skills, function calls, memory, tools, and subcomponents that affect output quality.

Resolution: Instrument traces/spans and evaluate the whole system or relevant spans and subcomponents, not just the standalone prompt.

Dataset representativeness drifts over time.

Root cause: Production data changes, so a static dataset may stop representing the actual request distribution.

Resolution: Run evals on production data, sample at larger scale, add hard examples, and keep curating the dataset as new examples appear.

PM/engineering ownership of prompts and evals becomes ambiguous.

Root cause: Prompts directly shape product experience, but historically engineers owned implementation while PMs wrote PRDs.

Resolution: Give PMs controlled prompt and eval workflows, and treat eval datasets plus acceptance criteria as a requirement artifact shared with engineering.

Judge variance can make evaluation results unstable.

Root cause: LLM judges are probabilistic and can vary their labels across runs.

Resolution: Lower temperature where appropriate, rerun evals multiple times, and profile judge variance rather than assuming one run is definitive.

Evaluation

The workshop's evaluation approach combines prompt/model experiments, LLM-as-judge classification, human labels, and meta-evaluation of the judge. Evals are constructed from role, task, context, goal, terminology, and text labels; run across datasets rather than single examples; and checked against human annotations to identify disagreement and improve the judge prompt.

Metrics: Friendly vs robotic classification. · Discount offer vs no discount classification. · Hallucination/use-of-context evaluation. · Correctness evaluation. · Agreement or disagreement between LLM judge labels and human labels. · Judge variance across repeated runs and temperature settings. · Experiment latency, including an example averaging about 32 seconds.

Outcomes

• The demo made the trip-planner agent graph inspectable through traces, spans, and an agent visualization.
• The prompt playground let the PM alter variables, model choice, output length, tone, and discount behavior without changing application code directly.
• A dataset and experiment workflow moved evaluation from one example to multiple examples.
• The workshop demonstrated that a friendliness judge could disagree substantially with human labels, motivating eval-prompt iteration.
• The closing recommendation was to turn evals and datasets into acceptance criteria for engineering rather than relying only on PRDs.

This is a workshop and product demonstration, not a case study reporting deployed production impact. Its concrete scale is mostly demo-level: 15 synthetic itineraries, 12-example experiments, hundreds/thousands of examples as the motivating target, and a product workflow that Arize positions for customers such as Uber, Reddit, and Instacart.