Building Production-Scale AI Agent Evaluation Systems for Customer Care

lyft · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer-care AI agent evaluation at production scale

Lyft needed to scale multiple AI Assist customer-care agents without using riders or drivers as test data, while ensuring offline evals actually predicted production behavior across high-stakes support workflows.

Constraints:

• AI Assist handled rider and driver support, including charge disputes, damage claims, compliance status, taxes, and refund explanations.
• Production scale made live-user experimentation an unacceptable substitute for offline quality gates.
• Generic LLM-judge scores did not provide actionable product or engineering insight.
• Off-the-shelf LLM users behaved too politely compared with real support contacts.
• Offline simulation had to mock MCP/server state while still approximating production trajectories.
• The team needed continuous monitoring and human feedback after production launch.

Architecture

Lyft built an evaluation-first operating model around AI Assist. Production agents run on a LangChain/LangGraph/LangSmith/MCP stack, but the core system-design contribution is the offline and online evaluation loop: a Tau-Bench-inspired simulator pairs the real LangGraph agent with an LLM user, drives permutations from YAML configs for intents, scenarios, personas, and mocked state, evaluates trajectories with task-specific LLM judges and code assertions, captures traces in LangSmith, routes failed traces to annotation queues, and feeds human-labeled failure analysis back into prompts, tools, graph control flow, harness design, and eventually custom model training.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  rider-or-driver[Rider or driver] --> ai-assist-agent[AI Assist agent]
  ai-assist-agent --> langgraph-workflow[LangGraph workflow]
  langgraph-workflow --> mcp-tools[MCP tools]
  langgraph-workflow --> production-monitoring[Production monitoring]
  offline-simulator[Offline simulator] --> langgraph-workflow
  offline-simulator --> llm-user[LLM user]
  yaml-scenario-config[YAML scenario config] --> offline-simulator
  yaml-scenario-config --> mocked-mcp-outputs[Mocked MCP outputs]
  mocked-mcp-outputs --> offline-simulator
  offline-simulator --> evaluator-suite[Evaluator suite]
  evaluator-suite --> llm-judge[LLM judge]
  evaluator-suite --> code-assertions[Code assertions]
  evaluator-suite --> langsmith[LangSmith]
  production-monitoring --> langsmith
  langsmith --> annotation-queue[Annotation queue]
  annotation-queue --> operations-team[Operations team]
  annotation-queue --> engineers[Engineers]
  operations-team --> human-labeled-ground-truth[Human-labeled ground truth]
  human-labeled-ground-truth --> llm-judge
  engineers --> ai-assist-agent
  engineers --> eval-harness[Eval harness]
  eval-harness --> offline-simulator

Components

Data flow

Before launch, a YAML configuration defines support scenarios, user intents, personas, and mocked world state. The offline simulator pairs the actual LangGraph agent with an LLM user, returns mocked MCP outputs for tool calls, and produces full interaction trajectories. Task-specific LLM judges and code assertions evaluate those trajectories. In production, LangSmith traces and online evals capture real interactions; failed traces go to annotation queues; operations and engineering teams label and analyze failures; and the lessons feed back into prompts, tool definitions, graph logic, harness design, and future custom model training.

Why this architecture

Lyft treats agent evaluation like traditional ML engineering: offline evaluation is a launch quality gate, not an afterthought. The simulator prevents production users from becoming test data, the task-specific rubric judges turn failures into actionable product insight, LangSmith connects offline and online traces to human annotation, and custom user simulation is planned because polite generic LLM users made the first benchmark too easy.

Models used: Claude · OpenAI models · LLM-as-judge models · Custom fine-tuned LLM user · LLMs used by AI Assist

Scale

• Lyft reports 79 million trips per month.
  “we have 79 million trips per month”transcript, stated
• AI Assist serves about 270,000 AI interactions per month.
  “our AI Assist chat today has served about 270,000 AI interactions per month.”transcript, stated
• Lyft had built seven or more AI agents in production.
  “We've been able to build seven AI agents and more in production.”transcript, stated
• The reported deflection rate is 65%.
  “Our deflection rate is 65%”transcript, stated
• The reported AI resolution rate is 35%, up from 10%.
  “AI resolution rate is at 35%.”transcript, stated

  “our resolution rate has really gone up from 10% to 35% and climbing.”transcript, stated
• The damage-claim photo agent returns claim decisions in about 15 minutes.
  “when a driver uploads photos of their damage claim, we get back to them in about 15 minutes with a claim decision.”transcript, stated
• The rider-side refund agent processes more than 80 automation rules.
  “an agent that literally processes 80-plus automation rules and refund logic in the back end”transcript, stated
• The first overly polite simulator produced a misleading 90% pass rate.
  “when we did this, our first AI agent was giving us a 90% pass rate.”transcript, stated

Alternatives considered (roads not taken)

Ship agents without offline evaluation

Rejected because: At Lyft's scale, using customers as test data is unacceptable.

Generic helpfulness or tone LLM-judge scores

Rejected because: Scalar generic scores were not actionable for improving agent behavior.

Make live MCP/network calls during offline evaluation

Rejected because: The offline environment should use controlled mocked outputs rather than live source-system calls.

Use off-the-shelf LLMs as nice, cooperative simulated users

Rejected because: They made the eval too easy by explaining issues more clearly than real support users.

Optimize only for support deflection

Rejected because: Lyft held the system to true end-to-end issue resolution rather than simply blocking access to support.

Patterns

eval-driven-development (evaluation)

Lyft uses offline simulation as a quality gate before launch and uses production traces to drive continuous agent improvements.

llm-as-judge (evaluation)

The evaluation suite uses task-specific LLM judges with rubrics instead of generic helpfulness scores.

human-in-the-loop-gate (quality-control)

Failed judge traces are sent to human annotation queues where operations teams label examples and engineers analyze systematic failures.

observability-tracing (operations)

LangSmith captures offline and production trajectories, evaluator results, automation, and annotation workflows.

synthetic-data-generation (evaluation)

The simulator creates synthetic end-to-end user/agent trajectories from configured intents, personas, and state permutations.

data-quality-loop (evaluation)

Human-labeled traces and judge alignment feed back into prompt, tool, control-flow, and model-training improvements.

Tradeoffs

Make offline evaluation a launch gate

Chose Simulator-based offline evaluation before production launch over Learning from live customer failures.

Rationale: At Lyft's scale, the team did not want real customers to be the test set.

Use task-specific judges

Chose Rubric judges for concrete support tasks over Generic helpfulness, tone, and conciseness scores.

Rationale: Task-specific rubrics create actionable failure modes; generic scalar scores did not tell product teams what to improve.

Mock MCP outputs in offline evaluation

Chose Configuration-defined mock world state and tool outputs over Calling live MCP servers during offline eval runs.

Rationale: Offline evaluation needs controlled, repeatable scenarios with state permutations rather than live service dependencies.

Make simulated users harder

Chose Fine-tuned or persona-grounded LLM users based on real verbatims over Out-of-the-box helpful assistant role-play.

Rationale: A harder benchmark may reduce offline scores, but it better predicts production because real users are terse and impatient.

Improve the harness before training custom weights

Chose Prompts, tools, graph, system prompt, RAG pipeline, and eval harness improvements first over Immediately updating LLM weights for AI Assist.

Rationale: The team had been using traces for product and harness improvement and identified model training as a next active area rather than the initial lever.

Gotchas (what bit them)

A high offline pass rate can be fake confidence if simulated users are too cooperative.

Root cause: Off-the-shelf LLM users explained their issues clearly, unlike real users who often send only one or two words.

Resolution: Train or condition LLM users on real Lyft user verbatims and explicit personas, even if the harder benchmark lowers scores.

Generic LLM-judge metrics do not tell teams what to fix.

Root cause: Scalar helpfulness, tone, or naturalness scores lack a task-specific failure interpretation.

Resolution: Write rubrics around agent tasks and success/failure criteria so failed traces reveal common failure modes.

Offline evaluation is only useful if it approximates production behavior.

Root cause: A simulator can generate diverse trajectories, but if the data distribution is unrealistic it will not predict launch quality.

Resolution: Use state/user/persona permutations and production verbatim data to make offline data a better proxy.

LLM judges themselves need evaluation.

Root cause: A judge prompt can drift away from domain-expert expectations even when it appears to score conversations.

Resolution: Treat the LLM judge like an ML model: collect human labels, tune prompts/reasoning, and measure human agreement.

Complex agents cannot be shipped safely by deflection metrics alone.

Root cause: A chatbot can block support without actually resolving the customer's issue.

Resolution: Define success as full end-to-end resolution and use eval gates to protect that standard.

Evaluation

Lyft evaluates AI Assist with a layered system: offline simulation acts as a pre-launch gate, LLM judges and code assertions evaluate generated trajectories, human-labeled domain-expert data calibrates judges, and production LangSmith traces plus human annotation drive online monitoring and iteration.

Metrics: Offline pass rate · Human agreement score for LLM judges · Task-specific rubric pass/fail · Code assertion success/failure · Deflection rate · AI resolution rate · Production LangSmith trace failures · Human annotation labels

Outcomes

• AI Assist serves about 270,000 AI interactions per month.
• Lyft reports seven or more AI agents in production.
• The reported deflection rate is 65%.
• The reported AI resolution rate reached 35%, up from 10%.
• A multimodal damage-claim agent returns decisions in about 15 minutes.
• A rider-side refund agent processes more than 80 automation rules.
• The initial simulator produced a misleading 90% pass rate, leading Lyft to pursue more realistic LLM-user simulation.
• LangSmith annotation queues give operations and engineering teams a feedback loop for systematic failure analysis.

The source is highly quantified: 79 million trips per month, 270,000 AI interactions per month, seven-plus production agents, 65% deflection, 35% resolution, resolution growth from 10% to 35%, 15-minute damage-claim decisions, 80-plus refund rules, and an initial 90% offline pass rate that failed to transfer.