Building Production AI Agents Platform for Non-Technical Users

zapier · Tech · year n/a · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: No-code business-process automation agents

Zapier needed to run an agent platform for non-technical users who describe business automations, accept proposed tools and triggers, test agents, and then enable them in production. The hard part was not the prototype, but operating probabilistic, tool-using agents against unpredictable user behavior and turning real runs into feedback, evaluations, releases, and product improvements.

Constraints:

• The platform serves non-technical users, so agent creation must start from natural-language intent rather than code.
• Model behavior is nondeterministic and user behavior is even less predictable than model behavior.
• An agent run can include multiple LLM calls, database interactions, tool calls, REST calls, and post-processing steps.
• Tool calls may create side effects, so trajectory evaluation cannot blindly replay production actions.
• Explicit user feedback is rare, so the system must mine implicit signals from behavior and product metrics.
• Fine-grained unit evals can overfit to one model's path and penalize a better model that solves the task differently.

Architecture

Zapier Agents combines a no-code agent builder with a production LLMOps loop. Users describe a desired automation, the product proposes tools and a trigger, and the user tests and enables the agent. Production traces record LLM calls, tool calls, tool errors, and preprocessing/postprocessing in runtime-shaped data so interesting failures can be converted into evals. Explicit feedback, implicit behavior, traditional product metrics, and LLM-classified frustration reports feed failure clustering and roadmapping. Evaluation is layered: unit-test-like evals reproduce specific failures, trajectory evals run agents to completion and grade tool calls/artifacts, LLM-as-judge rubrics provide higher-level scoring, and staged A/B rollouts decide whether model or prompt changes improve real user satisfaction.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  non-technical-user["Non-technical user"] --> zapier-agents-product["Zapier Agents product"]
  zapier-agents-product --> agent-runtime["Agent runtime"]
  agent-runtime --> tool-and-api-connectors["Tool and API connectors"]
  agent-runtime --> trace-instrumentation["Trace instrumentation"]
  non-technical-user --> feedback-collection-layer["Feedback collection layer"]
  zapier-agents-product --> feedback-collection-layer
  trace-instrumentation --> failure-clustering-and-roadmap["Failure clustering and roadmap"]
  feedback-collection-layer --> failure-clustering-and-roadmap
  failure-clustering-and-roadmap --> eval-dataset-builder["Eval dataset builder"]
  eval-dataset-builder --> unit-eval-harness["Unit eval harness"]
  eval-dataset-builder --> trajectory-evaluation-harness["Trajectory evaluation harness"]
  eval-dataset-builder --> llm-judge-and-rubric-scorer["LLM judge and rubric scorer"]
  trajectory-evaluation-harness --> tool-and-api-connectors
  unit-eval-harness --> model-comparison-workflow["Model comparison workflow"]
  trajectory-evaluation-harness --> model-comparison-workflow
  llm-judge-and-rubric-scorer --> model-comparison-workflow
  model-comparison-workflow --> a-b-rollout-monitor["A/B rollout monitor"]
  a-b-rollout-monitor --> zapier-agents-product

Components

Data flow

A non-technical user describes a process in Zapier Agents. The product proposes tools and a trigger, then the user tests the agent and may enable it. Each run emits traces containing completion calls, tool calls, tool errors, and pre/post-processing. Feedback collectors attach explicit feedback at contextual moments and mine implicit signals such as enabling an agent, copying an answer, stop messages, rephrased follow-ups, churn-adjacent sessions, and frustration reports. Interesting runs and clustered failure modes become unit evals, trajectory evals, or rubric-based judge rows. Candidate model/prompt changes are compared in these evals and finally shipped through staged A/B traffic while activation, retention, feedback, and other product metrics are monitored.

Why this architecture

Zapier's architecture treats the shipped product as the start of the engineering loop, because real user behavior reveals failures that prototypes and static benchmarks miss. Runtime-shaped tracing lowers the cost of converting production failures into evals. Multiple eval layers avoid overfitting to either tiny assertions or expensive end-to-end tests, while A/B rollouts keep the final objective anchored in user satisfaction rather than lab metrics.

Models used: LLM agent model · Reasoning LLM · LLM-as-judge

Scale

• Zapier frames the agent product as a two-year production learning effort.
  “a key lesson that we have after those those two years is that um building good AI agents is hard”transcript, stated
• A/B validation is described as routing a small slice such as 5% of traffic to a candidate model or prompt.
  “Just take a small proportion of your portion of your traffic, let's say 5% and route it to the new model”transcript, stated
• The aspirational eval set includes extremely long tasks such as 200 consecutive tool calls.
  “the aspirational data set of things that are extremely hard. For instance, like nailing 200 tool calls uh in a row.”transcript, stated
• Trajectory evals may take up to an hour, creating a practical cost/speed boundary.
  “also they are much slower right so they can sometimes take like up to an hour.”transcript, stated
• The release process monitors activation and retention metrics after staged rollout.
  “monitor the feedback, check your metrics like activation, user retention and so on.”transcript, stated

Alternatives considered (roads not taken)

Ship the LangChain-style prototype directly

Rejected because: The team argues that prototypes hide the operational detail of probabilistic software and do not create the feedback/eval flywheel needed after users arrive.

Trace only completion calls

Rejected because: Completion-only traces are insufficient for debugging multi-call, tool-using runs; Zapier needed tool calls, tool errors, and processing steps too.

Rely on explicit thumbs-up/thumbs-down feedback

Rejected because: Explicit feedback is high-signal but rare, so Zapier added contextual feedback prompts and implicit feedback mining.

Use only fine-grained unit evals

Rejected because: Unit evals helped hill-climb specific failures but over-indexing on them made stronger models look worse and obscured model-level behavior.

Mock trajectory-eval environments

Rejected because: Mocking would produce unrealistic data for side-effecting tool runs, so Zapier mirrored user environments with synthetic copies instead.

Optimize for eval scores as the primary goal

Rejected because: The team treats lab metrics as proxies and uses user satisfaction plus A/B rollouts as the ultimate arbiter.

Patterns

observability-tracing (operations)

Zapier records tool calls, tool errors, and processing steps in addition to completion calls, so production runs can be debugged and replayed as eval inputs.

data-quality-loop (feedback)

The product combines explicit feedback, implicit behavioral signals, frustration clustering, and product metrics to create a data flywheel for prioritizing agent improvements.

eval-driven-development (evaluation)

Zapier turns interesting runs and failure modes into evals, then uses a testing-pyramid hierarchy of unit, trajectory, rubric, and rollout evaluations to guide development.

llm-as-judge (evaluation)

Reasoning models and rubric-based LLM judges explain failures, compare model runs, and score trajectories with per-row human-authored criteria.

canary-rollout (deployment)

Zapier validates new model or prompt changes by routing a small percentage of production traffic and monitoring satisfaction, feedback, activation, and retention.

tool-use-orchestration (architecture)

The agent runtime coordinates LLM calls with tool calls, REST calls, database interactions, and generated artifacts, requiring evaluation that can inspect the whole trajectory.

Tradeoffs

Make tracing broad enough for agent debugging

Chose Full run traces with tool calls, errors, and processing steps over Only tracing model completion calls.

Rationale: Completion-only tracing was easy to start with, but production agent failures require the whole story and runtime-shaped data to become repeatable evals.

Combine bought and custom LLMOps tooling

Chose Buy/build hybrid tooling with custom internal workflows over Relying entirely on purchased LLMOps software.

Rationale: Zapier needed domain-specific understanding and one-click conversion from failures to evals; custom tooling became cheaper with AI-assisted coding.

Use contextual and implicit feedback

Chose Feedback prompts at critical moments plus implicit behavioral signals over Static thumbs-up/thumbs-down controls alone.

Rationale: Generic controls produced limited explicit feedback, while contextual prompts and behavioral signals gave more usable signal without waiting for users to write detailed reports.

Layer eval types instead of relying on one evaluation mode

Chose Unit evals, trajectory evals, LLM-as-judge rubrics, and A/B tests over A single benchmark or eval style.

Rationale: Each eval layer addresses a different need: unit evals hill-climb specific failures, trajectory evals capture multi-turn behavior, judge/rubric scores summarize capabilities, and A/B tests capture user impact.

Mirror user environments for trajectory evals

Chose Synthetic copies of user environments over Fully mocked environments.

Rationale: They wanted eval data to reflect reality while still avoiding repeated side effects against customer systems.

Split eval datasets into regression and aspirational pools

Chose Regression cases plus hard aspirational cases over One eval set optimized toward high average scores.

Rationale: Regression cases protect existing customer use cases, while aspirational cases keep the dataset challenging and prevent 100% scores from becoming false comfort.

Gotchas (what bit them)

A working prototype does not represent production readiness.

Root cause: Probabilistic agent behavior and unpredictable user behavior create failure modes only visible after real usage.

Resolution: Treat launch as the beginning of a data flywheel, then instrument runs, collect feedback, convert failures into evals, and iterate.

Completion-call traces are too shallow for debugging agent failures.

Root cause: A single agent run can fail in tool calls, tool errors, preprocessing, postprocessing, database interactions, or REST calls.

Resolution: Record runtime-shaped traces that include the full run story and can be converted into eval inputs.

Explicit feedback is valuable but too rare to drive the loop alone.

Root cause: Users rarely click generic thumbs-up/thumbs-down controls and often do not provide detailed actionable feedback.

Resolution: Ask for feedback at high-context moments and mine implicit behavior such as enabling agents, copying responses, stop messages, rephrasing, frustration, churn, and business metrics.

LLM frustration classifiers require domain tuning.

Root cause: A generic model does not automatically understand what frustration means in Zapier's product context.

Resolution: Tune and review the classifier until frustration grouping aligns with product semantics before using it for weekly reporting.

Unit evals can overfit to the current model's path.

Root cause: Fine-grained next-state assertions penalize different but valid solution paths from newer models.

Resolution: Use trajectory evals, reasoning-model comparisons, and A/B tests to complement unit evals.

Trajectory evals can accidentally trigger real side effects.

Root cause: Agent runs may send emails or perform customer actions through tools during replay.

Resolution: Use synthetic copies of user environments rather than replaying against production side-effect surfaces.

LLM judges can introduce subtle scoring bias.

Root cause: Judge prompts and rubrics can be influenced by overlooked details and may not judge the intended property.

Resolution: Use handcrafted rubrics per row and verify that judges are evaluating the intended behavior.

Eval scores can become targets instead of useful signals.

Root cause: Over-optimizing a lab metric can hide an unchallenging dataset or drift away from user satisfaction.

Resolution: Maintain regression and aspirational datasets, watch for 100% scores as a warning, and use user-facing rollout metrics as the final decision point.

Evaluation

Zapier evaluates agents with a hierarchy: unit-test-like next-state evals for specific failures, trajectory evals that run agents to completion and grade tool calls/artifacts, LLM-as-judge and rubric-based scoring for higher-level model comparison, regression and aspirational datasets for coverage, and staged A/B tests as the final production validation.

Metrics: Next-state assertions such as chosen tool call, correct tool parameters, answer keywords, and done-state detection. · Trajectory-level tool calls, artifacts, and end states. · Rubric-based LLM judge scores for per-row behavior. · Regression dataset pass/fail against existing customer use cases. · Aspirational dataset performance on very hard cases such as 200 tool calls in a row. · A/B rollout metrics including explicit feedback, activation, and user retention.

Outcomes

• Contextual post-run feedback requests produced a notable increase in feedback submissions.
• LLM frustration detection and grouping produced weekly reports for the team after tuning.
• Reasoning-model comparison helped explain model differences, including Claude being more decisive while Gemini asked more follow-up questions and sometimes produced malformed JSON.
• Zapier shifted from over-indexed unit evals toward trajectory evals and A/B rollout validation.
• The team established a regression/aspirational dataset split to protect existing use cases while keeping hard cases visible.

The transcript does not disclose Zapier Agents user counts or production request volume. It does give operational scale markers: two years of production learning, staged A/B traffic around 5%, trajectory evals sometimes taking up to an hour, and aspirational evals such as 200 tool calls in a row.