Iterative Development Process for Production AI Features

zapier · Tech · 2024 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI product development process

Zapier needed a repeatable way to take AI feature ideas from prototype to production without either over-investing before validation or shipping unreliable AI broadly. The process had to handle uncertain model capability, early low accuracy, real user feedback, regression risk, evolving datasets, and eventual cost/latency optimization.

Constraints:

• teams need to validate whether an AI feature is feasible before investing deeply
• initial versions can have sub-50% accuracy
• early rollout needs risk controls such as beta labels, internal users, small segments, opt-in use, and suggestions
• manual vibe checks are acceptable early but do not scale to every update
• feedback must include explicit and implicit user signals
• eval test sets must be diverse and reflect real customer usage
• dataset management and versioning are needed as test sets evolve
• cost and latency optimization should be benchmarked against quality before swapping models

Architecture

Zapier's AI development loop starts with rapid prototyping in Braintrust's playground using frontier models such as GPT-4 Turbo and Claude Opus to establish whether a feature is feasible. Teams then ship constrained v1s, sometimes below 50% accuracy, using beta labels, internal or small-segment rollout, opt-in access, and human-in-the-loop suggestion interfaces. After release, they collect explicit and implicit feedback, use Braintrust logs and datasets to turn user examples into eval sets and golden datasets, run evaluations before production updates, iterate until quality improves, then optimize cost and latency by benchmarking cheaper or smaller models against the established eval suite.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  AI_feature_idea["AI feature idea"] --> Braintrust_playground["Braintrust playground"]
  Braintrust_playground --> Frontier_model_prototypes["Frontier model prototypes"]
  Frontier_model_prototypes --> Risk_managed_v1_rollout["Risk-managed v1 rollout"]
  Risk_managed_v1_rollout --> Human_review_suggestions["Human review suggestions"]
  Risk_managed_v1_rollout --> User_feedback_stream["User feedback stream"]
  Human_review_suggestions --> User_feedback_stream
  User_feedback_stream --> Braintrust_logs["Braintrust logs"]
  Braintrust_logs --> Evaluation_datasets["Evaluation datasets"]
  Evaluation_datasets --> Dataset_versioning["Dataset versioning"]
  Evaluation_datasets --> Evaluation_runner["Evaluation runner"]
  Evaluation_runner --> Iterative_product_update["Iterative product update"]
  Iterative_product_update --> Evaluation_runner
  Evaluation_runner --> Ship_decision["Ship decision"]
  Ship_decision --> Risk_managed_v1_rollout
  Ship_decision --> Cost_latency_optimization["Cost latency optimization"]
  Cost_latency_optimization --> Evaluation_runner

Components

Data flow

A feature idea is tested in a playground with high-capability models and prompt/example variations. If feasible, the team builds a v1 and ships it in a limited, risk-managed form. User interactions, explicit feedback, implicit feedback, errors, accepted changes, and follow-up behavior are logged. Interesting logs and corrected failures are added to Braintrust datasets, which become natural-language eval suites and golden datasets. Each product update is evaluated against those datasets; examples that improved or degraded guide the ship/no-ship decision. Once quality is high, the same evals benchmark cheaper or smaller model alternatives for cost and latency optimization.

Why this architecture

The process deliberately separates feasibility discovery, risk-managed rollout, feedback capture, evaluation construction, iterative improvement, and optimization. Frontier models are used first to test whether the idea can work at all; real users then provide diverse inputs; evals convert those examples into a regression guard; and cost/latency work waits until cheaper model swaps can be measured against product accuracy.

Models used: GPT-4 Turbo · Claude Opus · GPT-4o · frontier models · cheaper or smaller models benchmarked later

Scale

• Zapier connects to more than 6,000 apps.
  “directly connecting to over 6,000 apps and supporting developers and non-developers at over 2 million companies.”article, stated
• Zapier supports users at more than 2 million companies.
  “directly connecting to over 6,000 apps and supporting developers and non-developers at over 2 million companies.”article, stated
• AI by Zapier launched with support for 4,000+ app integrations in early 2022.
  “Shipped AI by Zapier with support for 4,000+ app integrations in early 2022”article, stated
• Zapier launched OpenAI on Zapier less than two weeks after ChatGPT was released.
  “Launched OpenAI on Zapier in December of 2022, less than 2 weeks after ChatGPT was released”article, stated
• The process accepts sub-50% accuracy for early v1 releases under mitigations.
  “After validating your idea, you should build and ship v1. Sub 50% accuracy is okay!”article, stated
• The feedback-loop process reportedly improved many AI products from sub-50% accuracy to 90%+ within two to three months.
  “Zapier has improved many of their AI products from sub-50% accuracy to 90%+ within 2-3 months.”article, stated

Alternatives considered (roads not taken)

Start with cheaper or smaller models during initial feasibility testing

Rejected because: Zapier first uses the smartest available models to test the capability ceiling before optimizing cost or latency.

Wait for high accuracy before shipping v1

Rejected because: The methodology ships early to real users because real usage patterns and diverse inputs are foundational for improvement.

Ship updates and hope for user feedback

Rejected because: Once feedback accumulates, the process recommends establishing evals rather than relying on hope or repeated manual vibe checks.

"From here, you have 3 options:

1. Ship the change, hope for the best, and see what customers think
2. Do a manual vibe-checking exercise every time you want to make an update
3. Establish a set of evals"

— article, stated

Optimize cost and latency before evaluation coverage exists

Rejected because: Zapier usually waits until robust evals can quantify how cheaper or smaller models affect product accuracy.

Patterns

eval-driven-development (evaluation)

Zapier turns real user examples, corrected failures, and golden datasets into natural-language eval suites that gate product updates and prevent regressions.

human-in-the-loop-gate (governance)

Early v1 releases mitigate low accuracy by using suggestion-based interfaces, internal releases, beta labels, opt-in features, and manual sanity checks.

"To mitigate the risks of shipping, you can:

• Label the feature as “beta”
• Ship to internal users
• Ship to a very small subsegment of external users
• Make the feature opt-in
• Keep humans in the loop (e.g. suggestions)"

— article, stated

observability-tracing (operations)

Braintrust logs user interactions, feedback, and examples so teams can inspect failures, filter by feedback, and add representative logs to test sets.

canary-rollout (release)

Zapier limits early AI features through internal releases, small external subsegments, beta labels, and opt-in availability before expanding.

"Ship to internal users

• Ship to a very small subsegment of external users
• Make the feature opt-in"

cost-aware-routing (optimization)

After evals exist, Zapier benchmarks cheaper or smaller models against product accuracy before swapping away from frontier models.

Tradeoffs

Prototype with frontier models.

Chose GPT-4 Turbo and Claude Opus for v0 validation over cheaper and faster models during feasibility testing.

Rationale: The team first wants to know whether the strongest models can perform the task before investing or optimizing.

Ship early despite low accuracy.

Chose risk-managed v1 release with real users over waiting for high offline accuracy before any release.

Rationale: Real user interactions provide usage patterns and diverse inputs needed for future improvements.

Weight explicit feedback more heavily.

Chose thumbs, stars, and conscious user feedback as stronger signals over implicit behavior alone.

Rationale: Explicit feedback more directly captures user judgment, while implicit signals such as errors or follow-ups can be ambiguous.

Use evals instead of recurring manual vibe checks.

Chose natural-language eval suites built from real examples over manual vibe-checking every update or shipping and hoping.

Rationale: Evals make overall impact visible and give teams confidence to ship or reject updates.

"From here, you have 3 options:

1. Ship the change, hope for the best, and see what customers think
2. Do a manual vibe-checking exercise every time you want to make an update
3. Establish a set of evals"

Gotchas (what bit them)

Starting with cheap models can hide whether the idea is feasible at all.

Root cause: If a weaker model fails, the team may not know whether the feature is impossible or merely under-modeled.

Resolution: Prototype first with the smartest available models, then optimize after the capability ceiling is understood.

Sub-50% accuracy can be acceptable only if rollout risk is constrained.

Root cause: Early AI products need real user inputs, but broad/default release would expose too many users to poor outputs.

Resolution: Use beta labels, internal release, small external cohorts, opt-in features, and human-in-the-loop suggestions.

• Ship to internal users
• Ship to a very small subsegment of external users
• Make the feature opt-in
• Keep humans in the loop"

— article, stated

Fixing one set of feedback can regress overall performance.

Root cause: Manual prompt/code changes may solve visible failures while degrading other examples.

Resolution: Build eval suites from diverse customer examples before using them to decide whether an update should ship.

Negative feedback is valuable only after it is corrected into a usable test case.

Root cause: A bad output marks a failure but does not by itself define the desired behavior.

Resolution: Correct negative-feedback outputs and include them as regression examples in the test set.

Cost and latency optimization can degrade quality if done before eval coverage exists.

Root cause: Cheaper or smaller models may change behavior, and teams need a benchmark to quantify the accuracy impact.

Resolution: Use robust evals to benchmark model swaps, except when cost or latency constraints force earlier optimization.

Evaluation

The process itself is evaluation-centered: early teams manually sanity-check outputs, then collect explicit and implicit feedback, build diverse customer-derived test sets, version datasets, run evals, inspect improved/degraded examples, and use those results to decide whether product updates or cheaper model substitutions should ship.

Metrics: sub-50% early accuracy threshold · 90%+ accuracy target reported for many products after iteration · 2-3 month improvement window · explicit thumbs up/down or star ratings · implicit errors · accepted changes · follow-up questions · high-level eval performance · examples improved or degraded · cost · latency

Outcomes

• Zapier reportedly improved many AI products from sub-50% accuracy to 90%+ within two to three months using the feedback/eval loop.
• The process allows AI product availability and independent action capability to expand as accuracy increases.
• Braintrust provides logging, feedback tracking, dataset versioning, eval execution, and comparison views for Zapier's process.
• Cost and latency optimization happens after evals make cheaper or smaller model substitutions measurable.
• The case study reports broad Zapier AI adoption, including AI by Zapier, OpenAI on Zapier, ChatGPT plugins, AI chatbots, Zapier Central, and AI Actions.

Zapier connects over 6,000 apps and supports developers and non-developers at over 2 million companies. AI by Zapier supported 4,000+ app integrations in early 2022. The article claims many Zapier AI products improved from sub-50% accuracy to 90%+ within two to three months, but it does not provide detailed methodology for those accuracy measurements.