AI-Powered Code Generation for Support Team Bug Fixing

zapier · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Support-led integration bug fixing

Zapier faced app erosion across thousands of third-party integrations, with API changes and deprecations creating bugs faster than engineering could resolve them.

Constraints:

• over 8,000 integrations depend on changing third-party APIs
• tickets were arriving faster than teams could handle
• support staff needed guardrails to ship code safely
• context gathering across docs and the internet was slow
• generated fixes still required support testing and engineering review

Architecture

Zapier Scout Agent turns support-reported integration bugs into code-fix merge requests: it categorizes the issue, assesses fixability, gathers context through MCP tools, generates code, loops on support feedback, and sends the MR to engineering review.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support-issue-intake["Support issue intake"]
  categorization-step["Categorization step"]
  fixability-assessment["Fixability assessment"]
  scout-agent["Scout Agent"]
  scout-mcp-tools["Scout MCP tools"]
  code-generation-tool["Code generation tool"]
  gitlab-merge-request["GitLab merge request"]
  support-review["Support review"]
  engineering-review["Engineering review"]
  support-issue-intake --> categorization-step
  categorization-step --> fixability-assessment
  fixability-assessment --> scout-agent
  scout-agent --> scout-mcp-tools
  scout-agent --> code-generation-tool
  code-generation-tool --> gitlab-merge-request
  scout-agent --> gitlab-merge-request
  gitlab-merge-request --> support-review
  support-review --> scout-agent
  support-review --> engineering-review
  engineering-review --> support-issue-intake

Components

Data flow

Support issues flow through categorization and fixability assessment into Scout Agent, which orchestrates diagnosis tools, code generation, GitLab MRs, support testing, and engineering review.

Why this architecture

The architecture targets app erosion from thousands of third-party APIs, while guardrails keep support-led fixes narrow and reviewable.

Models used: code generation model · LLM-backed diagnosis tools

Scale

• Zapier describes more than 8,000 integrations and early support output moving from no official fixes to several fixes per person per week with Scout.
  “At Zapier, we have over 8,000 integrations built on thirdparty APIs, and they are constantly changing, which I'm now thinking of as app erosion.”transcript, stated

  “That's going from a support team that wasn't shipping any fixes, well unofficially they were sometimes to now shipping one to two per week per person to now shipping three to four with the help of Scout.”transcript, stated
• Zapier maintains more than 8,000 integrations affected by third-party API changes.
  “At Zapier, we have over 8,000 integrations built on thirdparty APIs, and they are constantly changing, which I'm now thinking of as app erosion.”transcript, stated
• Categorization and fixability evals reached 75% accuracy.
  “so far we have two eval to 75% accuracy for categorization and fixibility.”transcript, stated
• Forty percent of support-team app fixes were being generated by Scout.
  “40% of supports support teams app fixes are being generated by scout.”transcript, stated
• Some support engineers moved from one to two fixes weekly to three to four with Scout.
  “That's going from a support team that wasn't shipping any fixes, well unofficially they were sometimes to now shipping one to two per week per person to now shipping three to four with the help of Scout.”transcript, stated

Alternatives considered (roads not taken)

Standalone diagnosis API outside the workflow

Rejected because: The team embedded diagnosis and code generation in Scout Agent because a separate window saw low engagement.

Open-ended support code generation across all apps

Rejected because: The first rollout focused on four target apps with engineering review for every merge request.

Diagnose synchronously inside an IDE loop

Rejected because: The slow but valuable diagnosis tool fits better inside an asynchronous agent workflow.

Engineering team as the first customer

Rejected because: Zapier started with support because support sees small emergent app bugs first.

Patterns

tool-use-orchestration (orchestration)

Scout Agent coordinates diagnosis, categorization, fixability assessment, code generation, MR creation, and feedback iteration.

human-in-the-loop-gate (safety)

Support reviews and tests generated merge requests, then engineering reviews them before production code is accepted.

confidence-thresholding (evaluation)

The agent explicitly assesses whether an issue is fixable before attempting to generate a merge request.

eval-driven-development (evaluation)

Zapier turns processed tickets and feedback into test cases to improve Scout Agent over time.

Tradeoffs

Who should be the first Scout Agent customer.

Chose the support team handling small emergent app bugs over starting with broader engineering teams or complex fixes.

Rationale: Support was closest to the incoming small bugs and already needed a way to move faster through triage.

How to safely let support ship code.

Chose limited app-fix scope with engineering review over open-ended support changes across the whole codebase.

Rationale: Initial guardrails limited target apps and kept engineering review on every support-generated merge request.

Gotchas (what bit them)

Integration maintenance is a continuous erosion problem.

Root cause: Third-party APIs change and deprecate across a long-lived integration ecosystem.

Resolution: Empower support to fix smaller app bugs and use Scout Agent to automate diagnosis and MR generation.

Standalone AI APIs saw low engagement when they lived outside the workflow.

Root cause: Support and engineers had to leave their normal tools and visit another interface.

Resolution: Embed diagnosis APIs into Zaps, MCP tools, and eventually Scout Agent.

Slow diagnosis tools frustrated synchronous IDE workflows.

Root cause: The valuable context-gathering diagnosis step could take a long time to run.

Resolution: Move from user-driven tool use toward an orchestrated agent that can do work before support picks up the ticket.

Evaluation

Zapier evaluates Scout Agent with categorization and fixability evals, feedback-derived test cases, generated-fix share, and support-team velocity changes.

Metrics: 75% accuracy for categorization and fixability evals · 40% of support-team app fixes generated by Scout · some support members increased from one to two tickets per week to three to four

Outcomes

• Scout generates a substantial share of support-team app fixes.
• Some support-team members doubled their weekly fix velocity.
• Potentially fixable tickets are surfaced directly in the triage flow.
• Several support members moved toward engineering work according to the project summary.

Auto-caption transcript; the project is concrete, but several metrics are spoken informally and should be treated as directional.