Building Production-Ready AI Agents for Internal Workflow Automation

vercel · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal enterprise workflow automation with specialized AI agents

Vercel needed a repeatable way to identify and deploy internal AI agents that could automate high-volume but low-cognitive-load work across sales, trust and safety, and business intelligence without overreaching into tasks current models handle poorly.

Constraints:

• Many AI projects fail because teams pick tasks that are too hard or too poorly matched to current model strengths.
• Agents must integrate into existing business systems such as Slack, Salesforce, forms, APIs, and internal data platforms.
• High-stakes trust-and-safety decisions require human oversight rather than unrestricted automated takedowns.
• Data-analysis agents need business semantics, not only generic text-to-SQL translation.
• Reusable internal agents should be concrete enough to implement but specialized enough to test and reason about.
• Automation should save repetitive work without removing human strengths such as personalization and final judgment.

Architecture

Vercel frames agents as workflow steps where an LLM can autonomously use tools in a loop. The company selected internal projects by asking employees what busy work they hated, then built specialized agents for workflows with clear triggers, steps, and outputs. The lead-processing agent classifies contact-sales submissions, researches companies through LinkedIn and Google, drafts qualified-lead emails, hands them to salespeople for personalization, and records results in Salesforce. The anti-abuse agent ingests reports from a form or API, uses project/team/domain/payment/visual-inspection tools, summarizes findings with confidence, and routes recommendations to humans in Salesforce. The data-analyst agent accepts Slack questions, uses schema plus a curated semantic catalog to generate SQL, executes queries, summarizes results with graphics, and replies in the Slack thread. Vercel AI SDK and use workflow provide the TypeScript/workflow substrate, while sub-agents remain ordinary functions/tools.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  use-case-discovery-interviews[Use-case discovery interviews] --> workflow-design-layer[Workflow design layer]
  workflow-design-layer --> vercel-ai-sdk[Vercel AI SDK]
  workflow-design-layer --> use-workflow-abstraction[Use workflow abstraction]
  workflow-design-layer --> lead-processing-agent[Lead processing agent]
  workflow-design-layer --> anti-abuse-agent[Anti-abuse agent]
  workflow-design-layer --> data-analyst-agent[Data analyst agent]
  lead-processing-agent --> research-sub-agent[Research sub-agent]
  research-sub-agent --> linkedin-and-google-tools[LinkedIn and Google tools]
  lead-processing-agent --> salesperson-review[Salesperson review]
  lead-processing-agent --> salesforce[Salesforce]
  salesperson-review --> salesforce
  anti-abuse-agent --> abuse-report-intake[Abuse report intake]
  anti-abuse-agent --> project-inspection-tools[Project inspection tools]
  project-inspection-tools --> multimodal-model[Multimodal model]
  anti-abuse-agent --> salesforce
  salesforce --> trust-and-safety-queue[Trust and safety queue]
  data-analyst-agent --> slack-interface[Slack interface]
  data-analyst-agent --> semantic-catalog[Semantic catalog]
  semantic-catalog --> sql-execution-layer[SQL execution layer]
  data-analyst-agent --> sql-execution-layer

Components

Data flow

The shared workflow pattern starts with a trigger such as a contact-sales form, abuse report, or Slack message. A main agent enters a reasoning loop and calls specialized tools or sub-agents. For leads, it filters spam/support/low-quality submissions, researches the company and person, drafts an email, hands the draft to a salesperson, and updates Salesforce. For abuse reports, it examines the reported project, deployed assets, team history, domains, payment data, and visual content, then recommends block, manual review, or false positive for a human queue. For analytics, it reads schema and semantic-catalog context, builds and executes SQL, summarizes the result, generates graphics, and posts back to Slack.

Why this architecture

Vercel selected repetitive, hated tasks because they are frequent enough to matter and structured enough for current models. The workflow-step framing avoids overcomplicating agents: tools are functions, agents are functions, and sub-agents are simply specialized agents callable as tools. Human review remains where humans are better, such as personalization and takedown decisions, while the agent handles research, triage, and information synthesis.

Models used: 2025 frontier models · LLMs · Vercel AI SDK · use workflow · Multimodal models · Sub-agents · Text-to-SQL

Scale

• Roughly 40% of the audience had already shipped an agent, according to the speaker's show-of-hands estimate.
  “Who here has already shipped an agent in their in their career? All right. I I I call it 40%.”transcript, stated
• The speaker cited a 94% AI-project failure statistic as context for task-selection discipline.
  “see these statistics like 94% of AI projects fail. Um, and I have a thesis of why that is.”transcript, stated
• The lead-enrichment workflow took roughly 15 minutes per lead and was also described as 15-25 minutes of busy work per lead.
  “they spend about 15 minutes enriching the data um before they can actually talk to the person”transcript, stated

  “it's about 25 20 50 minutes of busy work every lead.”transcript, stated
• The lead agent saved hundreds of days of work, according to the speaker.
  “In our case, this actually so it has saved literally hundreds of of days.”transcript, stated
• Vercel handled about 400 million deployments in the prior year.
  “there were about 400 million deployments to Verscell in the last year”transcript, stated
• The abuse pipeline receives thousands of reports per week from forms and API reporters.
  “large scale reporters actually have an API and and so we actually getting thousands and thousands of reports per week.”transcript, stated
• The anti-abuse agent reduced time per case by about 59%.
  “since we introduced this, we got about a 59% reduction in in time per case.”transcript, stated
• The speaker described prior dashboard fragmentation at Google as about 15 different dashboard platforms.
  “I used to work at Google where we had like 15 different platforms of dashboard.”transcript, stated

Alternatives considered (roads not taken)

Top-down mandate to build agents

Rejected because: The speaker argues that successful projects start from concrete repetitive work rather than abstract organizational pressure to use agents.

Starting with the most difficult company problem

Rejected because: The methodology intentionally targets low-cognitive-load repetitive work that frontier models can do reliably today.

Monolithic general agents

Rejected because: Vercel treats sub-agents as simple specialized functions/tools, making workflows easier to compose and reason about.

Letting the anti-abuse agent take down websites automatically

Rejected because: Vercel's default as a hosting provider is not to take down websites, so the agent prepares evidence and recommendations for human review.

Generic text-to-SQL tools without business context

Rejected because: The data agent relies on a curated semantic catalog that teaches company-specific business meaning, which generic tools lack.

Opening the anti-abuse agent source publicly

Rejected because: Vercel withheld that code because abuse detection is adversarial and public details can help bad actors evade detection.

Patterns

tool-use-orchestration (agent-architecture)

Vercel defines agents as LLMs using tools in a loop and embeds agents as decision-capable steps inside ordinary workflows.

workflow-decomposition (workflow-design)

The talk decomposes business processes into trigger, steps, outputs, and replaceable agentic substeps; the same decomposition is used for lead, abuse, and analytics agents.

human-in-the-loop-gate (deployment-safety)

Vercel keeps humans at final personalization and high-stakes review points: salespeople personalize lead emails and trust-and-safety staff act on abuse recommendations.

multi-agent-orchestration (agent-architecture)

Vercel treats sub-agents as specialized agents callable by larger agents, such as a LinkedIn/Google research sub-agent inside the lead workflow.

rag-retrieval (data-access)

The data analyst agent uses schema and a curated semantic catalog as business-specific context before generating SQL and summarizing results.

structured-output-contract (analytics)

The analytics agent turns natural-language Slack questions into SQL queries and structured results before posting human-readable summaries and graphics.

safety-policy-enforcement (trust-and-safety)

The abuse agent combines platform metadata and visual inspection to recommend block/manual-review/false-positive outcomes while leaving final takedown authority to humans.

Tradeoffs

Find agent projects through employee pain interviews

Chose Ask departments what busy work they hate over Starting from a generic directive to build agents.

Rationale: The pain-interview method finds repetitive tasks that current models can handle and that employees actively want automated.

Keep salespeople in the lead-email loop

Chose Agent drafts plus human personalization over Fully automated outreach email sending.

Rationale: The agent is well suited to repetitive research, while humans remain better at nuanced personalization and relationship handling.

Use human review for abuse enforcement

Chose Agent recommendations in Salesforce over Automated website blocking.

Rationale: The agent accelerates investigation while preserving human accountability for potentially harmful takedown actions.

Invest in a semantic catalog for data analysis

Chose Company-specific semantic context plus SQL generation over Off-the-shelf generic text-to-SQL.

Rationale: The semantic catalog lets the agent choose tables and columns using business meaning, which generic tools do not know.

Open-source reference agents selectively

Chose Lead and data analyst agents as public reference architectures over Open-sourcing every internal agent or selling them as agent-as-a-service products.

Rationale: Reference implementations help developers learn concrete patterns, while adversarial anti-abuse logic stays private.

Gotchas (what bit them)

AI project selection fails when teams choose tasks outside current model strengths.

Root cause: Teams overgeneralize from AI's coding ability and assume the same capability applies to every business problem.

Resolution: Start with boring, repetitive, low-cognitive-load tasks employees hate, then scale from successful internal examples.

A sales agent can degrade customer experience if support requests are routed to sales.

Root cause: Contact-sales forms receive spam, low-quality submissions, and support requests mixed with legitimate leads.

Resolution: Run initial classification and route support requests to support before doing lead research.

Autonomous abuse takedowns would conflict with Vercel's hosting-provider default.

Root cause: False positives in takedown decisions can remove legitimate websites, and abuse detection is adversarial.

Resolution: Use the agent to gather evidence and recommend actions, but keep human operators responsible for the final queue decision.

Generic text-to-SQL lacks business context.

Root cause: Schema alone does not teach the agent which tables, columns, and metrics reflect the business's real semantics.

Resolution: Curate a semantic catalog that teaches the agent company-specific business meaning before query generation.

Open-sourcing adversarial detection code can help attackers.

Root cause: Abuse prevention is a cat-and-mouse game, so publishing detection details may teach bad actors how to evade the system.

Resolution: Open-source non-adversarial reference agents, but share the abuse-agent code privately only with similar teams.

Evaluation

The talk reports measured business impacts for deployed internal agents and frames evaluation as choosing tasks that match model capability, instrumenting workflow outputs, and preserving human review where correctness matters. The lead agent is measured through saved manual work; the abuse agent is measured by time per case; the data analyst agent is validated through real business questions, including the deployments-per-year statistic used in the talk.

Metrics: Hundreds of days saved by lead processing · 59% reduction in anti-abuse time per case · Thousands of abuse reports per week handled by intake · 400 million deployments per year as a data-agent researched fact · Human review of lead emails and abuse recommendations · Quality of data answers returned to Slack with readable summaries and graphics

Outcomes

• Vercel identified production agent candidates by interviewing departments about repetitive work employees hated.
• The lead-processing agent saved hundreds of days of manual lead enrichment while keeping salespeople responsible for personalization.
• The anti-abuse agent reduced time per case by 59% and maintained human review for takedown decisions.
• The data analyst agent answered Slack-based business questions using schema plus semantic-catalog context and generated human-readable summaries and graphics.
• Vercel open-sourced the lead and data analyst reference implementations while keeping anti-abuse code private due to adversarial risk.
• The talk argues that specialized workflow agents are a practical way to fill previously uneconomic automation gaps.

The case describes internal production agents at Vercel: a 400-million-deployments-per-year platform, thousands of abuse reports per week, lead work taking about 15 minutes per submission, hundreds of days saved by the lead agent, and a 59% reduction in anti-abuse time per case. The data agent's output was used to research the 400-million-deployments statistic cited in the talk.