Building and Deploying AI Agents for Account Qualification

unify · Tech · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Go-to-market account qualification

Unify needed AI agents that could research companies and people, decide whether an account matched an ideal customer profile, and make the agent's reasoning observable enough for users and developers to trust and iterate.

Constraints:

Account qualification requires web research, page navigation, synthesis, and criteria-based decisions.
High-quality planning improved results but increased runtime.
Users disliked staring at a spinner during long agent runs.
The team needed systematic experiment comparison without building extensive custom ML infrastructure.

Architecture

Unify built account-qualification agents with LangGraph as the agent state machine and LangSmith for experimentation and tracing. The architecture evolved from a trivial agent to a plan-reflect-tools loop, using o1-preview for stronger planning, parallel tool calls for speed, an async endpoint for long-running execution, and a UI that streams agent actions and decisions step by step.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user[User] --> agent-ui[Agent UI]
  agent-ui --> async-endpoint[Async Endpoint]
  async-endpoint --> langgraph-agent[LangGraph Agent]
  langgraph-agent --> planning-model[Planning Model]
  langgraph-agent --> reflection-model[Reflection Model]
  langgraph-agent --> research-tools[Research Tools]
  async-endpoint --> progress-database[Progress Database]
  langgraph-agent --> langsmith[LangSmith]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User`	GTM user defining ICP criteria and reviewing qualification results.	`Agent UI`
`Agent UI`	Displays real-time agent actions and decisions.	`Async Endpoint`
`Async Endpoint`	Starts long-running agent execution and exposes progress polling.	`LangGraph Agent`, `Progress Database`
`LangGraph Agent`	Plan-reflect-tools state machine for account research.	`Planning Model`, `Reflection Model`, `Research Tools`
`Planning Model`	Generates high-level account research plans.	`LangGraph Agent`
`Reflection Model`	Chooses next steps based on what is known or unknown.	`LangGraph Agent`
`Research Tools`	Searches web pages, visits sites, and scrapes information.	`LangGraph Agent`
`Progress Database`	Stores logged agent progress for frontend polling.	`Async Endpoint`
`LangSmith`	Versioned datasets, traces, and experiment comparison.	`LangGraph Agent`

Data flow

A user provides a company and qualification criteria. The agent creates a plan, loops between reflection and tool calls, searches and visits web pages, logs progress to the database, and the frontend polls for step-by-step updates until the final qualified/not-qualified result.

“Given a company and a set of questions and criteria, the agent performs some research and decides whether they are “qualified” or not.”article, stated

“converted this to an “async” endpoint that starts the agent execution and returns an ID that can be used to poll for progress”article, stated

Why this architecture

Unify used a plan-reflect-tools graph because account qualification requires iterative research rather than one model call. LangSmith gave the team versioned datasets and trace comparison, while the async UI made slow but capable agents usable.

“We’re still using the plan-reflect-tools state machine structure for our latest iteration. The areas we’ve been most actively tuning and experimenting with are speed and user experience.”article, stated

“In particular, versioned datasets are exactly what we were hoping they would be. Running and comparing experiments is straightforward, and the tracing is also excellent.”article, stated

Models used: o1-preview · GPT-4o · Claude 3.5 Sonnet

Scale

LangSmith let Unify run a new agent version on hundreds of examples.
“We’re able to run a new agent version on hundreds of examples and quickly compare it against previous versions on a given dataset with very little in house ML infra work.”article, stated
The agent uses parallel web-page scraping because each page can take several seconds to load.
“This helps a lot since each webpage can take several seconds to load. It works well intuitively because humans often do the same thing—quickly opening multiple Google results at once in new tabs.”article, stated

Alternatives considered (roads not taken)

Barebones agent without a real prompt or structure.

Rejected because: The team moved to an initial-plan plus reflect-step architecture.

“We used LangGraph as the framework for the agent state machine and LangSmith as the experimentation and tracing framework. Our starting point was a trivial agent that was about as barebones as you can imagine (not even a prompt):”article, stated

Synchronous prediction endpoint.

Rejected because: Longer runtimes and step-by-step UI required async execution and polling.

“this off also required some engineering changes. We originally had a simple prediction endpoint that would hold a request open while the agent ran. To deal with longer agent runtimes and accomplish the new step-by-step UI, we converted this to an “async” endpoint that starts the agent”article, stated

Hide execution behind a spinner.

Rejected because: Users needed real-time visibility into actions and decisions.

“The initial designs showed a spinner to the user while the agent ran (which gets pretty painful after a few seconds). Our updated interface instead shows the actions and decision-making process of the agent as it runs in real time.”article, stated

Patterns

`multi-agent-orchestration` (agent loop)

The account agent uses planning, reflection, and tool-calling roles inside a LangGraph state machine.

“looping between a “reflect” step and tool calling”article, stated

`tool-use-orchestration` (research)

Agents search the web, visit websites, navigate pages, and synthesize findings.

“searching the web, visiting websites, navigating between pages”article, stated

`observability-tracing` (development)

LangSmith traces and compares agent versions across datasets.

“experimentation and tracing framework”article, stated

`latency-optimization` (agent runtime)

Parallel tool calls and async progress polling reduce perceived wait time.

“biggest speed boosts we achieved came from parallelizing tool calls”article, stated

Tradeoffs

Planning model

Chose o1-preview for plan formation over Mainline models alone.

Rationale: o1 plans were more comprehensive and hard to reproduce with other models.

“best results we obtained at this stage were with OpenAI’s o1-preview model.”article, stated

Speed strategy

Chose Parallel tool calls over Strictly serial web scraping.

Rationale: Multiple pages can load at once, similar to a human opening several search results.

“allowed the model to scrape multiple web pages at a time.”article, stated

UX model

Chose Step-by-step progress UI over Spinner while the agent runs.

Rationale: The agent could stay capable without making waiting feel opaque.

“gets pretty painful after a few seconds”article, stated

Gotchas (what bit them)

Better planning can make agents too slow.

Root cause: o1-preview produced better plans but substantially increased runtime.

Resolution: Pair heavier planning with parallel tool calls and a real-time UI.

“o1-preview model is a step up from other models”article, stated

A black-box spinner destroys user trust during long agent runs.

Root cause: Users cannot tell what the agent is doing or whether it is stuck.

Resolution: Show actions and decision-making in real time.

“shows the actions and decision-making process of the agent as it runs in real time”article, stated

Reflection needs honest uncertainty.

Root cause: The agent must choose useful next steps based on what it does not know.

Resolution: Tune the reflection model for honesty about unknowns.

“honesty about what it does not yet know”article, stated

Evaluation

Unify evaluates agent versions using LangSmith versioned datasets, traces, experiment comparison, and hundreds of examples, while also assessing runtime and UX.

Metrics: Hundreds of examples · Agent-version comparison · Runtime · Trace quality · User-visible progress

“quickly compare it against previous versions on a given dataset”article, stated

Outcomes

Unify launched account-qualification agents powered by LangGraph and LangSmith.
The architecture evolved to a plan-reflect-tools state machine.
The UI shifted from spinner-based waiting to real-time action visibility.

The source reports hundreds of evaluation examples, multi-second web-page loads, and async endpoint polling for long-running agent executions.

“Agents are a new feature we’re launching”article, stated