Evolving Agent Architecture Through Model Capability Improvements

aomni · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: revenue-team AI agents and agent architecture simplification

Aomni needed AI agents that could orchestrate complex revenue playbooks across many SaaS tools while remaining reliable enough for enterprise sales teams and adapting as base models improved.

Constraints:

• Sales teams use many narrow SaaS tools that humans or support roles must orchestrate manually.
• Enterprise B2B sales should keep humans talking to customers rather than replacing them with AI SDRs.
• Agents have the same reliability issues as long-running microservice workflows.
• Early models required heavy scaffolding, guardrails, retries, reflection, and multi-agent structures.
• As model capabilities improve, old scaffolding can become baggage that reduces flexibility.

Architecture

Aomni's architecture treats revenue agents as workflow orchestration systems around tools and data. The original systems used heavy scaffolding and many prompts to make GPT-3.5/GPT-4 agents reliable, but newer model capabilities let Aomni rewrite products with less scaffolding, simpler recursive research loops, and fewer LLM calls while preserving enterprise reliability mechanisms and human sales ownership.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  sales-rep-prompt["Sales rep prompt"] --> revenue-agent["Revenue agent"]
  revenue-agent --> tool-and-data-layer["Tool and data layer"]
  tool-and-data-layer --> workflow-orchestration-layer["Workflow orchestration layer"]
  workflow-orchestration-layer --> recursive-research-loop["Recursive research loop"]
  recursive-research-loop --> clarification-loop["Clarification loop"]
  clarification-loop --> sales-rep-prompt
  evaluation-scripts["Evaluation scripts"] --> workflow-orchestration-layer

Components

Data flow

A sales rep asks an AI agent for a playbook such as account research, contact identification, and outreach preparation. The agent uses configured tools and data sources, runs workflow orchestration with guardrails/retries, produces a report or actions, may ask clarifying questions, and feeds learnings recursively into further research directions.

Why this architecture

Aomni keeps humans in enterprise sales relationships and uses agents for support work because high-value deals depend on human trust. Architecturally, it removes scaffolding as models improve because excess multi-agent structure can hard-code behavior that better models could handle more flexibly.

Models used: GPT-3.5 · GPT-4 · O3 Mini · Sonnet · Deep research agent · Tool-calling models · Langfuse

Scale

• Sales teams often use five to twenty pieces of software.
  “a sales team has anywhere between like five to 20 pieces of soft Ware”transcript, stated
• Aomni says support functions often start when companies have six to ten sales reps.
  “typically after you have you know six to 10 sales rep is usually when you should start pairing them up with these like support functions”transcript, stated
• The speaker says typical sales reps spend about 30% of time talking to customers.
  “the typical sales or only spend about like 30% of the time uh actually talking to customers”transcript, stated
• Enterprise deals can drive 60-80% of revenue in typical enterprise companies.
  “generally anywhere between 60 to 80% of the revenue comes from the Enterprise deals”transcript, stated
• The newer research architecture reduced roughly 20-30 prompts to two LLM calls in a loop.
  “there were probably like you know 20 to 30 like different prompts different L calls essentially in that whole process”transcript, stated

Alternatives considered (roads not taken)

AI SDRs that talk to customers.

Rejected because: Aomni argues enterprise sales still depends on human relationships for high-value deals.

Heavy multi-agent swarms.

Rejected because: The speaker treats excessive LLM calls, reflection, and validation as baggage that shows the core system is not strong enough.

Service discovery tool for loading tools at runtime.

Rejected because: The model was not reliable enough at calling the discovery tool before giving up.

Patterns

tool-use-orchestration (revenue playbooks)

Aomni agents coordinate sales intelligence, email, CRM, enrichment, and other revenue tools from a high-level prompt.

durable-execution (workflow reliability)

Aomni treats agents as long-running workflow orchestration systems with microservice-like reliability concerns.

human-in-the-loop-gate (sales relationship ownership)

The agent prepares support work while the human salesperson retains customer-facing relationship work.

eval-driven-development (tool-use evals)

Aomni uses eval datasets and scripts to check tool use, though the speaker notes they can fail improved model behavior.

Tradeoffs

Support sales reps instead of replacing them.

Chose AI support function for revenue teams over AI SDRs that talk directly to all customers.

Rationale: Enterprise deal value depends on human relationship-building while agents can handle research and tool orchestration.

Simplify deep research architecture.

Chose Two LLM calls in a recursive loop over 20-30 prompts with multi-agent reflection and editor personas.

Rationale: Newer models were strong enough that simpler recursive research worked better and with less hard-coded baggage.

Attach a large tool context instead of relying on service discovery.

Chose Always include tool descriptions over A service-discovery tool that loads tools at runtime.

Rationale: The model did not reliably call discovery before giving up, so direct tool context was more dependable.

Gotchas (what bit them)

Reliability expectations are enterprise-grade, not demo-grade.

Root cause: Sales agents are used for actual enterprise work, where failures affect jobs and deals.

Resolution: Add production-level guardrails, retries, and workflow reliability mechanisms.

Scaffolding can become obsolete as models improve.

Root cause: Guardrails and multi-agent structures encode assumptions needed for weaker models.

Resolution: Rewrite products and remove scaffolding when model capability jumps.

Eval scripts can punish better tool-use strategies.

Root cause: Hardcoded expected tool sequences may fail when a model solves the task correctly in a different order.

Resolution: Evaluate outcome correctness and tool-use quality, not only exact sequence adherence.

Evaluation

Aomni evaluates agents with Langfuse, custom eval scripts, expected tool-use sequences, internal datasets, production reliability targets, and manual judgment about whether scaffolding can be removed after model upgrades.

Metrics: Tool-use sequence adherence · Task completion despite reordered tools · Production reliability · Research depth · Research breadth · Scaffolding removed after model upgrades · Enterprise customer reliability

Outcomes

• Aomni focuses agents on revenue support work rather than customer replacement.
• Reliability mechanisms made early agents work where many AutoGPT-style runs failed.
• Newer models allowed a deep-research agent to shrink from many calls and personas to a simpler recursive loop.
• The company treats product rewrites as necessary when model capability changes substantially.

The source cites sales teams using 5-20 tools, support functions beginning around 6-10 reps, sales reps spending about 30% of time with customers, enterprise deals driving 60-80% of revenue, and deep research simplification from 20-30 calls to two.