Production Lessons from Building and Deploying AI Agents

rasgo · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data-analysis agents

Rasgo needed production AI agents that could query Fortune 500 internal data through tools, reason over failures, show their work, and generalize across many enterprise schemas without framework abstractions obscuring control.

Constraints:

Text-to-SQL and data analysis require reasoning, schema exploration, and self-correction.
Tool-call interface details strongly affect model behavior and vary by model version.
Enterprise deployment requires security, connectors, UI inspectability, memory, logging, and evaluation.
Agent evaluation is nondeterministic and never complete.

Architecture

Rasgo builds a model-agnostic data-analysis agent loop: conversation objective and system prompt, LLM completion, tool-call execution, observation, and repeated reasoning until completion. The platform connects to Snowflake/BigQuery-like warehouses, exposes schema/profile/search/SQL/Python/visualization tools through a carefully iterated Agent-Computer Interface, returns errors for self-correction, shows tool work in interactive UI, maintains explicit memory with human confirmation, and owns model calls directly instead of relying on production abstractions such as LangChain.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-user-objective["Enterprise user objective"] --> agent-reasoning-loop["Agent reasoning loop"]
  agent-reasoning-loop["Agent reasoning loop"] --> agent-computer-interface["Agent-Computer Interface"]
  agent-reasoning-loop["Agent reasoning loop"] --> model-provider-layer["Model provider layer"]
  agent-reasoning-loop["Agent reasoning loop"] --> evaluation-harness["Evaluation harness"]
  agent-computer-interface["Agent-Computer Interface"] --> schema-exploration-tools["Schema exploration tools"]
  agent-computer-interface["Agent-Computer Interface"] --> sql-execution-tool["SQL execution tool"]
  agent-computer-interface["Agent-Computer Interface"] --> python-analysis-tool["Python analysis tool"]
  agent-computer-interface["Agent-Computer Interface"] --> visualization-tool["Visualization tool"]
  schema-exploration-tools["Schema exploration tools"] --> enterprise-data-warehouse["Enterprise data warehouse"]
  sql-execution-tool["SQL execution tool"] --> enterprise-data-warehouse["Enterprise data warehouse"]
  enterprise-data-warehouse["Enterprise data warehouse"] --> security-and-connector-layer["Security and connector layer"]
  visualization-tool["Visualization tool"] --> interactive-audit-ui["Interactive audit UI"]
  memory-system["Memory system"] --> agent-reasoning-loop["Agent reasoning loop"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Enterprise user objective`	Natural-language data-analysis question from an enterprise user.	`Agent reasoning loop`
`Agent reasoning loop`	Model-driven loop that chooses tools, observes outcomes, and decides next steps.	`Enterprise user objective`, `Agent-Computer Interface`, `Model provider layer`, `Evaluation harness`
`Model provider layer`	Supports GPT-4, GPT-4o, Claude, Cohere, and other models.	`Agent reasoning loop`
`Agent-Computer Interface`	Precise syntax/semantics of tool inputs, outputs, names, and formats.	`Agent reasoning loop`, `Schema exploration tools`, `SQL execution tool`, `Python analysis tool`, `Visualization tool`
`Schema exploration tools`	Searches schemas, metadata, and profile distributions before query writing.	`Agent-Computer Interface`, `Enterprise data warehouse`
`SQL execution tool`	Runs SQL and returns detailed errors/context for self-correction.	`Agent-Computer Interface`, `Enterprise data warehouse`
`Python analysis tool`	Supports non-SQL analysis and transformations.	`Agent-Computer Interface`
`Visualization tool`	Produces charts and data visualizations for user inspection.	`Agent-Computer Interface`, `Interactive audit UI`
`Enterprise data warehouse`	Snowflake/BigQuery-like data source accessed with permissions.	`Schema exploration tools`, `SQL execution tool`, `Security and connector layer`
`Security and connector layer`	OAuth/SSO/token management and data connector maintenance.	`Enterprise data warehouse`
`Interactive audit UI`	Shows tool calls and results so users can trust and inspect agent work.	`Visualization tool`, `Agent reasoning loop`
`Memory system`	Explicit cross-session memory with human confirmation before committing.	`Agent reasoning loop`
`Evaluation harness`	Objective/completion pairs and tool-call accuracy checks.	`Agent reasoning loop`

Data flow

A Fortune 500 user asks a natural-language data question. The agent plans, explores schemas and data distributions, calls SQL/Python/visualization tools, observes errors/results, and iterates. Tool outputs are formatted through the ACI so the model can reason and users can inspect each step. Evaluation captures objective/completion pairs plus intermediate tool calls. Security, connectors, memory, and UI surround the loop.

3 supporting quotes

“Rasgo defines an AI agent as a system that can take action (tool calls) and reason about outcomes to make subsequent decisions.”summary, stated

“The agent architecture follows a straightforward loop pattern: starting a conversation with an objective and system prompt, calling a model for completion, handling any tool calls the model requests, iterating in a loop, and stopping when the task is complete.”summary, stated

“GPT-4, by contrast, would first create a plan with proper tool call sequencing and then execute that plan systematically.”summary, stated

Why this architecture

Rasgo treats reasoning and ACI design as the critical production levers. The model retrieves context and reasons through tool outcomes rather than acting as a database. Direct ownership of model calls gives the team the observability, logging, upgrades, and explanation control needed for enterprise agents.

3 supporting quotes

“agent performance depends more on reasoning capability than on what the model "knows."”summary, stated

“The ACI is described as requiring "as much art as science," more akin to UX design than traditional software development.”summary, stated

“full ownership of model calls—including all inputs and outputs—is essential for onboarding users, debugging issues, scaling, logging agent activity, upgrading versions, and explaining agent behavior.”summary, stated

Models used: GPT-4 · GPT-4o · GPT-4-32k · GPT-3.5-turbo · Anthropic Claude · Cohere Command R+

Scale

The lessons come from approximately one year of production agent work.
“over the course of approximately one year.”summary, stated
The platform serves Fortune 500 enterprise customers.
“allows Fortune 500 enterprise customers to query and analyze their internal data using natural language”summary, stated
Text-to-SQL benchmark accuracy caps around 80%, motivating self-correction.
“even on benchmarks, text-to-SQL accuracy caps around 80%.”summary, stated
Rasgo iterated on ACI hundreds of times.
“Rasgo reports iterating on their ACI "hundreds of times"”summary, stated
The author says GPT-4o was deployed in production within 15 minutes of API availability.
“the author deployed GPT-4o in production within 15 minutes of API availability”summary, stated

Alternatives considered (roads not taken)

Use the model as a knowledge database

Rejected because: Rasgo found reasoning over retrieved context more important than stored model knowledge.

“too much processing power goes into using models as databases rather than reasoning engines.”summary, stated

GPT-3.5-turbo for production reasoning

Rejected because: It hallucinated schemas and failed to plan reliably compared with GPT-4.

“On GPT-3.5, agents would frequently hallucinate table and column names, write failing queries”summary, stated

Fine-tune the main reasoning agent

Rejected because: Fine-tuned agents cheated from examples and reasoned worse on novel situations.

“fine-tuned models actually exhibited worse reasoning because the agent would "cheat"”summary, stated

Use LangChain/LlamaIndex in production

Rejected because: Framework abstractions hide model-call control needed for debugging, logging, upgrades, and explanation.

“These abstractions may be acceptable for rapid prototyping but become liabilities in production.”summary, stated

Patterns

`tool-use-orchestration` (agent architecture)

The agent reasons through schema exploration, SQL, Python, and visualization tool calls.

“they designed their agent to handle failures gracefully by returning detailed SQL errors with context, enabling the agent to iterate and self-correct.”summary, stated

`structured-output-contract` (ACI)

Tool-call syntax, inputs, outputs, and response formats are treated as a first-class interface.

“the exact syntax, structure, and semantics of tool calls including both inputs and outputs.”summary, stated

`rag-retrieval` (context)

The platform provides a metadata layer for RAG over enterprise schemas and data warehouses.

“The platform connects to enterprise data warehouses such as Snowflake and BigQuery, providing a metadata layer for RAG (Retrieval-Augmented Generation) and enabling data analysis through SQL, Python, and data visualization capabilities.”summary, stated

`agent-memory` (memory)

Cross-session memory requires explicit commit/retrieve mechanisms and human confirmation.

“Cross-session memory requires explicit mechanisms for committing and retrieving information. Notably, agents are described as poor at deciding what to commit to memory, typically requiring human confirmation.”summary, stated

`human-in-the-loop-gate` (memory governance)

Humans confirm what should be committed to long-term memory.

“Notably, agents are described as poor at deciding what to commit to memory, typically requiring human confirmation.”summary, stated

`eval-driven-development` (evaluation)

Rasgo evaluates both workflow success and individual tool-call accuracy with captured intermediate calls.

“Evaluation operates at two levels: overall workflow success and individual tool call accuracy.”summary, stated

Tradeoffs

Reasoning model choice

Chose GPT-4-class models for core agent reasoning over Faster GPT-3.5 for complex tasks.

Rationale: Users preferred GPT-4’s planning and decision-making despite speed tradeoffs.

“Despite the speed advantages of GPT-3.5, users strongly preferred GPT-4's superior decision-making.”summary, stated

Framework posture

Chose Direct ownership of model calls over LangChain/LlamaIndex abstractions.

Rationale: Production observability, scaling, logging, and explanation were more important than prototyping speed.

“This represents a significant operational choice that prioritizes observability and control over development speed”summary, stated

Cost optimization timing

Chose Avoid premature cost optimization over Optimizing agent cost before reliability.

Rationale: The technical recommendations explicitly defer cost optimization while agents and models improve.

“• Avoid premature cost optimization”summary, stated

Gotchas (what bit them)

ACI details are model-specific and fragile.

Root cause: Small changes to tool names, counts, formats, and outputs can alter agent behavior.

Resolution: Continuously test and iterate the ACI, including across model versions.

“Small changes can have cascading effects on agent behavior, and what works for one model may fail for another.”summary, stated

Fine-tuning can hurt general reasoning.

Root cause: The agent imitates fine-tuning examples rather than reasoning from the current problem.

Resolution: Use fine-tuned models as specialized tool components, not the main reasoning engine.

“using fine-tuned models as specialized components within tool calls rather than as the agent's primary reasoning engine.”summary, stated

Framework abstractions block production debugging.

Root cause: Production agents need full visibility into model inputs/outputs and tool behavior.

Resolution: Own model calls directly and avoid high-level frameworks where they hide control.

“full ownership of model calls—including all inputs and outputs—is essential for onboarding users, debugging issues, scaling, logging agent activity”summary, stated

Evaluation is manual and never complete.

Root cause: Agents can choose different tool sequences for the same objective.

Resolution: Capture objective/completion pairs and intermediate tool calls for workflow and tool-level evaluation.

“The inherent nondeterminism of agents—choosing different tool sequences for the same objective—complicates assessment.”summary, stated

Evaluation

Rasgo builds objective/completion pairs, captures intermediate tool calls, and evaluates both workflow success and individual tool-call accuracy while treating manual evaluation as ongoing.

Metrics: overall workflow success · individual tool-call accuracy · SQL error recovery · model comparison quality · user preference

“Rasgo's approach involves creating objective/completion pairs (initial direction and expected final tool call) and capturing intermediate tool calls for debugging.”summary, stated

Outcomes

Rasgo stabilized agents across diverse data sources and enterprise customers.
GPT-4-class reasoning substantially outperformed GPT-3.5 on tool-sequencing tasks.
The team concluded that infrastructure around the agent is the moat.

The summary cites Fortune 500 customers, approximately one year of production learning, hundreds of ACI iterations, text-to-SQL around 80%, and GPT-4o production deployment within 15 minutes.

“ultimately achieving stability across diverse data sources and enterprise customers.”summary, stated

“The case study emphasizes that the agent itself is not the competitive moat—the surrounding production infrastructure is where differentiation occurs.”summary, stated