Building and Deploying Production AI Agents for Enterprise Data Analysis

asterrave · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data-analysis agents

Asterrave needed production AI agents that could query enterprise data warehouses through natural language while respecting security credentials, model-specific interfaces, and the limits of agent frameworks.

Constraints:

Enterprise warehouses can contain 500- or 1000-column tables and very large schema contexts.
Agents need retrieval/tool calls rather than relying on model knowledge.
Different models prefer different Agent Computer Interface formats, such as JSON or XML.
User security credentials must cascade into the agent safely.

Architecture

Asterrave rebuilt around reasoning-focused agents that use discrete retrieval tools such as search tables, get table detail, and profile data rather than RAG over everything. The team tuned the Agent Computer Interface by model, avoided fine-tuning for reasoning, and used manager-worker hierarchies with specialized agents and tool sets rather than a single overloaded agent.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-user[Enterprise User] --> manager-agent[Manager Agent]
  manager-agent --> worker-agents[Worker Agents]
  worker-agents --> retrieval-tools[Retrieval Tools]
  retrieval-tools --> data-warehouse[Data Warehouse]
  manager-agent --> aci-formatter[ACI Formatter]
  worker-agents --> aci-formatter[ACI Formatter]
  credential-layer[Credential Layer] --> manager-agent[Manager Agent]
  credential-layer --> retrieval-tools[Retrieval Tools]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Enterprise User`	User asking natural-language questions over warehouse data.	`Manager Agent`
`Manager Agent`	Owns the final outcome and delegates subtasks.	`Worker Agents`, `Credential Layer`
`Worker Agents`	Specialized agents with narrower instructions and tools.	`Retrieval Tools`
`Retrieval Tools`	Search tables, get table details, profile data, and related warehouse operations.	`Data Warehouse`
`Data Warehouse`	Enterprise SQL/database environment.	`Retrieval Tools`
`ACI Formatter`	Formats tool/context responses as JSON, XML, or other model-specific structures.	`Manager Agent`, `Worker Agents`
`Credential Layer`	Cascades end-user security credentials into agent actions.	`Manager Agent`, `Retrieval Tools`

Data flow

A user asks a data question. The manager agent delegates to specialized workers, workers call retrieval tools against data warehouse metadata and tables, responses are formatted for the target model's preferred ACI, and user credentials cascade through the agent/tool chain.

“discreet um more simple uh building blocks of tool calls such as search tables get table detail or profile a column the agent is then tasked with using those iteratively to find the right columns for the right query similarly we saw this play out as”transcript, stated

“security credentials to Cascade down to the agent they were talking to so think about uh if if a human is trying to use an agent to query their snowflake account they may have very granular permissions within that snowflake”transcript, stated

Why this architecture

Asterrave focused on reasoning over model knowledge because enterprise data questions require fresh warehouse retrieval and secure tool execution. Model-specific ACI formatting and limited manager-worker hierarchies kept agents effective without overwhelming a single prompt.

“the agent's actions instead we focused on discrete tool calls that allowed it to perform retrieval and get the relevant context into its its context window while it was working the uh product that we built was enabling a AI agent to search and query”transcript, stated

“those tasks whereas if you gave all of that information to a single manager agent it could become overwhelmed it might um make bad decisions go down bad bad paths uh we also learned that the um number of Agents working together”transcript, stated

Models used: GPT-4 · Claude · manager agent · worker agents

Scale

The source discusses 500- or 1000-column data-warehouse tables.
“500 or thousand column tables in their data warehouse so it was understandable if you're getting you know 30,000 tokens back that there might be some challenges there but we felt like to consistently be completely blind to it um there had”transcript, stated
Large schema contexts can reach around 30,000 tokens.
“if you're getting you know 30,000 tokens back that there might be some challenges there but we felt like to consistently be completely blind to it um there had to be a way to improve this so we tested”transcript, stated
The recommended manager-worker hierarchy is limited to around 5-8 agents according to the summary.
“Key insights included optimizing the Agent Computer Interface (ACI) with format-specific responses (JSON for GPT-4, XML for Claude), avoiding fine-tuning in favor of reasoning-focused architectures, and designing multi-agent systems with manager-worker hierarchies limited to 5-8 agents.”summary, stated

Alternatives considered (roads not taken)

Use RAG as the main enterprise data interface.

Rejected because: Asterrave found discrete retrieval tools more useful than relying on model knowledge.

“focused on retrieval rather than trying to do rag where we inserted contents into the system prompt um to to guide the agent's actions instead we focused on discrete tool calls that allowed it”transcript, stated

Fine-tune models for the reasoning task.

Rejected because: The team concluded fine-tuning did not improve reasoning for this setting.

“fine-tuning models was a waste of time uh I think this is generally accepted now but there's still a little bit of work happening on building agents with fine-tuned models if you buy the premise”transcript, stated

Use framework defaults such as LangGraph for production.

Rejected because: Cascading user security credentials and other production needs pushed them toward custom implementation.

Patterns

`tool-use-orchestration` (data agents)

Agents use discrete retrieval tools for table search, table detail, and profiling.

“search tables get table detail or profile”transcript, stated

`multi-agent-orchestration` (agent hierarchy)

A manager owns outcomes and delegates subtasks to specialized workers.

“manager agent would own the final outcome but could delegate subtasks”transcript, stated

`structured-output-contract` (ACI)

Model-facing responses are formatted as JSON for GPT-style models and XML for Claude.

“provide XML back to the model”transcript, stated

`safety-policy-enforcement` (enterprise security)

End-user credentials cascade into agent actions.

“enduser security credentials to Cascade down”transcript, stated

Tradeoffs

Knowledge access

Chose Discrete retrieval tools over Model knowledge or broad RAG.

Rationale: Enterprise data analysis needs fresh, precise access to tables and schemas.

“underlying model knows so a lot of our tool calls were focused on retrieval”transcript, stated

Model adaptation

Chose Reasoning-focused architecture over Fine-tuning.

Rationale: Fine-tuning did not improve reasoning in their experience.

“fine-tuning models was a waste of time”transcript, stated

Framework choice

Chose Custom production implementation over LangGraph defaults.

Rationale: Security-credential cascading and production constraints required custom behavior.

“custom implementation was preferable to frameworks like LangGraph”summary, stated

Gotchas (what bit them)

Model-specific interface formats matter.

Root cause: Claude and GPT responded better to different ACI representations.

Resolution: Return XML to Claude and JSON-style formats where appropriate for GPT.

“depending”transcript, implied

A single manager agent can become overloaded.

Root cause: Putting all context, tools, and instructions in one agent overwhelms it.

Resolution: Use specialized workers with narrower instructions and tools.

“more context in their instructions and more specific tool calls”transcript, stated

Fine-tuning does not fix reasoning over fresh enterprise data.

Root cause: The value comes from retrieval/tool use and reasoning, not static model knowledge.

Resolution: Avoid fine-tuning and invest in tool calls and ACI quality.

“fine-tuning does not really improve reasoning”transcript, stated

Evaluation

Asterrave evaluated through two years of production lessons: ACI format quality, tool-call reliability, fine-tuning experiments, security-credential requirements, and manager-worker hierarchy behavior.

Metrics: ACI fit by model · Tool retrieval quality · Reasoning quality · Credential-cascade correctness · Hierarchy manageability

“two-year journey of rebuilding their entire product around AI agents”summary, stated

Outcomes

Asterrave rebuilt its product around enterprise data-analysis agents.
The team favored discrete retrieval tools, custom production implementation, and model-specific ACI formatting.
They concluded fine-tuning was not useful for reasoning-focused data agents.

The source reports 500- or 1000-column tables, 30,000-token schema contexts, and manager-worker hierarchies limited to 5-8 agents in the summary.

“product enables AI agents to search and query enterprise data warehouses”summary, stated