Building Production-Ready SQL and Charting Agents with RAG Integration

numbers_station · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Self-service enterprise analytics agents

Numbers Station needed to turn natural-language analytics demos into production systems that can answer business questions, write SQL, generate charts, find dashboards, and respect enterprise context and permissions.

Constraints:

• Data teams were overloaded by requests from data consumers.
• Enterprise semantics such as metric definitions live across warehouses, catalogs, dashboards, Slack, reports, and user feedback.
• Vanilla LLM SQL demos often fail once users expect accurate business-contextual answers.
• Manual SQL-generation pipelines break when queries fail or require iterative correction.
• The product has to serve non-technical users, so wrong answers are especially costly.
• Routing many specialized agents can become expensive or complex as the number of agents grows.

Architecture

Numbers Station's production architecture combines a unified knowledge layer, RAG tools, and a hierarchical multi-agent framework. Connectors ingest metadata and business knowledge from the modern data stack; the knowledge layer stores tables, columns, metrics, dimensions, dashboard/report context, Slack/message history, and customer-specific semantics; a SQL agent uses tool calling to search that knowledge and execute warehouse queries with retry/correction loops; a charting agent uses Python execution and documentation retrieval; and a planner agent decomposes user requests, routes to specialized agents, and preserves shared conversation history across handoffs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> planner-agent[Planner agent]
  planner-agent --> sql-agent[SQL agent]
  planner-agent --> charting-agent[Charting agent]
  planner-agent --> report-search-agent[Report search agent]
  planner-agent --> messaging-agent[Messaging agent]
  modern-data-stack-connectors[Modern data stack connectors] --> knowledge-layer[Knowledge layer]
  modern-data-stack-connectors --> governance-layer[Governance layer]
  knowledge-layer --> rag-tool[RAG tool]
  rag-tool --> sql-agent
  sql-agent --> sql-execution-tool[SQL execution tool]
  sql-execution-tool --> data-warehouse[Data warehouse]
  data-warehouse --> governance-layer
  charting-agent --> python-execution-tool[Python execution tool]
  charting-agent --> documentation-search-tool[Documentation search tool]
  sql-agent --> shared-conversation-history[Shared conversation history]
  charting-agent --> shared-conversation-history
  messaging-agent --> shared-conversation-history
  shared-conversation-history --> planner-agent

Components

Data flow

Connectors pull metadata and knowledge from warehouses, dashboards, catalogs, Slack, and other sources into a unified knowledge layer. A user asks the planner; the planner routes to a SQL, charting, report, or messaging agent; the SQL agent retrieves relevant business definitions from the knowledge layer, writes SQL, executes it against the warehouse, reads any error, and retries if needed. Results can then be handed through shared conversation history to a charting or messaging agent.

Why this architecture

The architecture exists because basic prompting and fixed pipelines do not meet production analytics needs. Tool-using agents can recover from SQL execution errors; RAG gives the model business definitions it cannot infer; a planner lets one chat surface expose multiple specialized capabilities; shared history makes handoffs natural; and source-system permissions avoid duplicating governance.

Models used: Large language models · Embedding models

Scale

• The talk describes enterprises investing in modern data stacks over the past decade.
  “in the past decade we've seen many Enterprises investing in setting up uh what we call a modern Data stock”transcript, stated
• Vouch is cited as a customer that used the platform to self-serve data through chat.
  “vouch which is an insure tech company they grew their business very quickly and they were getting overwhelmed with these data requests”transcript, stated

  “we help them self- serve and um bring AI experience for the end users through a chat uh interface”transcript, stated
• Setup improved from a couple of weeks to a couple of hours of training.
  “a year ago it was a very big process like couple of weeks of setup and US manually sitting with customers”transcript, stated

  “we've brought that down to like a couple of hours of training”transcript, stated
• Their current hierarchical design works smoothly at about a dozen agents and is expected to remain fine around 10-15 agents.
  “today like we have at most a dozen of agents and the experience is still very smooth”transcript, stated

  “if we get to up to like 10 15 agents uh it hasn't been a a problem”transcript, stated
• The speaker says a single hierarchy would not work at 50 or hundreds of agents.
  “if we had like 50 or hundreds obviously this design like the hierarchical would not work”transcript, stated

Alternatives considered (roads not taken)

Basic few-shot or zero-shot SQL prompting

Rejected because: It produces SQL text but leaves the user to copy, paste, execute, and handle failures manually.

Manual generation-then-execution control flow

Rejected because: A fixed pipeline gets stuck when a query does not compile unless many edge cases are hand-coded.

Put all organizational context directly into the prompt

Rejected because: The organization has too much context for the model context window, so retrieval is needed.

Rebuild internal access control

Rejected because: Governance already exists in source systems, so the platform should respect those permissions instead of recreating them.

A single flat hierarchy for very large agent counts

Rejected because: At 50 or hundreds of agents, the planner's routing space becomes too large; the speaker expects multiple hierarchy levels.

Patterns

rag-retrieval (context-engineering)

Numbers Station retrieves business-specific schema, metric, dashboard, and documentation context from a knowledge layer before asking the LLM to generate analytics answers.

tool-use-orchestration (agent-architecture)

The SQL and charting agents register execution and retrieval tools, then let the LLM decide which tool call or final response comes next.

multi-agent-orchestration (agent-architecture)

A planner coordinates SQL, charting, report-search, messaging, and other specialized agents under one chat interface.

agent-memory (state-management)

Shared conversation history lets specialized agents hand off context and results across an analytics workflow.

guardrails-io-filtering (safety-and-reliability)

The system grounds answers in a knowledge layer and can admit uncertainty instead of fabricating unsupported analytics answers.

latency-optimization (production-optimization)

The framework skips unnecessary model calls for some routing decisions to keep user-facing flows smooth.

Tradeoffs

Use agent-directed control flow for SQL

Chose LLM tool-calling that decides whether to retrieve, execute, retry, or answer over A deterministic generate-then-execute pipeline.

Rationale: Agentic control flow can recover from compilation errors and wrong table names without hand-coding every edge case.

Ground SQL generation with retrieval

Chose RAG over indexed organizational knowledge over Relying on vanilla prompting and model guesses.

Rationale: RAG lets the agent use customer-specific definitions and avoids making up calculations.

Coordinate specialized agents through a planner

Chose A hierarchical planner plus specialized SQL, charting, report, and messaging agents over One monolithic agent or forcing the user to choose each tool manually.

Rationale: The planner keeps the user interface simple while still exposing many capabilities and enabling handoffs through shared history.

Respect source-system governance

Chose Permission pass-through from original tools over Rebuilding an internal governance system.

Rationale: Source systems already know the organization's access rules, and recreating them is outside the platform's scope.

Gotchas (what bit them)

Demo-quality SQL generation fails when business semantics are missing.

Root cause: The model cannot guess customer-specific metric definitions such as the active-customer filter from generic pretraining.

Resolution: Build and retrieve from a knowledge layer that stores the organization's actual metric and schema definitions.

Manual control flow breaks at the first unhandled SQL error.

Root cause: A fixed generate-then-execute pipeline has no adaptive step when compilation fails or a wrong table name is used.

Resolution: Let the agent read execution errors, revise the query, and retry through tool calling.

The essential production work is often data wrangling, not novel model logic.

Root cause: Accurate analytics requires parsing, ingesting, and normalizing metadata stored across many enterprise tools.

Resolution: Treat ingestion and metadata wrangling as first-class platform work, even if it is not flashy AI work.

A single planner hierarchy may not scale to very large agent catalogs.

Root cause: As agent count grows, the number of possible routes and paths grows beyond what a single flat planner should consider.

Resolution: Use multiple levels of hierarchies with smaller agent groups and skip model calls where routing can be optimized.

Enterprise permissioning is easy to underestimate.

Root cause: Agent performance and data access control are separate concerns, and enterprises care deeply about who can see which data.

Resolution: Pass through and respect source-system permissions rather than building a competing internal authorization model.

Evaluation

The talk describes evaluation mostly through customer-realistic benchmarks, production demos, daily usage confidence, and setup/onboarding feedback rather than a formal published metric suite. The speaker specifically contrasts customer benchmarks with easy public benchmarks and says knowledge-layer grounding plus retry mechanisms produced large improvements.

Metrics: Accuracy on real customer SQL benchmarks · Comparison against vanilla prompting · Query retry/correction behavior · User confidence in grounded results · Setup time from customer onboarding · Planner smoothness at current agent counts

Outcomes

• Numbers Station reports large improvements over vanilla prompting on realistic customer SQL benchmarks.
• Vouch is cited as an insurtech customer using the chat interface for self-service analytics.
• Setup time reportedly improved from a couple of weeks of manual customer work to a couple of hours of admin-user training.
• The current hierarchy works smoothly at around a dozen agents and up to roughly 10-15 agents.
• The team expects to need multiple hierarchy levels if the platform grows to 50 or hundreds of agents.
• The platform intentionally grounds answers in retrieved organizational knowledge and can say it does not know when evidence is missing.

The transcript includes a decade-long modern data stack context, a Vouch customer example, onboarding reduced from a couple of weeks to a couple of hours, current use with at most a dozen agents, and an explicit 50/hundreds-of-agents scalability threshold.