Production Deployment of Toqan Data Analyst Agent: From Prototype to Production Scale

toqan · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Natural-language data analysis over customer databases

Toqan needed to turn a demo-friendly data analyst agent into a production system that could translate English questions into SQL, execute them, and return reliable answers/visualizations for hundreds of partner users.

Constraints:

• LLM behavior was unpredictable under vague user questions and schema ambiguity.
• Generated SQL could reference nonexistent columns or loop on the same wrong answer.
• Production partners require database safety, limited credentials, and precision because business decisions use the results.
• Domain experts, not ML engineers, hold the schema/documentation knowledge needed for reliable analysis.

Architecture

The Toqan Data Analyst accepts a natural-language question, runs deterministic preprocessing to decide whether the question is answerable, retrieves relevant schema/documentation, generates SQL, validates referenced columns, executes queries, and returns tabular/visualized answers. Production hardening adds expert-maintained documentation pipelines, hard and soft limits against loops, smarter schema retrieval, SQL-executor self-correction for context-independent errors, custom model routing, limited database credentials, and case-by-case deployment security.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-question-interface["User question interface"] --> preprocessing-gate["Preprocessing gate"]
  preprocessing-gate["Preprocessing gate"] --> data-analyst-agent["Data analyst agent"]
  data-analyst-agent["Data analyst agent"] --> schema-retrieval-tool["Schema retrieval tool"]
  data-analyst-agent["Data analyst agent"] --> sql-generator["SQL generator"]
  data-analyst-agent["Data analyst agent"] --> hard-and-soft-limit-controller["Hard and soft limit controller"]
  schema-retrieval-tool["Schema retrieval tool"] --> expert-documentation-pipeline["Expert documentation pipeline"]
  sql-generator["SQL generator"] --> schema-validation-check["Schema validation check"]
  schema-validation-check["Schema validation check"] --> sql-executor["SQL executor"]
  sql-executor["SQL executor"] --> customer-database["Customer database"]
  sql-executor["SQL executor"] --> user-question-interface["User question interface"]
  model-routing-layer["Model routing layer"] --> data-analyst-agent["Data analyst agent"]
  model-routing-layer["Model routing layer"] --> sql-generator["SQL generator"]

Components

Data flow

A user asks a business question in English. A preprocessing step checks whether the question can be answered from available context and requests clarification for vague inputs. The agent retrieves concentrated schema documentation, generates SQL, validates columns before execution, sends safe queries to a SQL executor, lets the executor repair simple dialect errors where possible, and returns results with optional visualizations. Expert-facing UIs keep documentation optimized for agent consumption.

Why this architecture

Toqan’s central lesson is that AI-first does not mean AI-only. Predictable behavior is enforced with deterministic checks, domain knowledge is maintained by experts, limits keep agents from looping while preserving useful self-correction, and smarter tools reduce cognitive load on the main agent.

Models used: OpenAI models · GPT-4-mini · o1 · GPT-4 Turbo

Scale

• The production journey covered approximately six months of deployment learning.
  “The presentation centers on four key lessons learned over approximately six months of production deployment.”summary, stated
• The agent served hundreds of users across development partners including iFood, Gloovo, and OLX.
  “serving hundreds of users across development partners including iFood, Gloovo, and OLX.”summary, stated
• One failure case repeated the same incorrect answer 58-59 times.
  “Producing the same incorrect answer repeatedly (58-59 times in one example)”summary, stated
• The system was developed over about two years of custom framework investment.
  “They started development approximately two years ago when framework options were less mature”summary, stated

Alternatives considered (roads not taken)

Let the agent answer vague questions directly

Rejected because: The LLM produced confident but inconsistent answers when context was insufficient.

Wait for database errors after bad SQL

Rejected because: Pre-validating columns avoids wasted cycles and user-system load.

One-off domain knowledge transfer to ML engineers

Rejected because: ML engineers could not generalize across highly specific domain problems without expert-owned documentation loops.

Use existing agent frameworks as the core

Rejected because: Toqan had invested in a custom framework that scaled for its requests/users and preserved flexibility.

Patterns

tool-use-orchestration (agent architecture)

The analyst agent uses schema retrieval, SQL generation, validation, and execution tools rather than a single prompt.

safety-policy-enforcement (production control)

Deterministic preprocessing and schema validation prevent predictable bad actions before database execution.

human-in-the-loop-gate (domain operations)

Data experts directly maintain agent documentation through pipelines/UIs.

fallback-degradation (resilience)

Hard limits stop runaway loops while soft nudges preserve useful self-correction.

model-routing (model operations)

Different models are selected by task complexity and cost.

context-engineering (tool/context design)

Schema retrieval filters documentation to concentrate relevant context and reduce distraction.

Tradeoffs

Where to use AI

Chose AI plus deterministic workflows over LLM reasoning for every behavior.

Rationale: Predictable problems are better solved with hard-coded checks and functional validation.

Limit design

Chose Hard limits plus soft nudges over Only strict stop rules or only unconstrained creativity.

Rationale: Hard limits prevent runaway behavior while soft limits preserve creative problem solving.

SQL error repair

Chose Let SQL executor handle common context-independent errors over Returning every error to the main agent.

Rationale: Routine dialect fixes do not need full agent context and would clutter the main loop.

Framework strategy

Chose Custom agentic framework over Adopting existing frameworks wholesale.

Rationale: Existing frameworks were immature when Toqan started and the custom system already scaled for requests/users.

Gotchas (what bit them)

Demo behavior did not survive production.

Root cause: Agents looped, hallucinated schema, ignored stop commands, and produced inconsistent answers under real user behavior.

Resolution: Add deterministic gates, validation, expert curation, and hard/soft limits.

Schema hallucination created unsafe SQL.

Root cause: The LLM inferred columns that seemed plausible but did not exist.

Resolution: Validate generated SQL against documented schema before execution.

Expert knowledge could not be delegated once to ML engineers.

Root cause: Use-case-specific schema/documentation details varied too much.

Resolution: Give experts ongoing tools to update documentation directly.

Unexpected user behavior can trigger abuse or loops.

Root cause: Testing with stakeholders does not cover real users unfamiliar with development assumptions.

Resolution: Combine hard limits with liberal input acceptance and learn from boundary-pushing queries.

Evaluation

Toqan validates predictable failures functionally before execution, relies on expert-maintained documentation/test loops, and optimizes for precision because business users make decisions from outputs.

Metrics: SQL schema validity · tool-call counts · execution timeouts · precision · user adoption · query correctness

Outcomes

• Toqan achieved adoption across hundreds of users at development partners.
• The system evolved from failing demos to production usage through deterministic controls and expert loops.

The summary reports hundreds of users across iFood, Gloovo, and OLX, six months of deployment lessons, and one repeated-answer failure at 58-59 iterations.