Building QueryAnswerBird: An AI Data Analyst with Text-to-SQL and RAG

delivery_hero · E-commerce · 2024 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data literacy and Text-to-SQL analytics

Woowa Brothers needed an internal AI data analyst that could help employees generate, understand, validate, and explore SQL without requiring every user to master company-specific data schemas and business terminology.

Constraints:

• Most employees used data, but many struggled with SQL and translating business logic into queries.
• GPT-4 needed domain-specific table metadata, business terminology, and data policy context.
• The assistant had to fit employee workflows through Slack rather than a separate web app.
• Query generation required syntax validation, interpretation, and table/column discovery, not only SQL text.
• Production operation required A/B testing, monitoring, load balancing, caching, feedback, and CI/CD.
• The system depended on ongoing maintenance of few-shot examples and business terminology by domain experts.

Architecture

QueryAnswerBird is a Slack-integrated AI data analyst built with GPT-4, LangChain, RAG, vector stores, enriched metadata, business terminology, few-shot SQL examples, a router-supervisor multi-chain architecture, and LLMOps infrastructure. The system retrieves domain context from unstructured data, table metadata, data catalogs, business glossaries, and analyst-maintained question-query examples, then routes each question to specialized chains for query generation, interpretation, syntax validation, table interpretation, or utilization guidance. Production operations include internal leaderboards, hundreds of A/B tests, monitoring dashboards, API load balancing, GPT caching, feedback loops, and CI/CD deployment.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-user[Slack user] --> slack-integration[Slack integration]
  slack-integration --> router-supervisor-chain[Router Supervisor chain]
  router-supervisor-chain --> query-generation-chain[Query generation chain]
  router-supervisor-chain --> query-interpretation-chain[Query interpretation chain]
  router-supervisor-chain --> syntax-validation-chain[Syntax validation chain]
  router-supervisor-chain --> table-guide-chains[Table guide chains]
  query-generation-chain --> retriever[Retriever]
  query-interpretation-chain --> retriever
  table-guide-chains --> retriever
  retriever --> vector-stores[Vector stores]
  vector-stores --> data-catalog-apis[Data catalog APIs]
  retriever --> business-terminology-glossary[Business terminology glossary]
  business-terminology-glossary --> data-governance-organization[Data governance organization]
  retriever --> few-shot-sql-examples[Few-shot SQL examples]
  few-shot-sql-examples --> data-analysts[Data analysts]
  query-generation-chain --> gpt-4[GPT-4]
  query-interpretation-chain --> gpt-4
  syntax-validation-chain --> gpt-4
  table-guide-chains --> gpt-4
  gpt-4 --> gpt-cache[GPT cache]
  slack-integration --> feedback-loop[Feedback loop]
  feedback-loop --> gpt-cache
  experiment-leaderboard[Experiment leaderboard] --> ci-cd-pipeline[CI/CD pipeline]
  experiment-leaderboard --> monitoring-dashboards[Monitoring dashboards]
  ci-cd-pipeline --> monitoring-dashboards

Components

Data flow

An employee asks a question in Slack. The Router Supervisor classifies the question and dispatches it to the right specialized chain. Chain-specific retrievers pull table metadata, enriched DDL, business terminology, and few-shot SQL examples from vector stores and curated knowledge bases. GPT-4 then generates or interprets SQL, validates syntax, or explains table usage. User feedback flows back into the cache and knowledge loop, while A/B testing, leaderboards, monitoring, load balancing, and CI/CD govern which prompts, retrievers, and chains reach production.

Why this architecture

The architecture addresses the gap between generic GPT-4 SQL ability and company-specific data reality. RAG, enriched metadata, business terminology, and few-shot examples provide domain grounding. Multi-chain routing avoids forcing one prompt to solve all data-literacy tasks. Slack integration lowers adoption friction, while testing, leaderboards, caching, monitoring, load balancing, and CI/CD make the system operable as a production LLM service.

Models used: GPT-4 · Microsoft Azure OpenAI GPT-3.5 API in hackathon prototype · Embedding models for vectorization · LangChain chains · Router Supervisor chain

Scale

• A survey found 95% of employees used data for work.
  “while 95% of employees used data in their work, more than half faced challenges with SQL.”summary, stated
• The initial concept won first place at a 2023 internal hackathon.
  “The project originated from an internal hackathon in 2023 focused on generative AI, where the initial concept won first place.”summary, stated
• A task force started in January 2024 and developed the production system over six months.
  “the company established the Langineer Task Force in January 2024 to develop a production-grade system over six months.”summary, stated
• The team conducted over 500 A/B tests.
  “conducting over 500 tests to optimize performance.”summary, stated
• Query generation and interpretation features were developed within two months.
  “The team successfully developed query generation and interpretation features within two months through their architecture and Text-to-SQL implementation.”summary, stated
• The service typically responds within 30 seconds to 1 minute.
  “The system typically provides responses within 30 seconds to 1 minute, offering high-quality queries that employees can reference for their work.”summary, stated
• Development used three roadmap steps with two-week sprints.
  “The task force operated using short sprint cycles, dividing their development roadmap into three steps with two-week sprints for each.”summary, stated

Alternatives considered (roads not taken)

Rely on generic GPT-4 SQL generation

Rejected because: Generic GPT-4 lacks company-specific metadata, business terminology, and data-policy context.

Use the simple GPT-3.5 hackathon prompt architecture

Rejected because: The team redesigned the architecture to meet systemization, efficiency, accessibility, and automation goals.

Use a separate web application

Rejected because: Slack reduced adoption friction by placing QAB in employees' existing workflow.

Evaluate only with public leaderboards

Rejected because: Public benchmarks could not capture company-specific business priorities and domain problems.

Use static schema metadata only

Rejected because: The team enriched metadata with purpose, characteristics, columns, key values, keywords, services, and example questions.

Patterns

rag-retrieval (context-engineering)

QAB retrieves table metadata, DDL, business terminology, policies, and few-shot examples to ground SQL generation in company context.

query-rewriting (context-engineering)

The system refines ambiguous or short user questions before retrieval and generation.

tool-use-orchestration (agent-architecture)

The Router Supervisor dispatches questions to specialized chains for generation, interpretation, validation, and table guidance.

eval-driven-development (quality)

Internal datasets, metrics, leaderboards, and over 500 A/B tests governed prompt, retriever, and chain deployment.

semantic-caching (operations)

GPT caching reuses common or similar query responses, reducing latency and API cost while incorporating user feedback.

observability-tracing (operations)

Monitoring dashboards track system performance, error rates, response times, and operational health.

structured-output-contract (output-control)

QAB produces SQL plus validation/explanation outputs and uses structured prompt categories for refinement and generation.

Tradeoffs

Use Slack as the interface

Chose Slack-integrated assistant over Separate web application.

Rationale: Employees could ask questions in their existing workflow without switching tools.

Use RAG and enriched metadata

Chose Domain-grounded retrieval over internal metadata, policies, terminology, and examples over Pure foundation-model prompting.

Rationale: Company-specific SQL requires context that the base model does not know.

Use specialized chains

Chose Router Supervisor and multiple task-specific chains over A single query-generation chain.

Rationale: The product needed data-literacy features beyond generating SQL, including interpretation, validation, and table guides.

Use internal leaderboards and A/B testing

Chose Quantitative competition among prompts, retrievers, chains, and datasets over Ad-hoc prompt deployment.

Rationale: The team wanted systematic performance improvement and approval before production deployment.

Cache common GPT responses

Chose GPT caching integrated with feedback over Calling the LLM from scratch for every common or similar query.

Rationale: Caching can reduce latency and API costs while improving consistency for repeated questions.

Gotchas (what bit them)

Foundation models lacked company-specific data semantics.

Root cause: GPT-4 can generate SQL generally but does not know internal table policies, business terminology, or data marts.

Resolution: Ground generation through RAG over enriched metadata, business glossaries, and few-shot examples.

Business terminology varies across services and organizations.

Root cause: Terms used differently across teams can cause miscommunication and incorrect SQL interpretation.

Resolution: Use a dedicated Text-to-SQL business terminology glossary maintained through data governance.

Public Text-to-SQL metrics did not fully fit internal business needs.

Root cause: Public leaderboards and frameworks struggle with domain-specific priorities and company-specific data problems.

Resolution: Build custom internal evaluation datasets, metrics, and leaderboards.

Few-shot examples and terminology require ongoing human maintenance.

Root cause: Business logic and data extraction criteria change over time across domains.

Resolution: Have domain-specialist data analysts maintain and update examples and glossary data.

The case study leaves production failure rates under-specified.

Root cause: The source is promotional and does not quantify detailed error rates or production performance against evaluation datasets.

Resolution: Treat reported outcomes as directional and rely on the described evaluation/monitoring machinery rather than unsupported accuracy claims.

Evaluation

The team moved beyond public benchmarks by building internal Text-to-SQL datasets, custom metrics, automated testing, dozens of detailed metrics, internal leaderboards, and over 500 A/B tests. High-performing configurations were approved in weekly sync-ups before production deployment, while LangServe Playground supported rapid prompt and chain checks.

Metrics: Custom Text-to-SQL performance metrics · Query syntax understanding · Query execution-result accuracy · Complex-domain understanding · Actual user questions · Dozens of detailed metrics · Internal leaderboard rank · Over 500 A/B tests · System performance · Error rates · Response times

Outcomes

• QAB moved from a 2023 hackathon winner to a production-grade Slack-integrated AI data analyst.
• The team developed query generation and interpretation features within two months.
• Employees used QAB to better understand unfamiliar service domains and work tasks.
• Feedback showed remaining room to improve business-logic understanding and question interpretation.
• Growing usage revealed demand for broader data discovery beyond query generation.

The source reports 95% employee data use, more than half struggling with SQL, a 2023 first-place hackathon result, a January 2024 task force, six months of production development, over 500 A/B tests, typical 30-60 second responses, two-month implementation of query generation/interpretation features, and three two-week sprint steps.