Building QueryAnswerBird: An LLM-Powered AI Data Analyst with RAG and Text-to-SQL

delivery_hero · E-commerce · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise Text-to-SQL and data discovery

Woowa Brothers needed to raise employee data literacy by helping employees generate, interpret, validate, and discover SQL/data information despite company-specific business terminology, data policies, and metadata complexity.

Constraints:

• About 95% of surveyed employees used data, but more than half struggled with SQL and reliable extraction.
• Generic LLMs could answer general questions but not company-specific data questions without internal context.
• Generated SQL quality suffered from missing domain knowledge, weak retrieval, and hallucination.
• The assistant needed to be accessible in Slack and available 24/7 without dedicated data personnel.
• The production service required AB testing, leaderboards, load balancing, caching, feedback, monitoring, and CI/CD.
• Few-shot SQL examples and business terminology needed ongoing management as business logic evolved.

Architecture

QAB is a Slack-based AI data analyst built on GPT-4o/Azure OpenAI, RAG, LangChain, vector stores, multi-chain routing, and LLMOps. It automatically collects business terminology, table metadata, and extraction code into VectorDBs, applies data-area indexing, routes questions through a Router Supervisor to specialized chains, retrieves the right metadata/DDL/few-shot examples, and generates or explains SQL. Production operations include AB testing with leaderboards, API load balancing, GPT caching, user feedback loops, monitoring dashboards, and CI/CD.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-employee[Slack employee] --> slack-qab-interface[Slack QAB interface]
  slack-qab-interface --> router-supervisor-chain[Router Supervisor chain]
  router-supervisor-chain --> query-generation-chain[Query generation chain]
  router-supervisor-chain --> query-interpretation-chain[Query interpretation chain]
  router-supervisor-chain --> syntax-validation-chain[Syntax validation chain]
  router-supervisor-chain --> data-discovery-guide-chains[Data discovery guide chains]
  query-generation-chain --> retriever[Retriever]
  query-interpretation-chain --> retriever
  data-discovery-guide-chains --> retriever
  retriever --> vectordbs[VectorDBs]
  vectordbs --> unstructured-data-pipeline[Unstructured data pipeline]
  unstructured-data-pipeline --> internal-data-catalog-api[Internal data catalog API]
  retriever --> business-terminology-glossary[Business terminology glossary]
  business-terminology-glossary --> data-governance-organization[Data governance organization]
  retriever --> few-shot-sql-dataset[Few-shot SQL dataset]
  few-shot-sql-dataset --> domain-data-analysts[Domain data analysts]
  query-generation-chain --> gpt-4o[GPT-4o]
  query-interpretation-chain --> gpt-4o
  syntax-validation-chain --> gpt-4o
  data-discovery-guide-chains --> gpt-4o
  gpt-4o --> gpt-cache[GPT cache]
  slack-qab-interface --> feedback-loop[Feedback loop]
  feedback-loop --> gpt-cache
  ab-testing-leaderboard[AB testing leaderboard] --> ci-cd-deployment[CI/CD deployment]
  ab-testing-leaderboard --> monitoring-dashboards[Monitoring dashboards]
  ci-cd-deployment --> monitoring-dashboards

Components

Data flow

Employees ask questions in Slack. The Router Supervisor identifies question purpose and sends each request to the appropriate chain. The retriever uses chain-specific search over VectorDBs, business terminology, table metadata, DDL, and few-shot SQL examples. GPT-4o then refines the question, reasons/searches through ReAct-style prompts, and generates, validates, interprets, or explains data artifacts. Feedback from Slack updates the GPT cache, while AB tests and leaderboards select best-performing prompts/chains for CI/CD deployment with monitoring and operational safeguards.

Why this architecture

The team redesigned the hackathon prototype because a simple prompt against GPT-3.5/GPT-4 was not enough for company-specific Text-to-SQL. RAG provides internal knowledge; vector stores make domain data searchable; multi-chain routing supports several literacy tasks; Slack provides accessibility; and LLMOps infrastructure manages response stability, speed, error handling, experiments, and deployment.

Models used: OpenAI GPT-4o via Microsoft Azure OpenAI · Microsoft Azure OpenAI GPT-3.5 API in the hackathon prototype · Embedding models · LangChain · Router Supervisor chain

Scale

• About 95% of survey respondents used data for work.
  “The survey results showed that about 95% of the respondents were using data for their work.”article, stated
• More than half of respondents said SQL was difficult because of learning time, business logic, extraction conditions, or reliability concerns.
  “more than half of the respondents said that even though they wanted to use SQL, they didn’t have enough time to learn it or had difficulty generating SQL queries”article, stated
• QAB won first place in the 2023 Woowa Hackathon.
  “At the hackathon, it was presented as a service that automatically extracts and visualizes data without the need for coding and won first place.”article, stated
• The Langineer Task Force began full-scale development in January 2024 and spent six months developing QAB further.
  “the Langineer Task Force was established for full-scale development in January 2024. For the last six months, the task force has developed QAB even further”article, stated
• The task force roadmap used three steps with two-week sprints for each step.
  “We divided the development roadmap into three steps, running two-week sprints for each step.”article, stated
• The team conducted more than 500 A/B tests.
  “We built an internal leaderboard and conducted more than 500 A/B tests to bring our individual ideas to life.”article, stated
• The query generation and interpretation feature was developed in two months and responds within 30 seconds to 1 minute.
  “We were able to develop a query generation and interpretation feature in just two months through QAB’s basic architecture and Text-to-SQL feature.”article, stated

  “This feature provides users with responses within 30 seconds to 1 minute, offering high-quality queries that can be referenced for work.”article, stated

Alternatives considered (roads not taken)

Keep the hackathon GPT-3.5 prompt prototype

Rejected because: The initial version could not satisfy systemization, efficiency, accessibility, and automation requirements.

Generate SQL using only GPT-4

Rejected because: The resulting queries lacked domain/data-policy knowledge, included unnecessary data, and suffered from inconsistent hallucination-prone quality.

Use public Text-to-SQL metrics and leaderboards as-is

Rejected because: They did not adequately cover company-specific domain issues or business-priority metric changes.

Use only basic table schemas

Rejected because: The team needed richer metadata with table purpose, characteristics, column descriptions, values, keywords, services, and questions.

Stop at query generation

Rejected because: User questions increasingly involved data discovery, so QAB needed broader data-literacy features.

Patterns

rag-retrieval (context-engineering)

QAB searches internal data, metadata, terminology, DDL, and few-shot examples so GPT-4o can answer company-specific SQL questions.

query-rewriting (context-engineering)

Ambiguous, short, or unclear user questions are refined before retrieval and SQL generation.

tool-use-orchestration (agent-architecture)

The Router Supervisor selects among specialized chains, and ReAct prompts combine reasoning with retrieval/tool actions.

eval-driven-development (quality)

The team built internal evaluation data, dozens of metrics, leaderboards, weekly approvals, and more than 500 A/B tests.

semantic-caching (operations)

GPT caching improves response speed and standardizes useful answers based on satisfaction feedback.

observability-tracing (operations)

Monitoring dashboards are part of the LLMOps layer for production response stability, speed, and error handling.

structured-output-contract (output-control)

Generated answers include SQL validation information so users know whether the query executes or contains errors.

Tradeoffs

Redesign beyond the hackathon prototype

Chose A production architecture with RAG, multi-chain routing, Slack, and LLMOps over Simple GPT-3.5 prompts from the hackathon service.

Rationale: The hackathon architecture could not meet the four product pillars.

Use Slack instead of a web app

Chose Slack-based interface over Web-based method.

Rationale: Slack made the assistant easy and quick to access inside employees' existing messaging workflow.

Create custom internal metrics

Chose Internal Text-to-SQL evaluation metrics and data over Relying only on Spider, BIRD, or RAGAS-style public metrics.

Rationale: The team needed metrics that could track domain-specific issues and business priorities.

Use leaderboard-gated deployment

Chose Weekly approval of best-performing test results before production over Manual or ad-hoc prompt/configuration deployment.

Rationale: The leaderboard gave the team a repeatable way to improve service performance while encouraging participation.

Maintain human-curated few-shot examples

Chose Domain analyst-managed SQL examples over Static or purely automatic example collection.

Rationale: Response quality depends on high-quality examples that track changing extraction criteria and business domains.

Gotchas (what bit them)

GPT-only SQL looked plausible but was not work-ready.

Root cause: The generated queries lacked domain and data-policy knowledge, retrieval quality, and hallucination control.

Resolution: Use domain-grounded RAG, enriched metadata, search algorithms, prompt engineering, and testing/evaluation.

Company-specific business terms can mislead retrieval and question refinement.

Root cause: Similar but irrelevant terms may cause the LLM to generate incorrect reformulated questions.

Resolution: Use business-term extraction and a Text-to-SQL glossary managed through data governance.

Few-shot quality becomes an operational dependency.

Root cause: Example queries directly affect response quality and business extraction criteria change over time.

Resolution: Collect high-quality analyst queries and have domain-specialized data analysts manage the examples.

Public benchmarks could not encode internal business priorities.

Root cause: Domain-specific issues and key metrics change according to business situation.

Resolution: Build custom evaluation metrics/data while benchmarking public leaderboards.

The service still had business-logic and question-understanding accuracy gaps.

Root cause: Employees' questions and domains exposed limitations beyond initial query generation.

Resolution: Continue improvement using feedback, question histories, and broader data discovery features.

Evaluation

The team built internal Text-to-SQL metrics and evaluation data, ran step-by-step tests from syntax understanding to execution accuracy with complex domain knowledge, built an automated evaluation system, measured dozens of detailed metrics, conducted more than 500 A/B tests, and promoted the best leaderboard results through weekly sync-ups into production.

Metrics: Internal Text-to-SQL performance · Query syntax understanding · Query execution accuracy · Complex-domain understanding · Actual user questions · Dozens of detailed metrics · A/B test leaderboard rank · Response stability · Response speed · Error handling · 30 seconds to 1 minute response time

Outcomes

• QAB evolved from a first-place 2023 hackathon project into a production-grade internal AI data analyst.
• The team built query generation and interpretation in two months.
• Employees reported that QAB helped them understand work in new or different service domains.
• QAB expanded from Text-to-SQL toward broader data discovery after observing user questions.
• The article acknowledges remaining accuracy work around business logic and question understanding.

Concrete scale includes about 95% employee data use, more than half struggling with SQL, January 2024 task-force formation, six months of development, three roadmap steps with two-week sprints, more than 500 A/B tests, two months to query generation/interpretation, 30-second-to-1-minute responses, and expanding test participants/target organizations.