AI Data Analyst with Multi-Stage LLM Architecture for Enterprise Data Discovery

delivery_hero · E-commerce · 2024 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce

Delivery Hero/Woowa Brothers needed employees with different data expertise levels to discover and understand internal structured tables and unstructured log data.

Constraints:

questions varied beyond simple Text-to-SQL
internal terminology needed interpretation
irrelevant general questions needed filtering
users needed guidance without reading documentation first

Architecture

QueryAnswerBird is a summary-only v2 upgrade, but the approved summary supports a clear multi-stage architecture. A Slack-facing assistant sends questions into a Router Supervisor question-understanding stage, augments domain-specific terms from a vector store, routes accepted questions into specialized single-turn or multi-turn chains, and connects those chains to Data Catalog and Log Checker capabilities for table, column, query, and log guidance. Low-quality questions receive refinement prompts instead of being forced through expensive retrieval or Text-to-SQL paths.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-integration["Slack integration"]
  router-supervisor-chain["Router Supervisor chain"]
  vector-store-for-internal-terminology["vector store for internal terminology"]
  single-turn-and-multi-turn-chains["single-turn and multi-turn chains"]
  data-catalog["Data Catalog"]
  log-checker["Log Checker"]
  guided-refinement-prompts["guided refinement prompts"]
  slack-integration --> router-supervisor-chain
  router-supervisor-chain --> vector-store-for-internal-terminology
  router-supervisor-chain --> single-turn-and-multi-turn-chains
  single-turn-and-multi-turn-chains --> data-catalog
  single-turn-and-multi-turn-chains --> log-checker
  router-supervisor-chain --> guided-refinement-prompts
  guided-refinement-prompts --> slack-integration

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Slack integration`	Employee-facing entry point embedded in an existing communication workflow.	`Router Supervisor chain`, `single-turn and multi-turn chains`
`Router Supervisor chain`	Question-understanding stage that scores, classifies, and routes user questions.	`Slack integration`, `vector store for internal terminology`, `single-turn and multi-turn chains`
`vector store for internal terminology`	Augments specialized or abstract questions with internal terminology before classification/routing.	`Router Supervisor chain`
`single-turn and multi-turn chains`	Conversation patterns optimized for different data-discovery use cases.	`Router Supervisor chain`, `Data Catalog`, `Log Checker`
`Data Catalog`	Authoritative metadata/data-discovery system used for table and column guidance.	`single-turn and multi-turn chains`
`Log Checker`	Existing tool integrated for log-data utilization and contextual answers.	`single-turn and multi-turn chains`
`guided refinement prompts`	Automated prompts/examples for low-quality questions that need more specificity.	`Router Supervisor chain`, `Slack integration`

Data flow

A user asks a data-discovery question in Slack. The Router Supervisor scores and classifies the question, uses a terminology vector store to make internal language interpretable, rejects/refines weak questions when needed, and routes valid questions to the appropriate single-turn or multi-turn chain. Those chains call Data Catalog and Log Checker-style tools to return contextualized answers.

4 supporting quotes

“The solution involved building a multi-layered architecture with question understanding (Router Supervisor) and information acquisition stages, implementing various features including query/table explanation, syntax verification, table/column guidance, and log data utilization.”summary, stated

“Questions are augmented with internal terminology from a vector store during this evaluation process, converting abstract or specialized questions into more interpretable forms.”summary, stated

“the Router Supervisor intelligently classifies questions and routes them to appropriate processing chains.”summary, stated

“ultimately creating a comprehensive data discovery platform that integrates with existing tools like Data Catalog and Log Checker to provide contextualized answers and improve organizational productivity.”summary, implied

Why this architecture

The architecture separates question understanding from information acquisition because employee questions vary in specificity, relevance, and domain language. Routing through a supervisor and terminology augmentation reduces hallucination risk, avoids sending non-data questions to expensive processing, and keeps authoritative metadata/log systems responsible for the underlying facts.

4 supporting quotes

“This separation of concerns is architecturally sound from an LLMOps perspective, as it allows for independent optimization of each component while minimizing hallucination risks through staged processing.”summary, stated

“Rather than attempting to process all questions through a single chain—which would be prone to errors and difficult to maintain—the Router Supervisor intelligently classifies questions and routes them to appropriate processing chains.”summary, stated

“Questions failing to meet quality thresholds receive automated prompts for refinement with examples, creating a guided experience that improves over time as users learn to formulate better queries.”summary, stated

“By embedding the system in a communication platform employees already use daily, adoption friction is minimized.”summary, stated

Scale

The system was iterated through two beta rounds with data analysts, engineers, and product managers.
“Through two rounds of beta testing with data analysts, engineers, and product managers, the team iteratively refined the system to handle diverse question types beyond simple Text-to-SQL”summary, stated
More than 10% of observed questions were unrelated to data, justifying automatic classification.
“The team found that over 10 percent of questions were unrelated to data entirely—a significant finding that justified implementing automatic classification to filter out weather inquiries, greetings, and other general conversations before they reach expensive Text-to-SQL processing.”summary, stated
The platform targets both structured tables and unstructured log data.
“employees with varying levels of data expertise struggled to discover, understand, and utilize the company's vast internal data resources, including structured tables and unstructured log data.”summary, stated

Alternatives considered (roads not taken)

Single-chain processing for every question

Rejected because: A single chain would be error-prone and hard to maintain across diverse question types.

Standalone data-discovery UI outside daily workflows

Rejected because: Slack integration reduces adoption friction by meeting employees where they already work.

“Users don't need to learn a new interface or remember to check another application; QAB becomes available wherever they're already working.”summary, implied

Patterns

`multi-agent-orchestration` (orchestration)

The Router Supervisor routes different question types to appropriate processing chains rather than forcing all requests through one chain.

`rag-retrieval` (retrieval)

The system augments questions with internal terminology from a vector store so domain-specific questions become interpretable.

“Questions are augmented with internal terminology from a vector store during this evaluation process, converting abstract or specialized questions into more interpretable forms.”summary, stated

`confidence-thresholding` (quality-control)

Low-quality questions are not processed directly; users receive refinement prompts and examples.

Gotchas (what bit them)

Combining many data functions increases hallucination risk.

Root cause: Feature integration complexity grows as the system connects question routing, table guidance, log data, and conversation state.

Resolution: The team kept multi-turn conversations under development and called out hallucination risk as an LLMOps challenge.

“The acknowledged risk of hallucination when connecting various functions highlights a key LLMOps challenge: feature integration complexity grows non-linearly as capabilities are combined.”summary, stated

Evaluation

Two rounds of beta testing with analysts, engineers, and product managers shaped the data-discovery scope.

Metrics: over 10 percent unrelated-question rate

“The team conducted two rounds of beta testing—first with data analysts and engineers, then with product managers—to identify gaps between their initial Text-to-SQL implementation and real-world needs.”summary, stated

Outcomes

Beta testing showed users needed broader data discovery, not only query generation.

The source is blocked, so this extraction is summary-only.

“This iterative approach revealed that users required not just query generation but comprehensive data discovery capabilities including query explanation, table exploration, and log data navigation.”summary, stated