Building a Production Text-to-SQL Assistant with Multi-Agent Architecture

linkedin · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise text-to-SQL assistant for data analytics

LinkedIn needed SQL Bot to help employees find data, write SQL, fix errors, and understand datasets across a massive enterprise warehouse where table metadata is incomplete, source-of-truth tables change, user questions carry implicit business context, and query correctness is hard to evaluate automatically.

Constraints:

• Data experts were spending time helping colleagues find data instead of doing strategic analysis
• LinkedIn had millions of warehouse tables and incomplete table/field descriptions
• The same natural-language question can require different datasets for different users or organizations
• Tables and fields are deprecated or replaced over time
• Simple questions should not receive overcomplicated queries
• Each dataset has access controls that must be respected

Architecture

SQL Bot is a multi-agent LangChain/LangGraph system integrated into DARWIN. It combines intent classification, embedding-based retrieval, personalized dataset filters, dataset certification, AI-augmented metadata, DataHub deprecation signals, a knowledge graph over users/table clusters/tables/fields, LLM rerankers for table and field selection, iterative plan-based query writing, validators, and a self-correction agent. DARWIN UI elements expose tables, queries, validation checks, feedback, guided flows, and Fix with AI.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  darwin-platform[DARWIN Platform] --> user-interface[User Interface]
  user-interface --> sql-bot[SQL Bot]
  sql-bot --> intent-classifier[Intent Classifier]
  intent-classifier --> table-retrieval[Table Retrieval]
  table-retrieval --> embedding-vector-store[Embedding Vector Store]
  table-retrieval --> personalization-layer[Personalization Layer]
  table-retrieval --> knowledge-graph[Knowledge Graph]
  embedding-vector-store --> datahub[DataHub]
  datahub --> knowledge-graph
  knowledge-graph --> llm-rerankers[LLM Rerankers]
  llm-rerankers --> query-writer[Query Writer]
  query-writer --> validators[Validators]
  validators --> self-correction-agent[Self Correction Agent]
  self-correction-agent --> table-retrieval
  access-control-layer[Access Control Layer] --> sql-bot
  personalization-layer --> access-control-layer

Components

Data flow

A user asks a question in DARWIN. SQL Bot classifies the intent, retrieves candidate tables through filtering and embedding-based retrieval, personalizes defaults using org chart and access-history ICA components, reranks the top tables and fields with LLMs using knowledge-graph context, writes a query iteratively from a plan, validates table/field existence and EXPLAIN syntax, and hands errors to a self-correction agent that can retrieve additional context before updating the query.

Why this architecture

LinkedIn treated text-to-SQL as enterprise retrieval, ranking, and correction rather than direct NL-to-query generation. The system narrows millions of tables through popularity, personalization, EBR, and knowledge-graph reranking; writes queries iteratively for complex questions; and uses validators plus self-correction to address syntax and schema errors. Integrating into DARWIN made the assistant appear where users already write and fix SQL.

Models used: LLMs · LLM rerankers · LLM-as-a-judge · embedding models

Scale

• LinkedIn's warehouse has millions of tables.
  “Another challenge is the sheer volume of tables—at LinkedIn, it’s in the millions—and the implicit context embedded in user questions. We can quickly narrow down the volume of tables to a few thousand by looking at access popularity.”article, stated
• The first retrieval stage narrows the table space to a few thousand tables.
  “We can quickly narrow down the volume of tables to a few thousand by looking at access popularity. However, addressing the implicit context is more subtle.”article, stated
• LLM reranking selects seven tables from the top twenty EBR results.
  “After retrieving the top 20 tables via EBR, we employ a LLM re-ranker to select the top 7 tables for query writing.”article, stated
• The benchmark covered more than 130 questions across 10 product areas.
  “We collaborated with domain experts across 10 product areas to define a set of over 130 benchmark questions. Each question includes a well-formulated question and ground truth answers.”article, stated
• DARWIN integration increased adoption 5-10x over the standalone prototype.
  “This integration increased adoption by 5-10x compared to our prototype chatbot application launched as a standalone app.”article, stated
• About 95% of surveyed users rated query accuracy Passes or above, and about 40% rated it Very Good or Excellent.
  “~95% rated SQL Bot’s query accuracy “Passes” or above, and ~40% rated the query accuracy “Very Good” or “Excellent”.”article, stated
• Fix with AI accounted for 80% of sessions.
  “One takeaway is that the “Fix with AI” feature was easy to develop but has very high usage—accounting for 80% of our sessions—frequently saving users time in debugging their queries. Identifying high-ROI pain points like this is a good place to start the text-to-SQL journey.”article, stated

Alternatives considered (roads not taken)

Standalone prototype chatbot

Rejected because: Users adopted SQL Bot far more when it was integrated into DARWIN where they already write and execute queries.

Always answer with SQL

Rejected because: Users wanted table discovery, dataset questions, reference queries, syntax help, and follow-up conversations too.

Use access popularity alone for table retrieval

Rejected because: Popularity narrows the space, but user questions contain implicit business context that needs personalization.

Run SQL queries and compare outputs for evaluation

Rejected because: LinkedIn used human and LLM-as-judge evaluation because output comparison required data access and gave less diagnostic insight.

Patterns

multi-agent-orchestration (agent-architecture)

SQL Bot delegates intent handling, table/field retrieval, query writing, follow-up chat, validation, and self-correction across agentic flows.

rag-retrieval (text-to-sql)

SQL Bot retrieves schemas, example queries, metadata, domain knowledge, and table context before query generation.

retrieval-ranking (enterprise-search)

LinkedIn narrows candidates by popularity and EBR, then reranks tables and fields with LLMs over knowledge-graph context.

agent-memory (query-planning)

Solutions to previous query-writing subtasks are stored in chatbot state and provided to later steps.

fallback-degradation (self-correction)

Validators feed syntax and schema errors into a self-correction agent that can retrieve more context before rewriting SQL.

llm-as-judge (evaluation)

LinkedIn combines human evaluation with LLM-as-a-judge scoring against schemas and ground truths.

safety-policy-enforcement (data-access)

SQL Bot checks whether users belong to groups with access and supplies group credential code when appropriate.

Tradeoffs

Integrate into DARWIN

Chose same browser window as query writing/execution over standalone chatbot application.

Rationale: The integrated placement dramatically increased adoption and surfaced SQL Bot at moments of query-writing need.

Use LLM-as-judge plus human evaluation

Chose rubric scoring against question, schemas, and ground truths over executing SQL and comparing result outputs.

Rationale: Judging queries avoided data-access requirements and gave richer diagnostics on tables, joins, filters, aggregations, efficiency, and complexity.

Use guided step-by-step query writing as an option

Chose guided experience with user feedback at each step over only opaque one-shot query generation.

Rationale: Users can inspect table/query choices, understand the output, and correct the bot during query construction.

Let users customize datasets and instructions

Chose self-serve product-area configuration over platform-team-only tuning.

Rationale: LinkedIn wanted domain experts to improve SQL Bot performance without waiting on the platform team.

Gotchas (what bit them)

Enterprise table metadata is often incomplete.

Root cause: Many tables and fields lack good descriptions, making semantic retrieval fragile.

Resolution: LinkedIn ran dataset certification and augmented descriptions with AI-generated annotations from docs and Slack.

Popular-table filtering does not resolve implicit context.

Root cause: The same metric phrase means different things across product areas.

Resolution: SQL Bot infers default datasets from org structure and ICA components over access history.

Source-of-truth tables drift over time.

Root cause: Tables and fields are deprecated, added, or replaced as warehouse systems evolve.

Resolution: LinkedIn automatically ingests popular queried tables and offboards deprecated datasets/fields using DataHub signals.

Iterative query writing can overcomplicate simple questions.

Root cause: Plan-and-solve decomposition helps complex questions but may add unnecessary steps for simple ones.

Resolution: LinkedIn instructs the planner to minimize the number of steps it creates.

Schema and syntax validation need information unavailable to the query writer.

Root cause: The generator may not know current table/field existence or syntax errors from the target engine.

Resolution: Validators check existence and EXPLAIN, then self-correction retrieves extra tables/fields as needed.

Evaluation

LinkedIn built a benchmark with domain experts, measured table/field recall, hallucination rate, syntax correctness, and latency, then added human plus LLM-as-judge evaluation for deeper query-accuracy assessment.

Metrics: table recall · field recall · table hallucination rate · field hallucination rate · syntax correctness · response latency · query accuracy survey rating · LLM-as-judge rubric score · adoption lift · Fix with AI session share

Outcomes

• SQL Bot became integrated into DARWIN and used by hundreds of LinkedIn employees across business verticals.
• DARWIN integration increased adoption 5-10x compared with the standalone prototype.
• A recent survey reported about 95% of users rated query accuracy Passes or above.
• Fix with AI accounted for 80% of sessions.
• The system supports query generation, query fixing, table finding, dataset questions, syntax help, guided flows, and rich output inspection.

LinkedIn reports millions of tables, candidate narrowing to a few thousand by access popularity, reranking from top 20 to top 7 tables, a benchmark of over 130 questions across 10 product areas, hundreds of employees using the tool, 5-10x adoption lift after DARWIN integration, and 80% of sessions using Fix with AI.