Text-to-SQL AI Agent for Democratizing Data Access in Slack

salesforce · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise self-service analytics and text-to-SQL

Salesforce needed to reduce the weekly burden on engineers and data scientists who wrote custom SQL for non-technical users, while giving Slack users reliable natural-language access to internal data.

Constraints:

Architecture

Salesforce built Horizon Agent, a Slack-based text-to-SQL agent using LLMs and RAG. The agent retrieves schema and business context, generates multiple SQL candidates, uses consensus to handle non-determinism, executes queries with Trino against an Iceberg data lake, and posts SQL, results, and explanations back to Slack.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack_interface["Slack interface"] --> horizon_agent["Horizon Agent"]
  horizon_agent --> rag_knowledge_base["RAG knowledge base"]
  rag_knowledge_base --> sql_generation_model["SQL generation model"]
  sql_generation_model --> consensus_selector["Consensus selector"]
  consensus_selector --> trino_execution_layer["Trino execution layer"]
  trino_execution_layer --> iceberg_data_lake["Iceberg data lake"]
  horizon_agent --> slack_interface
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Slack interface User-facing channel for questions, clarifications, explanations, and results. Horizon Agent
Horizon Agent Text-to-SQL agent that orchestrates retrieval, SQL generation, consensus, execution, and explanation. Slack interface, RAG knowledge base, SQL generation model, Trino execution layer
RAG knowledge base Retrieves schema, business context, and examples for the user's data question. Horizon Agent, SQL generation model
SQL generation model Generates SQL candidates and explanations. RAG knowledge base, Consensus selector
Consensus selector Mitigates non-determinism by comparing multiple SQL generations. SQL generation model, Trino execution layer
Trino execution layer Executes generated SQL against Salesforce's Iceberg data lake. Consensus selector, Iceberg data lake
Iceberg data lake Enterprise data source queried by Horizon. Trino execution layer

Data flow

A user asks a natural-language question in Slack. Horizon retrieves relevant context, generates SQL candidates, selects or validates a consensus answer, executes SQL through Trino over the Iceberg data lake, and returns the query, answer, and explanation in Slack.

3 supporting quotes
“The solution combines Large Language Models with Retrieval-Augmented Generation (RAG) to allow users to ask natural language questions in Slack and receive SQL queries, answers, and explanations within seconds.”summary, stated
“the application executes the SQL using Trino, retrieves data from Salesforce's Iceberg data lake, and posts results back to Slack with an AI-generated analysis”summary, stated
“Instead of giving the LLM a single opportunity to generate SQL, the system requests ten different generat”summary, stated

Why this architecture

Salesforce used Slack plus RAG and consensus generation because users already worked in Slack, the model needed schema/business context, and one-shot SQL generation was too nondeterministic for production analytics.

3 supporting quotes
“The choice of Slack as the interface was strategic: users already lived in this environment for collaboration”summary, stated
“Even with a perfect knowledge base, asking the same question ten times might yield eight correct SQL queries and two incorrect ones.”summary, stated
“This inconsistency is unacceptable in a production system where users need reliable answers.”summary, stated

Models used: Large Language Models · RAG

Scale

Alternatives considered (roads not taken)

Manual SQL by engineers and data scientists

Rejected because: It created bottlenecks and consumed dozens of hours weekly.

“address a data access gap where engineers and data scientists spent dozens of hours weekly writing custom SQL queries for non-technical users.”summary, stated

Single SQL generation

Rejected because: LLM non-determinism meant repeated generations could include incorrect queries.

“asking the same question ten times might yield eight correct SQL queries and two incorrect ones.”summary, stated

Opaque failure responses

Rejected because: They undermined trust and prevented learning from unsupported requests.

“Initial versions of Horizon Agent were opaque, simply responding "I don't know how to answer that" when queries fell outside its capabilities.”summary, stated

Patterns

rag-retrieval (context)

RAG retrieves schema and business context to ground text-to-SQL generation.

“The solution combines Large Language Models with Retrieval-Augmented Generation (RAG) to allow users to ask natural language questions in Slack and receive SQL queries, answers, and explanations within seconds.”summary, stated

tool-use-orchestration (agent workflow)

Horizon uses Slack interaction, SQL generation, query execution, and result explanation as one agent workflow.

“receive SQL queries, answers, and explanations within seconds.”summary, stated

eval-driven-development (quality)

Regression testing protected previously working queries as new errors were fixed.

“these updates included automated regression testing to ensure new changes didn't degrade performance on previously-working queries.”summary, stated

Tradeoffs

Use consensus generation for SQL.

Chose Ten-generation consensus approach over One-shot SQL generation.

Rationale: Consensus reduces nondeterministic incorrect answers in production.

“Instead of giving the LLM a single opportunity to generate SQL, the system requests ten different generations for each query.”summary, stated

Gotchas (what bit them)

LLM SQL generation is nondeterministic even with good context.

Root cause: Repeated generations can produce a mix of correct and incorrect SQL.

Resolution: Generate multiple candidates and use a consensus-based approach.

“To address this, Salesforce implemented a sophisticated consensus-based approach. Instead of giving the LLM a single opportunity to generate SQL, the system requests ten different generations for each query.”summary, stated

Opaque refusal or failure damages trust.

Root cause: Users cannot tell whether the agent lacks capability, needs clarification, or made a mistake.

Resolution: Ask clarifying questions and explain generated SQL.

“They redesigned the system to be more transparent, having it ask clarifying questions when ambiguous requests were received, and explaining the SQL it generated”summary, stated

Evaluation

Salesforce evaluated Horizon through correctness, regression tests for previously working queries, and production user trust signals around clarification and explanation.

Metrics: Correct SQL consensus. · Regression tests for previously working queries. · Minutes-to-insight versus hours/days wait time.

“Critically, these updates included automated regression testing to ensure new changes didn't degrade performance on previously-working queries.”summary, stated

Outcomes

The case reports dozens of technologist hours saved weekly and a product timeline from Early Access in August 2024 to GA in January 2025.

“freed technologists from routine query work and enabled non-technical users to self-serve data insights in minutes instead of waiting hours or days”summary, stated