SQL Generation and RAG for Financial Data Q&A Chatbot

q4 · Finance · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-data question answering for investor relations

Q4 needed a chatbot for Investor Relations Officers that could answer questions over structured and numerical financial data, where conventional semantic RAG retrieved incorrect context because embeddings struggled with numbers, transactions, stock quotes, and dates.

Constraints:

• The data mixed CRM, ownership records, stock-market data, proprietary customer-specific Q4 data, and public data.
• The system had to maintain security, privacy, and query response time of seconds.
• Semantic search and embeddings worked better for text than for mostly numerical financial data.
• The architecture needed to choose appropriate foundation models for SQL generation, validation, and summarization.

Architecture

Q4 implemented a Bedrock-backed text-to-SQL RAG pipeline. A user question, database schema, sample rows, and instructions are sent to an LLM through SQLDatabaseChain to generate SQL. The SQL runs against the structured database to retrieve context, then a second LLM step summarizes the answer. Model selection was tuned per stage, eventually using Claude variants for SQL and Titan Text Express or Claude Instant for summarization.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  investor_relations_user["Investor relations user"] --> q4_q_a_chatbot["Q4 Q&A chatbot"]
  q4_q_a_chatbot --> sql_generation_prompt["SQL generation prompt"]
  sql_generation_prompt --> sqldatabasechain["SQLDatabaseChain"]
  sqldatabasechain --> amazon_bedrock_models["Amazon Bedrock models"]
  sqldatabasechain --> structured_financial_database["Structured financial database"]
  structured_financial_database --> retrieved_context["Retrieved context"]
  retrieved_context --> summarization_prompt["Summarization prompt"]
  q4_q_a_chatbot --> summarization_prompt
  summarization_prompt --> amazon_bedrock_models

Components

Data flow

An IRO asks a natural-language question. The system combines the question with schema, sample rows, and instructions, then uses SQLDatabaseChain and Amazon Bedrock to generate SQL. The SQL query retrieves context from the database. The original question, retrieved context, and summarization instructions are sent back to an LLM to produce the final answer.

Why this architecture

Q4 moved from semantic RAG to SQL generation because numerical financial data requires precise structured retrieval. Bedrock and LangChain let the team switch models by task, generate SQL from schema-rich prompts, retrieve exact database context, and summarize answers while optimizing latency and cost.

Models used: Anthropic Claude V2 · Claude Instant · Titan Text Express · Amazon Bedrock foundation models

Scale

• The user-experience target was query response time of seconds.
  “As for performance, the goal was to maintain a query response time of seconds to ensure a positive experience for end-users.”article, stated
• Q4 found two to five sample rows sufficient, and used three rows in its prompt context.
  “we found 2–5 rows of table data to be sufficient to give enough information to the model without adding too much unnecessary overhead.”article, stated

  “Three rows were just enough to provide context, without overwhelming the model with too much input.”article, stated
• The final system achieved acceptable single-digit-second end-to-end latency.
  “The end-to-end latency of the overall solution came within what was defined as acceptable for the use case—single-digit seconds.”article, stated

Alternatives considered (roads not taken)

Conventional RAG with search, semantic search, and embeddings

Rejected because: It retrieved incorrect context for numerical and structured financial data because embeddings struggled with numbers and similarity ranking.

Keeping an LLM SQL validation stage

Rejected because: After testing, generated SQL was consistently accurate enough that validation did not justify its added latency and cost.

Using Claude V2 for every LLM call

Rejected because: Smaller or more performant models could satisfy simpler SQL or summarization tasks at better latency and cost.

Patterns

rag-retrieval (retrieval architecture)

Q4 used SQL retrieval as the grounding step in a RAG-style chatbot, replacing vector retrieval with structured database context extraction.

tool-use-orchestration (orchestration)

SQLDatabaseChain orchestrated SQL generation and database interaction between natural-language questions, schemas, LLMs, and structured data.

model-routing (serving)

The team selected different Bedrock models for query generation, validation, and summarization depending on accuracy, latency, and cost requirements.

prompt-chaining (prompt workflow)

The system chains SQL generation, optional validation, database retrieval, and final answer summarization as distinct LLM/context steps.

structured-output-contract (output control)

Q4 instructed the model to output clean SQL only and used schema/sample rows/field explanations to constrain generated queries.

Tradeoffs

Use SQL generation for retrieval.

Chose Text-to-SQL over the structured financial database over Semantic vector retrieval over numerical data.

Rationale: SQL gave higher recall and more domain-bounded context for numerical datasets where embeddings retrieved incorrect information.

Select models per task.

Chose Claude V2 for harder query generation and smaller models for simpler or summarization tasks over One large model for every LLM call.

Rationale: Bedrock made it easy to change model names, allowing Q4 to balance accuracy, performance, and cost by task.

Eliminate SQL validation after testing.

Chose Skip the validation stage over Always run native or LLM-based SQL validation.

Rationale: The generated SQL was consistently accurate enough that removing validation improved latency and cost without hurting output quality.

Gotchas (what bit them)

Vector retrieval can fail on numerical financial data.

Root cause: Embeddings generated from numbers struggled with similarity ranking and retrieved incorrect information.

Resolution: Generate SQL over a structured dataset and use the SQL result as the context for the final answer.

SQL generation needs schema context, not just the user question.

Root cause: The LLM must understand the database structure, sample rows, and field meanings to generate accurate SQL.

Resolution: Include schema, sample rows, and human-readable field explanations in the prompt.

Validation stages can add unnecessary latency and cost.

Root cause: If generated SQL is already consistently accurate, a second LLM/database validation trip may not add enough value.

Resolution: Q4 removed the SQL validation stage after testing showed no negative impact on output quality.

Evaluation

Q4 evaluated retrieval approach quality during proof-of-concept experiments, compared model suitability by task, tested SQL quality, and used latency/cost tradeoffs to decide whether to retain validation and which models to use.

Metrics: Target query response time: seconds. · Final end-to-end latency: single-digit seconds. · Prompt context: two to five rows sufficient, with three rows used. · SQL generation was consistently accurate enough to remove validation.

Outcomes

• Q4 fulfilled the chatbot solution requirements after replacing vector retrieval with SQL-based retrieval.
• The system generated high-quality SQL for simple filters and complex aggregations.
• The team achieved acceptable single-digit-second latency while reducing validation overhead.

The final proof-of-concept used structured ownership data and achieved single-digit-second end-to-end latency with schema-plus-three-row prompts.