Building a Financial Data RAG System: Lessons from Search-First Architecture

unspecified_client · Finance · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services RAG over proprietary documents

A financial-services client needed executives, finance teams, and investor-relations users to query SEC filings, earnings transcripts, analyst reports, and internal documents conversationally without spending hours or days on manual research.

Constraints:

answers needed correct attribution to the right company, analyst, document, and time period
financial tables, headers, footers, filings, and analyst reports behaved poorly under default token chunking
multi-company questions could exceed retrieval and context-window limits
users cared about recency, source category, and document reliability
search quality mattered more than the final LLM once the right evidence was retrieved

Architecture

The team started from Microsoft's Azure Search OpenAI Demo accelerator, using Azure AI Search, Azure OpenAI, Azure App Service, hybrid search, GPT-4/GPT-4-Turbo, and embeddings. After default chunking caused attribution and table-retrieval failures, they redesigned indexing around page-based chunks, generated searchable page summaries with GPT-4, added a query-routing decision agent to choose context versus content search, filtered disclaimers, passed filename/summary/content to the final answer step, and added recency scoring and category filters.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  financial-document-corpus[Financial document corpus] --> indexing-pipeline[Indexing pipeline]
  indexing-pipeline --> gpt-4-summary-generator[GPT-4 summary generator]
  gpt-4-summary-generator --> azure-ai-search-index[Azure AI Search index]
  indexing-pipeline --> azure-ai-search-index
  azure-ai-search-index --> recency-scoring-profile[Recency scoring profile]
  azure-ai-search-index --> category-filter[Category filter]
  chat-web-application[Chat web application] --> decision-agent[Decision agent]
  decision-agent --> azure-ai-search-index
  question-splitter[Question splitter] --> decision-agent
  azure-ai-search-index --> chunk-validator-agent[Chunk validator agent]
  chunk-validator-agent --> gpt-4-answer-synthesizer[GPT-4 answer synthesizer]
  azure-ai-search-index --> gpt-4-answer-synthesizer
  category-filter --> chat-web-application
  gpt-4-answer-synthesizer --> chat-web-application
  azure-openai-service[Azure OpenAI service] --> gpt-4-summary-generator
  azure-openai-service --> decision-agent
  azure-openai-service --> gpt-4-answer-synthesizer

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Financial document corpus`	SEC filings, earnings call transcripts, analyst reports, internal financial documents, and related source files.	`Indexing pipeline`
`Indexing pipeline`	Processes documents, extracts metadata, creates chunks, generates summaries, and writes fields into search.	`Financial document corpus`, `GPT-4 summary generator`, `Azure AI Search index`
`GPT-4 summary generator`	Creates page-level summaries designed to match what users would search for.	`Indexing pipeline`, `Azure AI Search index`
`Azure AI Search index`	Stores content, summary, filename, category, creation-date, and vector-search fields for retrieval.	`Indexing pipeline`, `Decision agent`, `Recency scoring profile`, `Category filter`, `GPT-4 answer synthesizer`
`Decision agent`	Classifies user questions as context search or content search and rewrites the search query.	`Azure AI Search index`, `GPT-4 answer synthesizer`
`Recency scoring profile`	Boosts documents with creation dates closer to the present.	`Azure AI Search index`
`Category filter`	Lets users restrict results to categories such as SEC filings, earnings transcripts, internal data, analyst reports, or FactSet sheets.	`Azure AI Search index`, `Chat web application`
`GPT-4 answer synthesizer`	Synthesizes final answers from retrieved filename, summary, and content.	`Decision agent`, `Chat web application`
`Question splitter`	Breaks multi-company questions into separate searches before recombining answers.	`Decision agent`, `GPT-4 answer synthesizer`
`Chat web application`	User-facing conversational application deployed from the Azure accelerator.	`Decision agent`, `Category filter`, `GPT-4 answer synthesizer`
`Azure OpenAI service`	Hosts GPT-4-32k, GPT-4-Turbo, and text-embedding-ada-02 for generation and embeddings.	`GPT-4 summary generator`, `Decision agent`, `GPT-4 answer synthesizer`
`Chunk validator agent`	Experimental LLM validation step for filtering irrelevant or unattributable chunks before answer synthesis.	`Azure AI Search index`, `GPT-4 answer synthesizer`

Data flow

Financial PDFs and documents were indexed into Azure AI Search. Early versions split documents into 1000-word chunks and used hybrid keyword/vector search before sending top chunks to GPT-4. The revised pipeline chunked by page, generated GPT-4 summaries as a searchable index field, used a decision agent to choose summary-focused context search or content-focused metric search, filtered junk disclaimer pages, passed filename plus summary plus content to GPT-4, and boosted fresh documents with a creation-date scoring profile.

3 supporting quotes

“Accelerator—a fully functional web application that can be tailored with your proprietary data sets. The solution integrates the following Azure services:”article, stated

“The default search algorithm utilized by this accelerator is "hybrid search," which concurrently conducts both vector and keyword searches”article, stated

“If the agent determined that context was more important, we would run a “context-focused query” (a search on the “summary” field of our index).”article, stated

Why this architecture

The system became search-first because the final LLM performed well when given the right content, but default retrieval repeatedly supplied wrong, unattributable, stale, or incomplete context. Page-based chunking preserved document context, generated summaries made semantic search align with user queries, and query routing separated context questions from metric lookups.

3 supporting quotes

“Generally speaking, if you give GPT4 all the right content, it gives you a high-quality answer. Getting it the right content is the hard part.”article, stated

“that more than one of our problems stemmed from how we were breaking up the documents, we felt the first logical first step would be to redesign our document chunking process. Instead of breaking them up by an arbitrary number of words, we decided to break them up in a way that preserved”article, stated

“Try to match your output with what someone would search for in a search engine if they were looking for this page.”article, stated

Models used: GPT-4-32k · GPT-4-Turbo · text-embedding-ada-02 · ChatGPT

Scale

The target workflow reduced research that could take multiple people multiple days into self-service minutes.
“All-in-all, this would take multiple people multiple days, if not more. With the chatbot, the executive can self-serve the information they need in a matter of minutes.”article, stated
Initial document chunks were 1000 words each.
“Optimal chunk size can vary, but typically, a range of 500 to 1000 words provides a balance for performance. In our case, documents were divided into chunks of 1000 words each.”article, stated
The default retrieval path returned the top five content chunks for a question.
“By default, we return the top 5 most relevant content chunks for a given question, and then feed that to GPT4.”article, stated
The discarded chunk-validation step added 15 to 20 seconds of latency.
“It added between 15-20 seconds of latency to the overall response time between when the original question was asked and when the answer was returned to the user.”article, stated
Multi-company questions could involve 10 to 12 companies and massive token use.
“Rate limits for the Azure OpenAI API would sometimes be hit. We were consuming a massive amount of tokens to address questions about 10-12 companies at one time.”article, stated
The front-end category filter covered five distinct data categories.
“The users categorized all of the data into 5 distinct categories: SEC Filings, Earnings Transcripts, Internal Data, Analyst Reports, and FactSet Data Sheets.”article, stated

Alternatives considered (roads not taken)

Fine-tuning a model on proprietary financial data

Rejected because: RAG was more accessible and avoided the cost and complexity of training while still grounding answers in proprietary data.

“A much more accessible alternative emerged called "Retrieval Augmented Generation" or RAG. RAG operates by retrieving the most pertinent data in response to a query and integrating it with ChatGPT's context, enabling the model to generate precise, conversational responses based on proprietary data.”article, stated

Default token-count chunking

Rejected because: It broke attribution context and split financial tables so the right rows or headers were missing.

“The chunk that contained the bottom half of the table was not being returned in the search results as it didn’t have any column headers”article, stated

Chunk validator agent before final answer generation

Rejected because: It helped filter bad content but added latency and masked poor retrieval rather than solving it.

“This approach helped filter out bad content, but it had its drawbacks. It added between 15-20 seconds of latency”article, stated

Current-date prompting for latest-data questions

Rejected because: The LLM could not map current date to fiscal quarters for each company, so recency needed to be represented in the search index.

“This didn’t have much of any effect, since it did not know the mapping between the current date & what fiscal quarter it was for the company in question.”article, stated

Breaking any multi-entity question into many separate searches

Rejected because: For lines of business and fiscal-quarter questions, splitting introduced more misses and wasted tokens because the needed data often lived on one page.

“This introduces more opportunity for a miss and is generally a waste of tokens/compute. So, we quickly refined the prompt to only break up questions asking about multiple companies.”article, stated

Patterns

`rag-retrieval` (retrieval)

The chatbot grounded financial answers in SEC filings, earnings transcripts, analyst reports, and internal documents instead of relying on model memory.

“The data sources for the project included SEC filings, earnings call transcripts, analyst reports, and internal financial documents.”article, stated

`hybrid-search` (retrieval)

The initial accelerator combined vector and keyword retrieval, then the team modified which fields were searched depending on the question.

“hybrid search," which concurrently conducts both vector and keyword searches, amalgamating the most effective results from each approach.”article, stated

`query-rewriting` (retrieval)

A decision agent classified questions as context or content search and returned an optimized search query.

“A decision “agent” that was responsible for determining what was more important for a given question: content or context.”article, stated

`retrieval-ranking` (retrieval)

The team added disclaimer filtering, summary-field search, creation-date boosting, and category tags to improve search result ranking and filtering.

“We added a filter on all searches that filtered out results where the world “disclaimer” showed up in the “summary” field.”article, stated

`llm-as-judge` (evaluation)

The discarded content chunk validation step used GPT-4 to judge whether retrieved chunks were relevant and attributable before answer synthesis.

“You are the chunk validator agent for a financial chatbot.”article, stated

`eval-driven-development` (evaluation)

The project iteratively diagnosed wrong answers and changed chunking, summaries, filters, recency, and prompts based on observed retrieval failures.

“At this point in the project, we had only seen very marginal improvement in the quality and accuracy of responses.”article, stated

Tradeoffs

Use the Azure accelerator for the first build

Chose a prebuilt Azure Search OpenAI Demo application over architecting the indexing and search layer from scratch first.

Rationale: The accelerator produced a fast client demo, but later required deep understanding to adapt its default behavior.

“we were able to quickly deliver the “wow” moment to the client.”article, stated

Chunk by page

Chose page-level chunks that preserved document context over arbitrary word-count chunks.

Rationale: Page headers and footers carried attribution and table context that financial answers required.

“we decided to break them up in a way that preserved important context. We opted to chunk by page”article, stated

Search summary and content fields differently

Chose context-focused summary search for analyst/performance questions and content search for metric lookups over one retrieval field for every question.

Rationale: Summary search improved context questions, while raw content still worked better for exact financial metrics.

“For specific financial metric retrieval, searching on the content field was still superior. So, we were still searching on both fields.”article, stated

Remove LLM chunk validation after improving retrieval

Chose better search architecture over an extra GPT-4 validation pass before answer synthesis.

Rationale: Validation added latency and token cost; better retrieval made it unnecessary.

“We officially removed the “content chunk validation” step that had been implemented earlier on.”article, stated

Split only multi-company questions

Chose separate search per company over splitting all multi-entity questions.

Rationale: Other entity splits often chased the same page repeatedly and wasted tokens.

“we quickly refined the prompt to only break up questions asking about multiple companies.”article, stated

Gotchas (what bit them)

The system returned right content with wrong attribution.

Root cause: Token chunks lost headers or speaker context, so GPT-4 inferred that the wrong analyst or company made the statement.

Resolution: Chunk by page and preserve filename, summary, headers, and page context in retrieval.

“Our vector search is failing in this instance. It is finding a content chunk that has the right content (analyst commentary on XXX), but the wrong context”article, stated

Financial tables failed under arbitrary token chunking.

Root cause: The row containing the answer could be separated from column headers or table context and never retrieved.

Resolution: Switch to page-based chunking to keep tables intact more often.

“The chunk that contained the bottom half of the table was not being returned in the search results as it didn’t have any column headers”article, stated

Keyword search over-ranked legal disclaimers and disclosure pages.

Root cause: Disclaimers contained repeated analyst/company terms but no useful answer content.

Resolution: Filter out results whose summary field contained disclaimer pages and add better summary fields.

“the page from the report that had the most instances of the term “Contoso” would often times be the disclaimers & certifications page.”article, stated

The chunk validator added latency without solving retrieval quality.

Root cause: It used another GPT-4 call to mask bad search results rather than improving the index and ranking.

Resolution: Remove validation after search architecture changes made the right chunks appear directly.

“Ultimately, we decided to abandon content chunk validation as a solution.”article, stated

Naive recency prompts did not work for financial-period questions.

Root cause: GPT-4 did not know how to map current dates to each company's fiscal calendar.

Resolution: Add creation-date metadata and a custom scoring profile that boosts fresh source documents.

“GPT4 proved to be very bad at this, so we scrapped this idea entirely.”article, stated

Evaluation

The team evaluated through observed user failures and iterative manual inspection of wrong answers, search results, chunks, and generated responses; the source explicitly calls for a future LLMOps testing framework.

Metrics: 15-20 seconds latency added by chunk validation · top 5 chunks by default · GPT-4-Turbo max response parameter increased to 4000 · 10-12 company questions could hit rate limits

“What is our baseline accuracy & quality? When we make a change, how does it affect our baseline? Testing & measuring this manually is laborious, time-consuming, and subjective.”article, stated

“we also bumped up the max token response parameter of GPT4-turbo to 4000 and added in “few-shot” examples to our main system prompt.”article, stated

Outcomes

page-based chunking improved table retrieval and attribution
summary-field search plus filename context improved analyst commentary and macro-trend questions
the client saw an almost overnight quality improvement after search architecture changes
the project reframed RAG quality as primarily a search/indexing problem

The public source does not provide exact production usage counts, but it describes executive and investor-relations workflows, multiple financial document categories, and latency/token pressure for 10-12 company queries.

“Almost overnight (from their perspective), the responses became more detailed, accurate, insightful, and higher quality.”article, stated

“RAG is more of a search exercise than it is an AI exercise”article, stated