Advanced RAG Implementation for AI Assistant Response Accuracy

nippon_india_mutual_fund · Finance · 2025 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services assistant

Nippon India Mutual Fund needed to improve AI assistant accuracy over large document collections where naive RAG missed context and hallucinated.

Constraints:

Large document volume
Compound questions
Hallucination risk
Limited top-k context

Architecture

Nippon Life India's assistant is a Bedrock-centered RAG system for accurate report generation from internal knowledge. Source documents are parsed, chunked, embedded, and stored in a vector database through Amazon Bedrock Knowledge Bases; user queries arrive through a business application such as Lex; conversation history and caches preserve context; Bedrock retrieves top matches, optionally reformulates compound questions into sub-queries, reranks retrieved chunks, and sends the grounded context to a foundation model. Evaluation and guardrails measure groundedness and hallucination reduction.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  source-documents[Source documents] --> document-parsing[Document parsing]
  document-parsing --> bedrock-knowledge-bases[Bedrock Knowledge Bases]
  bedrock-knowledge-bases --> vector-database[Vector database]
  business-application[Business application] --> conversation-context[Conversation context]
  conversation-context --> query-reformulator[Query reformulator]
  query-reformulator --> retriever[Retriever]
  bedrock-knowledge-bases --> retriever
  vector-database --> retriever
  retriever --> reranker[Reranker]
  reranker --> foundation-model[Foundation model]
  foundation-model --> rag-evaluation[RAG evaluation]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Source documents`	Business documents and knowledge sources used to ground assistant answers.	`Document parsing`
`Document parsing`	Custom parsing, Amazon Textract, or foundation-model parsing before ingestion.	`Source documents`, `Bedrock Knowledge Bases`
`Bedrock Knowledge Bases`	Managed RAG service handling ingestion, chunking, embedding, and query matching.	`Document parsing`, `Vector database`, `Retriever`
`Vector database`	Stores embeddings for source-document chunks.	`Bedrock Knowledge Bases`, `Retriever`
`Business application`	User-facing assistant workflow, such as Lex or application logic, that receives queries.	`Conversation context`, `Retriever`
`Conversation context`	Conversation history and caches used to contextualize follow-up questions.	`Business application`, `Query reformulator`
`Query reformulator`	LLM-based splitter for compound queries into multiple sub-queries.	`Conversation context`, `Retriever`
`Retriever`	Retrieves top matching chunks or sub-query results from the knowledge base.	`Bedrock Knowledge Bases`, `Reranker`
`Reranker`	Optional model that reorders retrieved results before generation.	`Retriever`, `Foundation model`
`Foundation model`	Generates the final answer using the query and retrieved context.	`Reranker`, `RAG evaluation`
`RAG evaluation`	Measures completeness, correctness, groundedness, hallucination, and answer quality.	`Foundation model`

Data flow

Documents enter an ingestion workflow where they are parsed, semantically chunked, embedded, and stored in a vector database. At generation time a user query and conversation context are processed by the business application, retrieved against Bedrock Knowledge Bases, optionally split into sub-queries and reranked, then passed with top context to a foundation model that generates a grounded response.

3 supporting quotes

“The RAG pattern for this solution has two primary workflows: a data ingestion workflow and a text generation phase”article, stated

“During the ingestion phase, the solution chunks the content of the source documents, creates embeddings for the created chunks, and stores them in a vector database.”article, stated

“The solution then sends the user query along with the retrieved top results as the context to the FM, which then provides the final response.”article, stated

Why this architecture

Nippon used Bedrock Knowledge Bases to avoid hand-building ingestion, chunking, embeddings, and matching, then layered semantic chunking, multi-query reformulation, reranking, and evaluation because naive top-k retrieval was not sufficient for complete, correct, grounded answers. The slight latency cost was accepted for better response quality.

3 supporting quotes

“does the heavy lifting of implementing a RAG pattern—including data ingestion, data chunking, data embedding, and query matching.”article, stated

“However, in a large enterprise scenario with a large number of relevant documents, the final response is generated based on only the top 5 or top 10 results provided by the database.”article, stated

“Because this requires additional processing, the response time is slightly longer, which is justified by the quality of the response.”article, stated

Models used: Amazon Bedrock foundation models · embedding model · reranker model

Scale

Accuracy increased by more than 95%.
“and ingestion. ## Results Nippon saw the following improvements after implementing RAG: * Accuracy was increased by more than 95% * Hallucination was reduced by 90–95% * They were able to add source chunks and file links (through file metadata), which improves the user confidence in the response”article, stated
Hallucination was reduced by 90-95%.
“Nippon saw the following improvements after implementing RAG: * Accuracy was increased by more than 95% * Hallucination was reduced by 90–95%”article, stated
Report generation time fell from two days to around ten minutes.
“The time needed to generate a report was reduced from 2 days to approximately 10 minutes ## Summary In this post, we discussed RAG and some of the challenges associated with processing large volumes of documents.”article, stated

Alternatives considered (roads not taken)

Use only the top 5 or 10 vector results

Rejected because: Compound questions may need broader retrieval, query splitting, and reranking to avoid incomplete answers.

“Nippon used an LLM to split the compound query into multiple sub-queries. The search results for each of the sub-queries are returned simultaneously and are then ranked and combined.”article, stated

Use fixed-size chunking for all documents

Rejected because: The documents had semantically similar sections, so semantic chunking better preserved meaning.

“Nippon uses semantic chunking because the documents have sections with semantic similarity. During the text generation workflow, Amazon Bedrock Knowledge Bases creates embeddings from the user input and performs semantic search in the Amazon Bedrock knowledge base.”article, stated

Optimize only for response speed

Rejected because: The team accepted slightly longer response time for higher response quality.

“Because this requires additional processing, the response time is slightly longer, which is justified by the quality of the response.”article, stated

Patterns

`rag-retrieval` (retrieval)

The assistant retrieves relevant document context before generation.

“To reduce hallucination and improve overall accuracy, Retrieval Augmented Generation (RAG) remains the most commonly used method to retrieve reliable and accurate responses that use enterprise data when responding to user queries.”article, stated

`query-rewriting` (retrieval)

The solution rewrites and aggregates queries to improve retrieval over large document sets.

“com/) that improves the accuracy of the response over a regular (naive) RAG approach by rewriting the user queries and aggregating and reranking the responses.”article, stated

`retrieval-ranking` (retrieval)

Reranking addresses the limited-context problem from naive top-k retrieval.

“com/) that improves the accuracy of the response over a regular (naive) RAG approach by rewriting the user queries and aggregating and reranking the responses.”article, stated

Tradeoffs

How to improve retrieval quality for compound questions.

Chose query splitting and reranking over basic top-k retrieval only.

Rationale: The system traded extra processing and latency for higher-quality responses from multiple sub-query results.

“The search results for each of the sub-queries are returned simultaneously and are then ranked and combined.”article, stated

How to chunk documents for Bedrock Knowledge Bases.

Chose semantic chunking over fixed uniform chunks.

Rationale: Semantic sections in the source documents made meaning-aware chunking a better fit.

“Nippon uses semantic chunking because the documents have sections with semantic similarity.”article, stated

Gotchas (what bit them)

Naive RAG can miss relevant sections in large document sets.

Root cause: Top-k similarity retrieval provides only a small context set.

Resolution: The team added advanced RAG workarounds.

“* Limited use of relevant data – As the dataset grows, only the configured set of top results are used for context, impacting accuracy”article, stated

Evaluation

The case reports accuracy, hallucination, and report-generation-time improvements.

Metrics: accuracy · hallucination reduction · report generation time

“The solution resulted in over 95% accuracy improvement, 90-95% reduction in hallucinations, and reduced report generation time from 2 days to approximately 10 minutes.”summary, stated

Outcomes

The summary reports >95% accuracy improvement and 90-95% hallucination reduction.

Report generation time fell from days to minutes.