Scaling RAG Accuracy from 49% to 86% in Finance Q&A Assistant

amazon_finance · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise finance Q&A

Amazon Finance Automation needed to reduce hours-to-days analyst response time for Accounts Payable and Accounts Receivable policy questions while maintaining accuracy in a finance operations context.

Constraints:

• Analysts had to consult SMEs and multiple SOP/policy documents.
• The assistant needed to retrieve grounded policy context and avoid hallucinated answers.
• Accuracy had to be measured systematically enough to guide iterative RAG improvements.

Architecture

Amazon Finance Automation built a Bedrock RAG assistant using OpenSearch as the vector store, Amazon Titan embeddings, a Bedrock foundation model generator, diversity and lost-in-the-middle rankers, Bedrock Guardrails for PII/prompt-injection protection, and a validation engine that returns an explicit 'I don't know' when generated answers do not align with retrieved context.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Analyst chat UI"] --> B["RAG orchestrator"]
  B --> C["OpenSearch vector store"]
  D["Titan embeddings"] --> C
  C --> E["Diversity ranker"]
  E --> F["Lost-in-the-middle ranker"]
  F --> G["Bedrock generator"]
  B --> H["Bedrock Guardrails"]
  G --> I["Validation engine"]
  I --> B

Components

Data flow

A finance analyst submits a query; the RAG pipeline retrieves relevant policy/SOP documents from an OpenSearch-backed knowledge base using Titan embeddings; rankers arrange retrieved context; a Bedrock foundation model generates the answer; guardrails and validation check PII, prompt injection, and answer-context alignment before the UI returns a cited response or fallback.

Why this architecture

The team used RAG because analysts needed answers grounded in finance policy documents, then improved retrieval, prompting, and embeddings only after SME/manual evaluation showed initial accuracy was too low.

Models used: Amazon Titan Multimodal Embeddings G1 · Amazon Titan Text Embeddings G1 · Amazon Bedrock foundation model · LLM-as-judge evaluation model

Scale

• Manual analyst response took hours to days before the assistant.
  “This back-and-forth communication process often takes from hours to days”article, stated
• The evaluation dataset contained 100 questions.
  “question – This consists of 100 questions from policy documents”article, stated
• Accuracy improved from 49% to 86%.
  “Our results have shown a significant improvement in RAG accuracy from 49% to 86%.”article, stated
• LLM-based scoring varied about 5% from human analysis, versus about 30% for ROUGE/METEOR.
  “The variance of this approach was approximately 5% compared to human analysis”article, stated

Alternatives considered (roads not taken)

Manual SME/policy lookup

Rejected because: It took hours to days and was especially hard for new hires.

Fixed-size document chunks

Rejected because: They created incomplete contexts because chunks ignored logical document structure.

ROUGE/METEOR as the main automatic evaluator

Rejected because: Word-matching scores ignored semantic meaning and diverged from human evaluation by about 30%.

Patterns

rag-retrieval (retrieval)

The assistant retrieves finance policy/SOP context from a vector knowledge base before generation.

llm-as-judge (evaluation)

A Bedrock foundation model scored generated answers against expected answers with accuracy, acceptability, factualness, and reasoning citations.

eval-driven-development (evaluation)

The team used measured error sources to drive semantic chunking, prompt engineering, and embedding-model changes.

safety-policy-enforcement (safety)

Guardrails and validation prevent PII leakage, prompt injection, and hallucinated responses.

Tradeoffs

RAG infrastructure

Chose OpenSearch vector store initially over Bedrock Knowledge Bases from the start.

Rationale: OpenSearch supported the initial knowledge base, while migration to managed Knowledge Bases was planned to reduce cluster management.

Fallback behavior

Chose explicit I don't know when context is incomplete over force an answer without enough retrieved context.

Rationale: Finance policy answers should not be hallucinated.

Evaluation automation

Chose LLM-based judge after SME calibration over continued fully manual SME evaluation.

Rationale: Manual evaluation was not sustainable, while LLM judging aligned more closely with human analysis than ROUGE/METEOR.

Gotchas (what bit them)

Initial RAG accuracy was far below production expectations.

Root cause: Retrieval, chunking, prompting, and embedding choices all affected answer correctness.

Resolution: Use an evaluation loop to improve semantic chunking, prompts, and embeddings.

Traditional NLP metrics missed semantic answer quality.

Root cause: ROUGE and METEOR match words rather than meaning.

Resolution: Use LLM-based evaluation aligned with SME judgments.

Retrieved context relevance stayed low after chunking.

Root cause: Embedding model choice still caused incorrect contexts to rank highly.

Resolution: Switch to Amazon Titan Text Embeddings G1.

Evaluation

The team used SME-labeled expected answers, ROUGE/METEOR comparisons, LLM-as-judge scoring, context relevance metrics, and staged accuracy measurements after each RAG improvement.

Metrics: RAG accuracy · acceptability · factualness · context relevance · context coverage · correctness · completeness · helpfulness

Outcomes

• Semantic chunking improved accuracy from 49% to 64%.
• Prompt engineering improved accuracy from 64% to 76%.
• Titan embeddings improved accuracy from 76% to 86%.
• Analysts can retrieve answers in the same communication thread instead of waiting hours or days.

This is a high-stakes enterprise finance RAG system with transparent staged accuracy reporting rather than a perfect-accuracy claim.