Enterprise-Wide RAG Implementation with Amazon Q Business

principal_financial · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services knowledge management

Principal Financial needed an enterprise AI assistant to help employees navigate fragmented legacy knowledge resources, work instructions, and case-processing procedures while preserving human oversight in regulated financial-services customer interactions.

Constraints:

A 300+ employee team processed more than 680,000 customer cases in 2023.
Employees searched more than 9,000 pages of work instructions and needed about 1.5 years to become fully efficient.
Knowledge lived across S3, SharePoint, Confluence, Jira, ServiceNow, and mixed file formats including ASPX pages.
Legacy content was organized for human navigation rather than retrieval systems.
The team needed measurable retrieval accuracy and feedback before widening use.

Architecture

Principal first built a RAG assistant using QnABot on AWS, Amazon Lex web UI, Amazon Kendra retrieval from S3, and Claude on Bedrock for summarization, then shifted rapidly to Amazon Q Business. The final architecture keeps QnABot, Lex web UI, and Amazon Q Business, with OpenSearch/Lex intent routing, CloudFront identity integration, Lambda processing, Kinesis Firehose conversation logs to OpenSearch, S3 backups, and Athena/QuickSight analytics. Retrieval quality improved through custom metadata generation with Bedrock, DynamoDB metadata storage, Q Business relevance tuning, and Top-K document return.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  enterprise-document-sources["Enterprise document sources"] --> amazon-q-business["Amazon Q Business"]
  qnabot-and-lex-interface["QnABot and Lex interface"] --> intent-routing-layer["Intent routing layer"]
  intent-routing-layer --> amazon-q-business
  amazon-q-business --> metadata-enrichment-pipeline["Metadata enrichment pipeline"]
  metadata-enrichment-pipeline --> amazon-q-business
  amazon-q-business --> conversation-logging-pipeline["Conversation logging pipeline"]
  human-review-workflow["Human review workflow"] --> qnabot-and-lex-interface

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Enterprise document sources`	S3, SharePoint, Confluence, Jira, ServiceNow, and files such as HTML, ASPX, Word, PDF, Excel, and PowerPoint.	`Amazon Q Business`
`QnABot and Lex interface`	Conversational multi-channel interface and branded web UI for the assistant.	`Intent routing layer`
`Intent routing layer`	Amazon OpenSearch and Lex routing that directs user requests.	`Amazon Q Business`
`Amazon Q Business`	Core retrieval and generation engine replacing much of the custom RAG stack.	`Metadata enrichment pipeline`, `Conversation logging pipeline`
`Metadata enrichment pipeline`	Bedrock, Lambda, DynamoDB, and Q relevance tuning for custom metadata, acronyms, keywords, and summaries.	`Amazon Q Business`
`Conversation logging pipeline`	Kinesis Firehose, OpenSearch, S3, Athena, and QuickSight pipeline for prompts, returns, metadata, and reporting.	—
`Human review workflow`	Human-in-the-loop usage model for customer interactions and validation.	`QnABot and Lex interface`

Data flow

Documents from S3, SharePoint, Confluence, Jira, ServiceNow, and other systems are indexed into Amazon Q Business. For SharePoint list pages, Principal enriches content by sending page contents to Bedrock to generate keywords, acronym expansions/collapses, and summaries, stores metadata in DynamoDB, attaches it during tuning, routes user intent through QnABot/Lex/OpenSearch, retrieves/generates with Q Business, and logs prompts, returns, and metadata through Kinesis Firehose into OpenSearch and reporting tools.

“So our final solution continues to integrate QnABot on AWS, AWS Lex web UI, and Amazon Q for Business”transcript, stated

“send it out to Bedrock and generate custom metadata including keywords, expanded and collapsed acronyms”transcript, stated

Why this architecture

Principal moved from a self-managed QnABot/Kendra/Claude stack to Amazon Q Business because a small team struggled with hyperparameter tuning, access control, data management, prompt engineering, and AI/ML maintenance. The final architecture keeps the conversational interface and logging/reporting pieces, while Q Business carries retrieval/generation and metadata/relevance tuning addresses the legacy-data shape problem.

“we were able to shift literally overnight that first week of January, 2024, to Amazon Q.”transcript, stated

“the quality of your metadata is absolutely crucial.”transcript, stated

Models used: Amazon Q Business · QnABot on AWS · Amazon Lex · Amazon Kendra · Anthropic Claude · Amazon Bedrock · Amazon OpenSearch · AWS Lambda · Kinesis Firehose · DynamoDB · Athena · QuickSight

Scale

Principal is a 145-year-old business serving more than 68 million customers in more than 80 markets.
“serving more than 68 million customers and more than 80 markets around the world.”transcript, stated
Principal has more than 19,000 employees.
“leveraging the experience and expertise of over 19,000 employees”transcript, stated
The first large use case involved over 300 employees.
“we had over 300 employees sitting within one team”transcript, stated
The team processed more than 680,000 customer cases in 2023.
“this team processed more than 680,000 individual customer cases.”transcript, stated
Employees searched more than 9,000 pages of work instructions.
“searched through over 9,000 pages of work instructions”transcript, stated
Initial retrieval accuracy improved from 67% to about 84%.
“Before we did absolutely anything with this, we were only getting 67% accuracy.”transcript, stated

Alternatives considered (roads not taken)

Continue managing the full custom RAG stack.

Rejected because: Maintaining tuning, access control, data management, prompt engineering, and architecture was too much for the small team.

“managing the complete stack—hyperparameter tuning, architecture maintenance, user access control, data management, and prompt engineering—proved extremely challenging for their small team.”summary, stated

Use existing human-oriented metadata as-is.

Rejected because: Legacy metadata was missing, incorrect, and designed for human navigation rather than machine retrieval.

“our metadata was meant for humans. We had it missing. It was incorrect.”transcript, stated

Index ASPX SharePoint list pages without workaround.

Rejected because: The SharePoint connector initially could not index server-rendered ASPX pages.

“the SharePoint connector at the time couldn't actually index these types of pages because ASPX pages specifically need to be rendered”transcript, stated

Patterns

`rag-retrieval` (enterprise search)

Amazon Q Business retrieves and generates answers over work instructions and enterprise documents.

“Amazon Q Business serves as the core retrieval and generation engine”summary, stated

`data-quality-loop` (metadata tuning)

Principal generates custom metadata with Bedrock and tunes relevance weights to improve document retrieval accuracy.

“the relevance tuning within Q Business, we were able to put more or less emphasis on those specific metadata fields.”transcript, stated

`eval-driven-development` (retrieval evaluation)

The team built a golden-truth dataset and programmatically modified metadata tuning to measure document-return accuracy.

“Their approach involved programmatically running prompts and evaluating outputs and document relevance while systematically modifying metadata tuning features.”summary, stated

`human-in-the-loop-gate` (regulated workflow)

Principal keeps humans in the loop for AI interactions with customers.

“we firmly believe in having a human in the loop in all AI interactions.”transcript, stated

`observability-tracing` (logging)

Prompts, returns, and metadata are logged through Firehose/OpenSearch and integrated with Athena/QuickSight for reporting.

“the prompt, the return, any related metadata, through Kinesis Firehose into OpenSearch”transcript, stated

Tradeoffs

Move from Kendra/Claude custom RAG to Amazon Q Business.

Chose Amazon Q Business as the core retrieval/generation engine over Owning more of the QnABot/Kendra/Claude stack.

Rationale: Q Business reduced AI/ML and retrieval-integration maintenance burden for a small team.

“we were able to shift literally overnight that first week of January, 2024, to Amazon Q.”transcript, stated

Use Bedrock-generated custom metadata.

Chose Keywords, acronyms, and summaries generated at index time over Only standard or manual metadata.

Rationale: Manual/human metadata was missing or incorrect for retrieval, so enrichment improved search relevance.

“send it out to Bedrock and generate custom metadata including keywords, expanded and collapsed acronyms”transcript, stated

Use Top-K equals one for this use case.

Chose Return only one document combined with metadata tuning over Summarizing across many documents.

Rationale: The answer usually lived on one page, while multi-page summarization returned incorrect policies or procedures.

“what was called a Top-K feature, which meant that we could choose the number of documents to return.”transcript, stated

Gotchas (what bit them)

Human-readable knowledge structures do not automatically work for machine retrieval.

Root cause: Legacy pages were linked and organized for people, but became flat objects when indexed.

Resolution: Generate custom metadata, keywords, acronyms, and summaries with Bedrock and attach them during tuning.

“these legacy systems was logically organized for human readability and human consumption.”transcript, stated

Connector support can block apparently simple document ingestion.

Root cause: SharePoint ASPX list pages required server-side rendering and were not indexable by the connector at first.

Resolution: Work with AWS on connector support and use print-to-PDF attachments for early experimentation.

“ASPX pages specifically need to be rendered on a server in order to be read”transcript, stated

Retrieval accuracy is highly sensitive to tuning choices.

Root cause: Some metadata tuning combinations performed far worse than baseline.

Resolution: Use labeled datasets and systematic tuning tests before deploying a configuration.

“We found combinations that got us down as low as 43%, which is bad, that's a bad number.”transcript, stated

Evaluation

Principal evaluates retrieval with a golden-truth labeled dataset, programmatic prompt/output/relevance testing, metadata tuning experiments, accuracy measurement, user feedback, and production log analysis.

Metrics: Correct-document retrieval accuracy · 67% baseline accuracy · 84% tuned accuracy · 43% poor tuning configuration · Positive or built-upon feedback · Rejected feedback rate · Workload reduction

“we got up to 84%. Before we did absolutely anything with this, we were only getting 67% accuracy.”transcript, stated

“over 97% have been accepted or built upon, and only 3% have returned”transcript, stated

Outcomes

Correct-document retrieval accuracy improved from 67% to about 84%.
About 800 users had platform access at review time.
Some users reported at least a 50% reduction on some workloads, and over 97% of logged interactions were accepted or built upon.

Principal's first large use case targeted a 300+ person team processing more than 680,000 annual cases across more than 9,000 pages of instructions.

“Right now, we have about 800 users with access to the platform”transcript, stated

“many of these users have stated that they've seen at least a 50% reduction on some workloads”transcript, stated