RAG System for Investment Policy Search and Advisory at RBC

arcane · Finance · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial investment policy search and advisor support

RBC needed investment advisors to pinpoint policy answers from proprietary semi-structured financial documentation without creating bottlenecks for specialized experts or exposing clients to misleading RAG outputs.

Constraints:

• The domain is financial services, with proprietary internal information and heavy privacy, security, and safety requirements.
• The source documents include semi-structured HTML, PDFs, internal websites, and other irregular formats.
• Specialized advisors and escalation experts are expensive bottlenecks, so delays have large productivity costs.
• Multi-turn conversations and hallucinations can produce convincing but false financial guidance.

Architecture

RBC built Arcane, an internal RAG system for direct-investment policy search. The system provides a chat interface with frequent advisor questions and chat-memory management, parses semi-structured internal policy sources, chunks and indexes them for retrieval, and evaluates outputs with both traditional retrieval metrics and RAGAS-style faithfulness and answer-relevance metrics. The pilot was tested in production, with Borealis AI positioned to scale it to enterprise users.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Semi-structured policy sources"] --> B["Parser and chunker"]
  B --> C["Vector retrieval index"]
  C --> D["Arcane chat interface"]
  D --> E["Generation model"]
  E --> D
  C --> F["RAG evaluation layer"]
  E --> F
  F --> G["Security and education controls"]
  D --> H["Advisor user"]
  G --> H
  I["Borealis AI"] --> D

Components

Data flow

Semi-structured financial policy sources such as HTML, PDFs, internal websites, and other documents are parsed into meaningful chunks and indexed for retrieval. Advisors ask policy questions in the Arcane chat interface, retrieval returns relevant parsed chunks, generation uses those chunks to answer, and evaluation checks retrieval precision/recall plus faithfulness and answer relevance. Human education and security controls are treated as part of the operating system around the RAG application.

Why this architecture

RBC chose RAG because policy answers were not in a structured database and advisors needed specific proprietary information, but the design emphasis is that parsing, chunking, evaluation, security, and user education matter as much as the model. The speaker repeatedly frames semi-structured parsing as the bottleneck and RAG evaluation as necessary because retrieval and generation can both fail.

Models used: RAGAS · Vector databases · GPT-3.5 · GPT-4 · GPT-4 Turbo

Scale

• The pilot was tested in production.
  “we worked on the pilot um side of it um so we tested everything in production”transcript, stated
• Enterprise scaling was framed as millions of users through Borealis AI.
  “taking it and um scaling it up and use it uh you know to millions and millions of um users that's something that our partner Borealis AI”transcript, stated
• Financial advisors are highly trained, often with 5 to 10 years of post-secondary STEM education.
  “these individuals typically arrive with 5-10 years of post-secondary STEM education, often including graduate studies.”summary, stated
• The bottleneck can have millions of dollars of bottom-line impact.
  “every second can be uh you know uh can have really um uh multiplicative productivity um impacts usually resulting in millions”transcript, stated

Alternatives considered (roads not taken)

Treat financial policy sources like structured database records

Rejected because: The information was not structured enough for exact queries and responses.

Throw RAG at every problem because it is available

Rejected because: The speaker explicitly warns against technology bias and urges matching RAG to useful cases.

Rely only on larger generation models to fix retrieval quality

Rejected because: Without proper parsing and chunking, even the best LLMs struggle to find specialized internal information.

Patterns

rag-retrieval (retrieval)

Arcane retrieves investment-policy context from parsed internal documentation before answering advisor questions.

eval-driven-development (quality)

RBC used RAGAS-style faithfulness and answer relevance alongside retrieval metrics such as precision and recall.

context-engineering (context)

The system's success depended on parsing and chunking financial policy content into meaningful context that retrieval and generation could use.

safety-policy-enforcement (risk)

The operating guidance combines evaluation, education, and systemic security to manage hallucination, privacy, and model-security risks.

Tradeoffs

Prioritize parsing and chunking over treating vector databases as the bottleneck.

Chose High-quality parsing/chunking of semi-structured sources over Assuming vector database choice alone solves retrieval.

Rationale: The speaker says vector databases were not the bottleneck; how they were used depended on parsing and chunking.

Use multiple evaluation methods.

Chose Retrieval precision/recall plus RAGAS faithfulness and answer relevancy over Only inspecting retrieved chunks or only judging generated answer fluency.

Rationale: Retrieval and generation fail differently, so evaluation needs both context and answer metrics.

Keep humans educated about RAG risk.

Chose User education plus guardrails over Trying to eliminate all RAG risk purely systematically.

Rationale: The speaker says the system cannot absorb all risk automatically, especially in multi-turn financial advice.

Gotchas (what bit them)

Semi-structured document parsing was the main bottleneck.

Root cause: HTML, PDFs, internal sites, and other semi-structured sources had irregular structure and inconsistent tagging.

Resolution: Invest in parsing and chunking before relying on vector retrieval or generation.

Multi-turn RAG can compound errors into convincing false answers.

Root cause: The model has to rely on retrieved context for unseen internal knowledge while also tracking conversational memory.

Resolution: Use careful prompt engineering, user education, and guardrails instead of assuming memory is safe.

False answers are worse than irrelevant answers in financial advice.

Root cause: A relevant-looking hallucination can misadvise the advisor or client.

Resolution: Evaluate faithfulness and answer relevance and maintain security/education controls.

Proprietary internal knowledge changes what models can know.

Root cause: The materials are internal and specialized, so public or broadly trained models have never seen them.

Resolution: Use RAG over proprietary parsed sources rather than expecting the base model to contain the knowledge.

Evaluation

RBC used a combination of information-retrieval metrics and RAGAS-style generation metrics: precision, recall, F-score, faithfulness, and answer relevance. The speaker also emphasizes production testing, user education, and security controls as part of the evaluation and risk envelope.

Metrics: Precision · Recall · F-score · Faithfulness · Answer relevancy · Production pilot testing

Outcomes

• RBC completed and production-tested an Arcane pilot for investment-policy search.
• The speaker identified parsing/chunking as the biggest practical bottleneck.
• The system established a combined retrieval and generation evaluation approach.
• Enterprise scaling was delegated toward Borealis AI rather than completed in the pilot.

The transcript reports a production-tested pilot, anticipated scaling to millions of users through Borealis AI, highly trained advisors with 5-10 years of post-secondary study, and potential millions of dollars in bottom-line impact from service bottlenecks.