Building a Secure and Scalable LLM Gateway for Financial Services

wealthsimple · Finance · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services internal GenAI platform

Wealthsimple needed a way for employees and production systems to use generative AI without sending sensitive financial data directly to public LLM providers, while still supporting provider choice, RAG, self-hosted models, multimodal workflows, and client-operations use cases.

Constraints:

• Financial-services data made public LLM use risky without guardrails and auditability
• The first gateway tracked usage but did not prevent sensitive data from leaving
• PII redaction improved safety but could reduce answer relevance
• Some employees legitimately needed unredacted PII in their workflows
• The original gateway API was not compatible with emerging OpenAI-style standards
• Multiple GenAI surfaces created adoption and productivity friction

Architecture

Wealthsimple built an internal LLM gateway as the secure entry point for GenAI. The gateway proxies employee and production traffic to external providers, logs outbound data, supports model selection, retries, fallback, and later OpenAI-compatible APIs. Around it, Wealthsimple added PII redaction, self-hosted open-source LLMs with llama.cpp for sensitive workflows, an Elasticsearch/OpenSearch RAG API, Streamlit-based data applications, multimodal gateway inputs, and Boosterpack as a context-grounded assistant.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  employee[Employee] --> llm-gateway[LLM Gateway]
  employee[Employee] --> boosterpack[Boosterpack]
  employee[Employee] --> data-applications-platform[Data Applications Platform]
  llm-gateway[LLM Gateway] --> pii-redaction-model[PII Redaction Model]
  llm-gateway[LLM Gateway] --> external-llm-providers[External LLM Providers]
  llm-gateway[LLM Gateway] --> self-hosted-llms[Self Hosted LLMs]
  llm-gateway[LLM Gateway] --> rag-api[RAG API]
  rag-api[RAG API] --> knowledge-bases[Knowledge Bases]
  knowledge-bases[Knowledge Bases] --> boosterpack[Boosterpack]
  data-applications-platform[Data Applications Platform] --> llm-gateway[LLM Gateway]

Components

Data flow

A user or production system sends a request through the gateway behind VPN and Okta. The gateway logs who sent what data externally, routes the request to a selected provider or self-hosted model, applies PII redaction for external providers when needed, and exposes APIs with retry and fallback. RAG workloads use Airflow to index knowledge bases into Elasticsearch/OpenSearch, while Boosterpack and data applications consume those platform building blocks.

Why this architecture

Wealthsimple chose a governed platform rather than banning GenAI because secure adoption required making the safe path the easy path. The gateway gave auditability, provider optionality, reliability mechanisms, and production APIs; PII redaction and self-hosted models handled data-sensitivity tradeoffs; RAG and data applications reduced friction for grounded internal use cases.

Models used: OpenAI models · Cohere models · Gemini models · Anthropic models through Bedrock · Llama 2 · Mistral models · Whisper · llama.cpp

Scale

• Wealthsimple launched the LLM gateway in 2023.
  “The first thing that we did in 2023 was launching our LLM gateway. When ChatGPT first became popularized, the security awareness from the general public for fourth, and third-party data sharing was not as mature as it was today.”article, stated
• The gateway is used internally by over half the company.
  “We developed and open sourced our LLM gateway, which, internally, is used by over half the company. We developed and shipped our in-house PII redaction model.”article, stated
• The data applications platform had over seven applications running within two weeks.
  “Within just the first two weeks of launching our data application platform, we had over seven applications running on it.”article, stated
• Two of those early data applications reached production.
  “Of those seven, two of them eventually made it into production where they're adding value and optimizing operations”article, stated
• Close to one-third of end users used multimodal gateway features at least weekly after launch.
  “Within the first few weeks of launching this, close to a third of all of our end users started leveraging a multi-modal feature at least once a week.”article, stated
• Across Wealthsimple's GenAI tools, more than 2,200 messages are sent daily.
  “Across all the different tools we offer, over 2200 messages get sent daily. Close to a third of the entire company are weekly active users. Slightly over half of the company are monthly active users.”article, stated
• Close to one-third of the company are weekly active users and slightly over half are monthly active users.
  “Close to a third of the entire company are weekly active users. Slightly over half of the company are monthly active users.”article, stated

Alternatives considered (roads not taken)

Ban employee use of ChatGPT and similar public LLM providers

Rejected because: Wealthsimple wanted productivity gains while controlling data exposure through a gateway.

Rely on PII redaction for every sensitive workflow

Rejected because: Redaction was sometimes inaccurate and interfered with relevance, while some jobs required unredacted PII.

Keep a custom non-standard gateway API

Rejected because: OpenAI-compatible APIs reduced integration headaches with LangChain and the maturing GenAI ecosystem.

Build all GenAI platform capabilities internally

Rejected because: By 2024, improved vendor security and business-alignment pressure made managed services such as Bedrock worth adopting.

Patterns

model-routing (platform)

The gateway gives employees and production systems centralized access to multiple LLM providers and model choices.

privacy-preserving-processing (security)

Wealthsimple combines PII redaction for external providers with self-hosted models when sensitive data must remain unredacted.

rag-retrieval (retrieval)

The platform indexes company knowledge bases into Elasticsearch/OpenSearch and exposes semantic search as a RAG API.

fallback-degradation (resilience)

The LLM service increased reliability with retry and fallback behavior across providers.

structured-output-contract (integration)

Wealthsimple moved the gateway API toward OpenAI-compatible specs to align with ecosystem integrations.

Tradeoffs

Use a governed internal gateway rather than banning LLM tools

Chose A VPN/Okta-gated LLM gateway over A blanket employee ban on public GenAI assistants.

Rationale: The gateway allowed exploration and productivity while giving the company an audit and policy control point.

Self-host models for sensitive workflows

Chose Open-source LLMs running inside Wealthsimple VPCs over Redacting every prompt before external provider calls.

Rationale: Self-hosting preserved useful sensitive context while keeping data within Wealthsimple's cloud environments.

Adopt managed Bedrock capabilities as the market matured

Chose A more buy-focused build-versus-buy posture over Continuing to recreate all marketplace GenAI tooling internally.

Rationale: Provider security had improved, and Wealthsimple saw more leverage in business-specific applications than in rebuilding commodity platform services.

Gotchas (what bit them)

An audit-only gateway left security gaps.

Root cause: The first gateway tracked outbound data but lacked preventive guardrails against external sharing.

Resolution: Wealthsimple added PII redaction and self-hosted LLM options.

PII redaction reduced usefulness for some legitimate workflows.

Root cause: The redaction model was imperfect and some employees' jobs required sensitive context.

Resolution: Self-hosted LLMs kept data in Wealthsimple cloud environments without redaction.

Non-standard APIs created integration cost.

Root cause: The original gateway API did not match the emerging OpenAI API standard.

Resolution: Wealthsimple shipped a v2 API mirroring OpenAI specs.

Separate GenAI tools split adoption.

Root cause: Boosterpack and the gateway created multiple places for similar GenAI needs.

Resolution: The team concluded that future investments needed to centralize tooling where work already happens.

Evaluation

Wealthsimple evaluated the platform through adoption, message volume, feature usage, survey/interview feedback, and production use cases such as client-experience triaging.

Metrics: over 2,200 daily messages · close to one-third weekly active users · slightly over half monthly active users · nearly one-third weekly multimodal feature use · two early data applications reaching production

Outcomes

• The LLM gateway is used by over half of Wealthsimple
• The internal GenAI tools process over 2,200 daily messages
• Close to one-third of the company are weekly active users
• Two data-app-platform applications reached production
• Whisper and self-hosted LLMs extended client-experience triaging beyond email tickets

The source describes a 2023-2024 evolution from gateway to redaction, self-hosted models, RAG, data apps, multimodal input, Bedrock adoption, and production triaging.