Building a Secure and Scalable LLM Gateway for Enterprise GenAI Adoption

wealthsimple · Finance · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services internal GenAI platform

Wealthsimple wanted employees to benefit from generative AI without leaking sensitive financial data to third-party providers, while supporting provider optionality, reliability, multimodal inputs, RAG, self-hosted models, employee adoption, and future client-facing GenAI work.

Constraints:

• Regulated financial data created security and privacy concerns around third-party LLM tools
• The first gateway had audit trails but few guardrails against external sharing
• PII redaction improved security but sometimes harmed answer relevance
• Employees needed some unredacted PII for day-to-day work
• The first internal API was not OpenAI-compatible, creating integration headaches
• Separate GenAI tools hurt adoption and productivity

Architecture

Wealthsimple built a centralized LLM gateway and internal GenAI platform. The gateway proxies employee and production-system requests to providers, records audit trails, supports model selection, retries, fallback, and rate-limit work, and later moved toward OpenAI-compatible APIs. The platform adds a custom PII redaction model for external providers, self-hosted open-source LLMs via llama.cpp for sensitive unredacted data, a semantic-search RAG API, a Streamlit data-app platform, multimodal uploads, and internal productivity tools grounded in Wealthsimple context.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  employee[Employee] --> llm-gateway[LLM Gateway]
  employee[Employee] --> boosterpack[Boosterpack]
  employee[Employee] --> data-applications-platform[Data Applications Platform]
  llm-gateway[LLM Gateway] --> pii-redaction-model[PII Redaction Model]
  llm-gateway[LLM Gateway] --> external-llm-providers[External LLM Providers]
  llm-gateway[LLM Gateway] --> self-hosted-llms[Self Hosted LLMs]
  llm-gateway[LLM Gateway] --> rag-api[RAG API]
  rag-api[RAG API] --> company-context[Company Context]
  company-context[Company Context] --> boosterpack[Boosterpack]
  data-applications-platform[Data Applications Platform] --> llm-gateway[LLM Gateway]

Components

Data flow

An employee or internal application sends a prompt through the gateway. The gateway logs who sent what data and where it goes, proxies requests to selected LLM providers with retry/fallback behavior, redacts PII for external providers when needed, or routes sensitive unredacted work to self-hosted models. Developers can build on the RAG API and data-app platform, while adoption is guided by nudges, guidelines, and making the gateway free and convenient.

Why this architecture

Wealthsimple chose a gateway rather than banning LLMs because it wanted the secure path to be the easy path. Audit trails, redaction, self-hosted models, RAG, and provider optionality let employees use GenAI while keeping sensitive data controls and reliability inside a financial-services environment.

Models used: OpenAI models · Cohere models · Gemini 1-million-token context window models · self-hosted open-source LLMs · llama.cpp

Scale

• Wealthsimple started the gateway in 2023 after ChatGPT's November 30, 2022 launch.
  “On November 30, 2022, OpenAI released ChatGPT. That release changed the way the world understood and consumed Generative AI (GenAI).”article, stated

  “The first thing that we did in 2023 was launch our LLM gateway. When ChatGPT first became popular, the general public was not as aware of third-party data sharing as it is today.”article, stated
• The gateway is used internally by over half the company.
  “We developed and open sourced our LLM gateway, which, internally, is used by over half the company. We developed and shipped our in-house personally identifiable information (PII) redaction model.”article, stated
• The data applications platform had more than seven applications within two weeks.
  “Within just the first two weeks of launching our data application platform, we had over seven applications running on it.”article, stated
• Two of the first data applications reached production.
  “Among those seven, two eventually made it into production where they're adding value and optimizing operations and creating a more delightful client experience.”article, stated
• Nearly one-third of end users used multimodal upload at least once a week after launch.
  “Within the first few weeks of launching this tool, nearly one-third of our end users started leveraging a multi-modal feature at least once a week.”article, stated
• Across GenAI tools, employees send over 2,200 messages daily.
  “Across all the different tools we offer, over 2200 messages are sent daily. Close to a third of the entire company are weekly active users. Slightly over half of the company are monthly active users.”article, stated
• Close to one-third of the company are weekly active users and slightly over half are monthly active users.
  “Close to a third of the entire company are weekly active users. Slightly over half of the company are monthly active users.”article, stated

Alternatives considered (roads not taken)

Ban employee use of ChatGPT and external LLMs

Rejected because: Wealthsimple wanted to capture GenAI productivity gains while controlling data exposure.

Use PII redaction for all sensitive workflows

Rejected because: Redaction sometimes hurt answer relevance and employees sometimes needed unredacted PII for work.

Expose a non-standard internal API

Rejected because: OpenAI API compatibility became necessary for integrations with LangChain and other libraries.

Patterns

model-routing (platform)

The gateway centralizes access to multiple LLM providers and lets users select models.

privacy-preserving-processing (security)

Wealthsimple uses PII redaction for external providers and self-hosted models when unredacted PII is needed.

rag-retrieval (retrieval)

The first RAG API gave developers and users semantic search grounded in company context.

fallback-degradation (resilience)

The LLM service handles retries and fallback to improve availability across providers.

structured-output-contract (integration)

The v2 API mirrored OpenAI's API specs to reduce integration friction.

Tradeoffs

Build a gateway instead of banning LLM use

Chose Controlled internal gateway over Employee ban on ChatGPT-style tools.

Rationale: The gateway preserved exploration and productivity while adding audit and privacy controls.

Self-host open-source LLMs for sensitive data

Chose llama.cpp-backed self-hosted models over Redacting all data before external provider calls.

Rationale: Self-hosting preserved unredacted context for legitimate workflows while keeping data inside Wealthsimple's cloud.

Adopt OpenAI-compatible API standards

Chose OpenAI API compatibility over A custom internal API structure.

Rationale: Compatibility reduced headaches integrating with LangChain and maturing GenAI tooling.

Gotchas (what bit them)

Audit trails alone did not prevent data leakage.

Root cause: The first gateway tracked outbound data but had few guardrails preventing sensitive data from being shared externally.

Resolution: Wealthsimple added a PII redaction model and self-hosted LLM paths.

PII redaction hurt answer relevance for legitimate work.

Root cause: The model was not always accurate and employees sometimes needed the original sensitive data.

Resolution: Wealthsimple introduced self-hosted open-source LLMs so sensitive data could stay in its cloud environments.

A non-standard gateway API created integration rework.

Root cause: The initial internal API did not mirror the de facto OpenAI API standard.

Resolution: Wealthsimple shipped a v2 API that mirrored OpenAI's specs.

Two separate assistant surfaces damaged adoption.

Root cause: Employees had multiple places to satisfy GenAI needs, splitting attention and usage.

Resolution: Wealthsimple unshipped/combined earlier tool approaches as its platform matured.

Evaluation

Wealthsimple used adoption, message volume, weekly/monthly active users, surveys, client interviews, and production promotion of data applications as practical evaluation signals for GenAI productivity.

Metrics: over 2,200 daily messages · weekly active users close to one-third of company · monthly active users slightly over half of company · over seven applications in first two weeks · two applications reaching production · productivity survey/interview feedback

Outcomes

• The LLM gateway is used by over half the company
• Employees send over 2,200 daily messages across GenAI tools
• Close to one-third of the company are weekly active users
• Slightly over half the company are monthly active users
• The foundations for employee productivity enabled client-experience GenAI work

The source describes a 2023-2024 platform journey with gateway, redaction, self-hosted models, RAG API, data apps, multimodal inputs, and OpenAI-compatible API v2.