Building Internal LLM Tools with Security and Privacy Focus

wealthsimple · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal fintech LLM platform

Wealthsimple wanted employees to use LLMs for productivity and operations without leaking sensitive company, customer, or code data to external providers. The platform had to make secure usage easy, preserve optionality across models and techniques, support RAG and multimodal workflows, and help a very small ML/platform team prioritize what to build versus buy.

Constraints:

• The company works in fintech, so security, privacy, PII handling, and vendor trust were first-order constraints.
• Employees needed optionality across external hosted models, self-hosted open-source models, multimodal models, and RAG tools.
• PII redaction helped external-provider use but interfered with legitimate workflows that needed unmasked PII.
• The team had to support internal productivity tools and production ML/LLM use cases with only three ML engineering people.
• Multiple GenAI tools created a confusing user experience as most people stuck to one tool.
• Hallucination guardrails were mostly educational and grounding-based rather than fully integrated technical checks.

Architecture

Wealthsimple built an internal LLM platform around a central LLM Gateway. Employees use the Gateway to select external models, self-hosted models, or multimodal Gemini models. Inputs bound for external providers pass through an in-house PII redaction model, while self-hosted models run inside Wealthsimple's cloud/VPC without PII masking. The platform also supports conversation checkpointing across models, multimodal file/image/audio inputs, self-hosted open-source LLM deployment, and Booster Pack, a RAG tool with public, private, and limited knowledge bases. For operations optimization, they extended a Transformer-based ticket triage workflow with self-hosted Whisper transcription and self-hosted LLM-generated metadata.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  employee-user[Employee user] --> llm-gateway[LLM Gateway]
  employee-user --> booster-pack-rag[Booster Pack RAG]
  llm-gateway --> pii-redaction-model[PII redaction model]
  pii-redaction-model --> external-genai-providers[External GenAI providers]
  llm-gateway --> self-hosted-llms[Self-hosted LLMs]
  llm-gateway --> gemini-multimodal-models[Gemini multimodal models]
  llm-gateway --> conversation-checkpoint-store[Conversation checkpoint store]
  booster-pack-rag --> knowledge-bases[Knowledge bases]
  nightly-refresh-jobs[Nightly refresh jobs] --> knowledge-bases
  internal-data-applications-platform[Internal data applications platform] --> booster-pack-rag
  internal-data-applications-platform --> employee-user
  client-experience-triage-workflow[Client experience triage workflow] --> transformer-ticket-classifier[Transformer ticket classifier]
  client-experience-triage-workflow --> whisper-transcription-model[Whisper transcription model]
  client-experience-triage-workflow --> self-hosted-llms
  client-experience-triage-workflow --> cx-agents-and-annotators[CX agents and annotators]
  ml-engineering-platform-team[ML engineering platform team] --> llm-gateway
  ml-engineering-platform-team --> self-hosted-llms
  ml-engineering-platform-team --> booster-pack-rag
  ml-engineering-platform-team --> client-experience-triage-workflow

Components

Data flow

An employee sends a prompt, document, image, or audio input into the LLM Gateway. The user selects a model from external providers or self-hosted models. For external GenAI partners, the Gateway sends the input through Wealthsimple's PII redaction model before forwarding it; for self-hosted models, the data remains in the company cloud and is not redacted. Multimodal requests use Gemini 1.5's long context to extract file or media content into the prompt. RAG use cases go through Booster Pack, where uploaded or nightly refreshed knowledge bases are indexed and queried as public, private, or limited sources. In client-experience triage, tickets or transcribed voice calls flow through an existing Transformer classifier and are enriched by self-hosted LLM metadata for agents and annotators.

Why this architecture

The platform makes the secure path the easiest path. A Gateway gives employees model optionality while inserting PII redaction before external calls. Self-hosted models preserve sensitive workflows inside Wealthsimple's cloud. Booster Pack separates retrieval-grounded work from multimodal prompt enrichment. The build-vs-buy strategy keeps internal investment focused on security/privacy, speed/cost, and proprietary-data leverage rather than buying every vendor's GenAI integration.

Models used: GPT-4 · GPT-3.5 Turbo · Gemini 1.5 · Cohere models · Llama 3 · self-hosted open-source LLMs · self-hosted code-specialized models · Whisper · Transformer-based in-house classifier

Scale

• The LLM Gateway was used internally by over half of the company.
  “around a year ago we developed an open source our llm Gateway which is used internally by over half of the company”transcript, stated
• Wealthsimple had deployed four open-source LLMs inside its own cloud environment.
  “today we've deployed four open source llms this way and we've built up the platform support for fine-tuning and mod training with Hardware acceleration”transcript, stated
• Gemini 1.5 was selected partly for 1-2 million token context windows.
  “Gemini 1.5 has 1 million or 2 million um they can accept 1 to two million tokens”transcript, stated
• Daily, weekly, and monthly Gateway usage were tracked from around the prior September and were trending upward.
  “these are our daily active users weekly active users and then monthly active users since last September that's around the time we started actually tracking these metrics”transcript, stated

  “our daily weekly monthly usage is increasing time over time”transcript, stated
• Close to 50% of LLM adoption came from R&D teams.
  “at well simple close to 50% of this adoption is driven by teams within research and development”transcript, stated
• The first self-hosted model deployment took two weeks, while the latest took 20 minutes.
  “the first cellos model that we deployed it took two weeks but then the last one took 20 minutes”transcript, stated
• The ML engineering/platform team consisted of only three people.
  “on the ml engineering front it's a very small team there's only actually three people”transcript, stated
• The initial LLM Gateway took five days to build, followed by a few hundred hours of iteration.
  “with the LM Gateway it took us 5 days to build it initially and then probably a few hundred hours afterwards just iterating improving”transcript, stated

Alternatives considered (roads not taken)

Let employees use public ChatGPT/OpenAI directly.

Rejected because: The team was worried about fourth-party data sharing and oversharing sensitive information, so they inserted a Gateway and redaction path before external models.

Redact PII for every model call.

Rejected because: Self-hosted models run inside the company cloud, so redaction is skipped there to preserve workflows that need PII while keeping data inside the environment.

Buy every vendor's GenAI integration.

Rejected because: The speaker compared the vendor landscape to streaming subscriptions: many useful integrations exist, but buying all of them is not economical and must be weighed against time-to-market, cost, and proprietary-data leverage.

Treat multimodal upload and retrieval grounding as one tool pattern.

Rejected because: The team described Booster Pack's RAG grounding as distinct from multimodal prompt enrichment, while also planning better consolidation for user experience.

Force GenAI into every operational workflow.

Rejected because: The team explicitly wanted organic extensions where GenAI addressed a real gap, as in voice-ticket triage and metadata enrichment.

Patterns

privacy-preserving-processing (security)

External model calls pass through an in-house PII redaction model, while sensitive workflows can use self-hosted models whose data stays inside Wealthsimple's cloud/VPC.

model-routing (model-operations)

The Gateway lets users choose among external provider models and self-hosted models, including GPT, Cohere, Gemini, Llama 3, and code-specialized models.

rag-retrieval (retrieval)

Booster Pack grounds conversations against uploaded or maintained knowledge bases, including public, private, and limited sources.

tool-use-orchestration (llm-platform)

The internal platform offers a suite of tools, including Gateway, Booster Pack, data-app prototyping, multimodal uploads, checkpointing, and production triage extensions, rather than one monolithic chat UI.

guardrails-io-filtering (safety)

The platform masks PII before external calls and bakes prompt guidance about masking into production prompts.

data-quality-loop (operations)

Self-hosted LLM generations enrich ticket classifications and support annotator workflows as new model iterations are evaluated.

Tradeoffs

Use a Gateway with redaction for external models.

Chose central Gateway plus PII redaction before external-provider calls over unmediated employee use of public model providers.

Rationale: The Gateway let Wealthsimple give employees accessible LLM usage while reducing fourth-party data-sharing risk and oversharing of sensitive information.

Support both external and self-hosted models.

Chose model optionality across external partners and self-hosted cloud/VPC models over one model/provider for every task.

Rationale: No single model or technique fit every use case, and self-hosting preserved workflows that could not tolerate PII masking.

Build internal tools while the vendor landscape was immature.

Chose building Gateway, Booster Pack, and platform guardrails over buying all available AI vendor integrations.

Rationale: Building created expertise, guardrails, and confidence, but the team acknowledged improved vendor security could make buying more attractive over time.

Use Gemini multimodal inputs through prompt enrichment.

Chose extracting multimodal content into the prompt using Gemini 1.5 long context over requiring users to manually translate screenshots, files, images, or audio into text.

Rationale: The team saw manual extraction from screenshots and documents as toil, and long-context multimodal models reduced friction for technical and non-technical users.

Improve ticket triage by extending an existing ML workflow.

Chose Whisper transcription and self-hosted LLM metadata enrichment over replacing the existing Transformer ticket-classification workflow wholesale.

Rationale: The existing classifier already improved routing; GenAI added coverage for voice tickets and richer metadata where it naturally fit gaps in the workflow.

Gotchas (what bit them)

PII redaction can make legitimate work impossible when employees need the sensitive fields.

Root cause: Masking customer or company data before external-provider calls also obscures information required for some internal workflows.

Resolution: Offer self-hosted models inside the cloud/VPC for sensitive workflows where data should remain unredacted.

More internal GenAI tools can increase cognitive load instead of adoption.

Root cause: Users had to visit multiple places and choose among overlapping tools such as Gateway multimodal support and Booster Pack.

Resolution: Consolidate tools and abstract the cognitive complexity of selecting the right tool for the job.

External GenAI vendor sprawl makes buy decisions economically and strategically messy.

Root cause: Many vendors added bespoke AI integrations, so purchasing every integration would recreate the streaming-platform problem.

Resolution: Evaluate build-vs-buy through security/privacy, time-to-market/cost, and proprietary-data leverage.

Hallucination and reliability were not fully handled by retrieval-system guardrails.

Root cause: The team had not integrated additional hallucination checks into retrieval systems, relying instead on education, grounding, prompts, and future fine-tuning.

Resolution: Use grounding and user education now, but treat technical hallucination checks as an acknowledged gap.

A tiny platform team has to balance new GenAI tools, model maintenance, production use cases, and the broader ML platform.

Root cause: Only three ML engineering people were responsible for both GenAI and non-GenAI ML platform work.

Resolution: Automate model deployment and drive roadmap decisions from stakeholders, end-user interviews, team interest, and business priorities.

Evaluation

Wealthsimple evaluated the platform primarily through adoption metrics, surveys, user interviews, engagement trends, use-case sampling, and production workflow impact. They tracked daily/weekly/monthly Gateway use, surveyed both users and non-users, studied engagement patterns, and used production triage outcomes such as ticket coverage and annotator support. The source also explicitly acknowledges that hallucination checks were not yet integrated into retrieval systems.

Metrics: daily active users · weekly active users · monthly active users · share of company using the Gateway · adoption share from R&D teams · top use-case categories · survey-reported productivity lift · self-hosted model deployment time · Gemini rate-limit headroom · ticket triage coverage

Outcomes

• Wealthsimple built an internal LLM Gateway used by over half the company.
• The platform shipped in-house PII redaction, four self-hosted open-source LLMs, multimodal Gateway support, and model-selection optionality.
• Booster Pack provided RAG grounding over public, private, and limited knowledge bases.
• Gateway engagement metrics trended upward across daily, weekly, and monthly active users.
• Surveyed users reported significant productivity increases, with top use cases in programming support, content generation, and information retrieval.
• The production ticket-triage workflow expanded from text tickets to voice calls using self-hosted Whisper and self-hosted LLM metadata enrichment.
• Deployment automation reduced self-hosted model rollout time from two weeks to 20 minutes.

The case reports Gateway usage by over half the company, four deployed open-source LLMs, Gemini 1.5 context windows of 1-2 million tokens, close to 50% of adoption from R&D, an initial five-day Gateway build plus hundreds of iteration hours, a self-hosted model deployment improvement from two weeks to 20 minutes, and a three-person ML engineering/platform team.