Enterprise-Wide Virtual Assistant for Employee Knowledge Access

bny_mellon · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise employee knowledge access in financial services

BNY Mellon needed to help roughly 50,000 employees find accurate policy, procedure, compliance, and operational knowledge without forcing them through traditional search results and long documents.

Constraints:

The bank operates in a sensitive financial-services environment with large volumes of internal procedures and compliance content.
Employees are spread across countries and need location-specific policy answers.
Knowledge was not uniformly curated, tagged, or structured for either humans or LLM retrieval.
Traditional search returned long documents rather than direct answers.
The assistant had to scale from pilots to all 50,000 employees while continuing to improve answer quality.

Architecture

BNY Mellon built an LLM-based virtual assistant on Google Cloud's Vertex AI ecosystem. Internal knowledge sources are ingested, chunked, tagged, and exposed through an assistant that answers employee questions directly rather than returning documents. The system evolved from department pilots into an enterprise rollout, adding Document AI for complex document formats, metadata feedback loops with content owners, and planned AI-based evaluation, structured-data coverage, and metrics dashboards.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  employee-knowledge-sources["Employee knowledge sources"] --> ingestion-and-chunking-pipeline["Ingestion and chunking pipeline"]
  document-ai-parser["Document AI parser"] --> ingestion-and-chunking-pipeline
  ingestion-and-chunking-pipeline --> virtual-assistant-retrieval-layer["Virtual assistant retrieval layer"]
  vertex-ai-model-layer["Vertex AI model layer"] --> virtual-assistant-retrieval-layer
  virtual-assistant-retrieval-layer --> employee-assistant-interface["Employee assistant interface"]
  virtual-assistant-retrieval-layer --> evaluation-and-metrics-loop["Evaluation and metrics loop"]
  evaluation-and-metrics-loop --> content-owner-feedback-loop["Content owner feedback loop"]
  content-owner-feedback-loop --> employee-knowledge-sources

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Employee knowledge sources`	Policies, procedures, compliance documents, department content, and structured data the assistant must answer from.	`Ingestion and chunking pipeline`
`Ingestion and chunking pipeline`	Loads internal knowledge and chunks it for retrieval, with strategy changes when document types break naive chunking.	`Virtual assistant retrieval layer`
`Document AI parser`	Parses complex documents such as cafeteria menus when regular LLM parsing struggles.	`Ingestion and chunking pipeline`
`Vertex AI model layer`	Provides the LLM experimentation and serving environment used to build the virtual assistant.	`Virtual assistant retrieval layer`
`Virtual assistant retrieval layer`	Combines retrieved internal knowledge with LLM responses to provide direct employee answers.	`Employee assistant interface`, `Evaluation and metrics loop`
`Employee assistant interface`	Enterprise-wide interface employees use to ask policy and knowledge questions.	—
`Evaluation and metrics loop`	Reviews answer quality, source performance, bias/context issues, and areas for model or content improvement.	`Content owner feedback loop`
`Content owner feedback loop`	Asks knowledge-management teams to improve tagging, metadata, and content structure for better retrieval.	`Employee knowledge sources`

Data flow

Employee policy and knowledge content is ingested into the assistant, chunked and enriched with metadata, and then retrieved for user questions. When documents are complex, Document AI helps parse the source before retrieval. User feedback and manual review identify weak answers, while planned metrics and AI evaluation loops feed improvements back to content owners and model tuning.

“we started to tap across all the knowledge that exist in the bank and started to get the information right”transcript, stated

“you know you need to add more metadata that's how the virtual assistant is becoming smarter and smarter”transcript, stated

Why this architecture

The bank chose an LLM assistant with retrieval over traditional search because employees needed answers from large policy documents, not links. Vertex AI gave the team a fast experimentation base, while staged pilots reduced rollout risk and metadata/content feedback made the assistant adapt to country-specific and department-specific policy needs.

“instead of you searching for something and get a document that has 100 Pages trying to find the answer how can I get you the answer you need”transcript, stated

“we already had inroads into the ver uh vertex AI ecosystem we had been using it for some time so it became very easy for us to experiment it”transcript, stated

Models used: LLM-based virtual assistant · Google Cloud Vertex AI · Google Document AI · Conversational AI

Scale

The assistant was rolled out across approximately 50,000 employees.
“yes we have roll it out across the bank um which which is 50,000 employees”transcript, stated
The company has operated for roughly 250 years.
“it's a 250 years old uh um Bank uh and we have been helping all our clients in managing their financial Ambitions”transcript, stated
The rollout started with one department and expanded to risk and then bank-wide coverage.
“we started a very small pilot with you know one department for example you know we we worked with our people experience team”transcript, stated

Alternatives considered (roads not taken)

Continue with traditional enterprise search.

Rejected because: Search returned long documents and left employees to locate the relevant policy answer themselves.

“we move away from traditional search such that instead of you searching for something and get a document that has 100 Pages trying to find the answer”transcript, stated

Treat ingestion as solved by simply loading and chunking all knowledge.

Rejected because: Different document types broke the initial chunking strategy and required parsing and strategy changes.

“initially we just thought that you know bring in all the knowledge let's do the chunking we are done only when we ran into a different kind of a document”transcript, stated

Rely only on unstructured documents.

Rejected because: The roadmap explicitly expands beyond unstructured content into structured data as another enterprise knowledge source.

“just not unstructured but even structure data we are going after it now”transcript, stated

Patterns

`rag-retrieval` (enterprise knowledge access)

The assistant retrieves internal policies and knowledge so employees get direct answers instead of long search-result documents.

“we build this virtual based assistant to help people find the information”transcript, stated

`data-quality-loop` (knowledge management)

Assistant failures feed back to knowledge-management and content-owner teams so documents gain tags and metadata the LLM can use.

“give the feedback to my knowledge management folks content owner folks that hey you need to start redefining you know how you are storing the content”transcript, stated

`eval-driven-development` (assistant quality)

The team plans to use AI and metrics to evaluate whether the assistant answers correctly, stays in context, and avoids bias.

“how do I apply AI itself to start understanding you know how well my virtual assistant is behaving you know is it answering right is it in the context you know is there a bias”transcript, stated

`canary-rollout` (enterprise rollout)

BNY Mellon expanded from small departmental pilots to risk content and then to bank-wide availability.

“that one pilot led to another and then slowly we started to expand”transcript, stated

Tradeoffs

Use a cloud AI platform for experimentation.

Chose Google Cloud Vertex AI over Building the assistant entirely on-prem or selecting an unproven model stack upfront.

Rationale: Existing Vertex AI usage made experimentation and model trials faster for the bank.

“which llm to use you know which cloud provider to use should we do it on Prem should we do offprem there were a lot of these questions”transcript, stated

Roll out gradually by department.

Chose Pilot departments before enterprise rollout over Immediate bank-wide content onboarding.

Rationale: Department pilots let the team learn ingestion, chunking, content, and policy problems before scaling to 50,000 employees.

“we started a very small pilot with you know one department for example you know we we worked with our people experience team”transcript, stated

Improve documents and metadata rather than treating model tuning as the only lever.

Chose Content-owner feedback and metadata enrichment over Relying only on the LLM to infer missing policy context.

Rationale: Country- and employee-specific policies require explicit metadata for the assistant to retrieve the right answer.

“llm doesn't know that necessarily unless the data has that information so it's a backward thing we started to work to give the feedback”transcript, stated

Gotchas (what bit them)

Naive chunking broke on real enterprise document variety.

Root cause: The team assumed loading and chunking all knowledge would be enough until unusual documents exposed parsing and chunking weaknesses.

Resolution: They changed the chunking strategy and used Document AI for complex documents.

“oops chunking strategy doesn't work so we had to change the chunking strategy right again for example we had a very uh interesting use case where that it's a cafeteria menu was very complex document”transcript, stated

Knowledge content was not automatically usable by LLM retrieval.

Root cause: Policies and documents were written and organized for humans who already knew where to look, not for models needing explicit context such as location.

Resolution: The assistant program pushed content owners to add tags and metadata.

“not everything was curated for the llm to work or search to work or even for normal human to to view the information”transcript, stated

Assistant quality review was still manual during rollout.

Root cause: The team had not yet automated evaluation of answer correctness, context, or bias.

Resolution: The roadmap adds AI-based assistant evaluation and metrics dashboards.

“whole feedback module or you know how well the virtual assistant is performing we still do it manually in terms of you know reviewing the question”transcript, stated

Evaluation

BNY Mellon had manual review of assistant questions at the time of the talk and planned to add AI-based evaluation plus metrics showing answer quality, source performance, improvement areas, and model-tuning needs.

Metrics: Answer correctness · Context fit · Bias checks · Knowledge-source performance · Areas of improvement · Model tuning signals

“how do we produce all the metrics to make sure that it's visible to everyone which knowledge source is working which knowledge source is not working what are our areas of improvement”transcript, stated

Outcomes

The assistant rolled out across BNY Mellon's roughly 50,000 employees.
The program exposed and improved document chunking, parsing, tagging, and metadata practices.
The roadmap expanded from unstructured knowledge to structured data, AI evaluation, and organization-wide metrics.

The transcript gives concrete deployment scale: an enterprise-wide rollout to 50,000 employees from departmental pilots inside a 250-year-old financial institution.

“yes we have roll it out across the bank um which which is 50,000 employees”transcript, stated

“second thing is we are obviously going across remaining content in the bank right just not unstructured but even structure data we are going after it now”transcript, stated