Scaling ESG Compliance Analysis with RAG and Vector Search

intellectai · Finance · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: ESG compliance analysis for investment decisions

IntellectAI needed to help a sovereign wealth fund analyze ESG compliance across thousands of companies and millions of multi-format documents.

Constraints:

The fund manages over $1.5 trillion across 9,000 companies.
The system had to process almost 10 million documents in more than 30 text and image formats.
Investment decisions required higher-than-typical RAG accuracy and multilingual global coverage.

Architecture

IntellectAI built Purple Fabric on MongoDB Atlas and Atlas Vector Search. The platform prepares structured and unstructured enterprise, policy, market, regulatory, and tacit-knowledge data for RAG; ingests hundreds of millions of vectors; uses MongoDB schema flexibility, multimodality, dynamic linking, and time-series collections; and powers an AI Expert Agent System for ESG insights.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Enterprise and ESG data sources"] --> B["Purple Fabric ingestion"]
  B --> C["MongoDB Atlas"]
  C --> D["Atlas Vector Search"]
  C --> E["Time series collections"]
  D --> F["AI Expert Agent System"]
  F --> G["Analyst workflow"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Enterprise and ESG data sources`	Structured and unstructured data, policies, market data, regulatory information, tacit knowledge, reports, text, and images.	`Purple Fabric ingestion`
`Purple Fabric ingestion`	Collects, transforms, and prepares data for RAG and vector search.	`Enterprise and ESG data sources`, `MongoDB Atlas`
`MongoDB Atlas`	Flexible operational database and data platform for multimodal ESG records.	`Purple Fabric ingestion`, `Atlas Vector Search`, `Time series collections`
`Atlas Vector Search`	Retrieves high-dimensional vectorized ESG evidence for RAG.	`MongoDB Atlas`, `AI Expert Agent System`
`Time series collections`	Process company reports over multiple years to extract trends and performance metrics.	`MongoDB Atlas`
`AI Expert Agent System`	Generates ESG compliance insights and goal-driven outcomes from retrieved knowledge.	`Atlas Vector Search`, `Analyst workflow`
`Analyst workflow`	Investment and ESG analysts use outputs for responsible investment decisions.	`AI Expert Agent System`

Data flow

Purple Fabric collects structured/unstructured enterprise and regulatory data, converts it into RAG-ready knowledge, stores vectors and flexible records in MongoDB Atlas, uses Atlas Vector Search to retrieve relevant evidence, and feeds the AI Expert Agent System that produces ESG compliance insights for analysts and investment decisions.

3 supporting quotes

“Purple Fabric helps transform enterprise data into actionable AI insights and solutions by making data ready for retrieval-augmented generation (RAG).”article, stated

“The platform collects and analyzes structured and unstructured enterprise data, policies, market data, regulatory information, and tacit knowledge”article, stated

“We ingested hundreds of millions of vectors across these documents, and this is where we truly understood the power of MongoDB.”article, stated

Why this architecture

The architecture centers on MongoDB Atlas because ESG analysis needed flexible schemas, multimodal data, dynamic data linking, vector search, and scale across many document formats and companies.

3 supporting quotes

“MongoDB Atlas and Vector Search give us flexibility around the schema and how we can turn particular data into knowledge”article, stated

“there are four unique advantages of working with MongoDB—particularly using MongoDB Atlas Vector Search—that other companies should consider when building long-term AI strategies: a unified data model, multimodality, dynamic data linking, and simplicity.”article, stated

“If you want to do it at scale, you need proper enterprise support, and that’s where MongoDB became really handy for us.”article, stated

Models used: RAG models · AI Expert Agent System · MongoDB Atlas Vector Search

Scale

The target fund manages over $1.5 trillion across 9,000 companies.
“one of the largest sovereign wealth funds in the world, which manages over $1.5 trillion across 9,000 companies.”article, stated
The system processed almost 10 million documents in more than 30 formats.
“We had to process almost 10 million documents in more than 30 different data formats—text and image”article, stated
The platform ingested hundreds of millions of vectors.
“We ingested hundreds of millions of vectors across these documents”article, stated
Coverage expanded from 100-150 companies to over 8,000 companies.
“Previously, analysts could examine only between 100 and 150 companies. With MongoDB Atlas and Atlas Vector Search, Purple Fabric can now process information from over 8,000 companies”article, stated
The system delivers more than 90% accuracy compared with typical 80-85% RAG accuracy.
“Generally, RAG will probably give you 80% to 85% accuracy. But in our case, we are talking about a fund deciding whether to invest billions or not in a company, so the accuracy should be 90% minimum”article, stated
The workflow performs the analyst work almost a thousand times faster.
“We are able to give 100% coverage and do what the ESG analysts were able to do for this organization almost a thousand times faster”article, stated

Alternatives considered (roads not taken)

Manual analyst coverage

Rejected because: Analysts could only cover 100-150 companies, far short of the fund's global portfolio.

“Previously, analysts could examine only between 100 and 150 companies.”article, stated

Typical 80-85% RAG accuracy

Rejected because: Investment decisions involving billions required at least 90% accuracy.

“we are talking about a fund deciding whether to invest billions or not in a company, so the accuracy should be 90% minimum”article, stated

Simple search

Rejected because: The use case needed contextual high-dimensional ESG analysis over structured and unstructured data.

“What we are doing is not ‘simple search’; it is very contextual, and MongoDB helps us provide that high-dimension data.”article, stated

Patterns

`rag-retrieval` (retrieval)

Purple Fabric prepares enterprise data for retrieval-augmented generation and uses vector search to ground ESG analysis.

“making data ready for retrieval-augmented generation (RAG).”article, stated

`tool-use-orchestration` (agent-architecture)

The AI Expert Agent System combines retrieved structured, unstructured, regulatory, and tacit-knowledge sources into goal-driven ESG outcomes.

“The platform collects and analyzes structured and unstructured enterprise data, policies, market data, regulatory information, and tacit knowledge to enable its AI Expert Agent System to achieve precise, goal-driven outcomes with accuracy and speed.”article, stated

`hybrid-search` (retrieval)

The platform correlates structured and unstructured data across text and image formats for ESG insights.

““We had to process almost 10 million documents in more than 30 different data formats—text and image—and correlate both structured and unstructured data to provide those particular hard-to-find insights,” said Dastrala.”article, stated

Tradeoffs

Analysis automation

Chose RAG and vector-search platform over manual analyst review alone.

Rationale: The automated platform expands coverage and speed far beyond manual analyst capacity.

“We are able to give 100% coverage and do what the ESG analysts were able to do for this organization almost a thousand times faster”article, stated

Data platform

Chose MongoDB Atlas and Atlas Vector Search over less flexible search/storage setup.

Rationale: MongoDB provided flexible schemas, vector search, multimodality, dynamic linking, and operational support.

“MongoDB Atlas and Vector Search give us flexibility around the schema and how we can turn particular data into knowledge”article, stated

Accuracy target

Chose 90% minimum accuracy over typical 80-85% RAG accuracy.

Rationale: Billion-dollar investment decisions required stricter accuracy than generic RAG applications.

“But in our case, we are talking about a fund deciding whether to invest billions or not in a company, so the accuracy should be 90% minimum,” said Dastrala.”article, stated

Gotchas (what bit them)

Manual ESG analysis did not scale to the portfolio.

Root cause: Analysts could cover only 100-150 companies while the fund spans thousands.

Resolution: Use Atlas Vector Search and RAG to process information from over 8,000 companies.

“Previously, analysts could examine only between 100 and 150 companies. With MongoDB Atlas and Atlas Vector Search, Purple Fabric can now process information from over 8,000 companies”article, stated

ESG evidence spans heterogeneous formats and modalities.

Root cause: The corpus includes structured/unstructured data, text, images, reports across years, and many document formats.

Resolution: Use MongoDB Atlas flexibility, multimodality, vector search, and time series collections.

“process almost 10 million documents in more than 30 different data formats—text and image—and correlate both structured and unstructured data”article, stated

Typical RAG accuracy was insufficient for high-stakes investment decisions.

Root cause: The fund uses the output to decide whether to invest billions.

Resolution: Target at least 90% accuracy rather than typical 80-85% RAG results.

“Generally, RAG will probably give you 80% to 85% accuracy. But in our case, we are talking about a fund deciding whether to invest billions or not”article, stated

Evaluation

IntellectAI reports accuracy, company coverage, document/vector scale, speed versus analysts, and multilingual/global coverage as evaluation dimensions.

Metrics: AI accuracy · company coverage · documents processed · vectors ingested · speed versus analysts · language/global coverage

““If you want to do it at scale, you need proper enterprise support, and that’s where MongoDB became really handy for us. We are able to give 100% coverage and do what the ESG analysts were able to do for this organization almost a thousand times faster,” said Dastrala.”article, stated

Outcomes

Coverage expanded from 100-150 companies to over 8,000 globally.
The system processed almost 10 million documents in more than 30 formats.
It ingested hundreds of millions of vectors.
The platform reports more than 90% accuracy and almost 1,000x speed versus analyst work.

The article is a MongoDB customer story, so promotional bias is high; the extraction keeps claims attributed to the source.

“Purple Fabric can now process information from over 8,000 companies across the world, covering different languages and delivering more than 90% accuracy.”article, stated

““We ingested hundreds of millions of vectors across these documents, and this is where we truly understood the power of MongoDB.””article, stated