RAG-Based System for Climate Finance Document Analysis

climatealigned · Finance · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Climate finance document analysis and investment screening

ClimateAligned needed to assess whether financial instruments align with climate goals by extracting reliable evidence from complex corporate climate disclosures, under startup resource constraints and high accuracy expectations from financial institutions.

Constraints:

Architecture

ClimateAligned built a RAG workflow around GPT-4 and hybrid search. An in-house ingestion layer collects corporate climate documents into a structured dataset. Retrieval combines vector similarity with BM25 keyword search to supply source passages to GPT-4, which proposes answers and citations for analysts. Human reviewers verify outputs before production, creating labeled examples. As the dataset grew, ClimateAligned added classifiers for review flagging and topic classification, enabling some outputs to bypass full review while preserving production accuracy.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  climate-document-corpus["Climate document corpus"] --> in-house-ingestion-pipeline["In-house ingestion pipeline"]
  climate-document-corpus["Climate document corpus"] --> hybrid-search-retriever["Hybrid search retriever"]
  in-house-ingestion-pipeline["In-house ingestion pipeline"] --> postgres-data-store["Postgres data store"]
  postgres-data-store["Postgres data store"] --> hybrid-search-retriever["Hybrid search retriever"]
  hybrid-search-retriever["Hybrid search retriever"] --> gpt-4-answer-generator["GPT-4 answer generator"]
  gpt-4-answer-generator["GPT-4 answer generator"] --> human-analyst-review["Human analyst review"]
  human-analyst-review["Human analyst review"] --> labeled-dataset["Labeled dataset"]
  labeled-dataset["Labeled dataset"] --> review-routing-classifier["Review-routing classifier"]
  review-routing-classifier["Review-routing classifier"] --> human-analyst-review["Human analyst review"]
  labeled-dataset["Labeled dataset"] --> topic-classifier["Topic classifier"]
  topic-classifier["Topic classifier"] --> topic-based-ranking["Topic-based ranking"]
  topic-based-ranking["Topic-based ranking"] --> hybrid-search-retriever["Hybrid search retriever"]
  labeled-dataset["Labeled dataset"] --> fine-tuning-pipeline["Fine-tuning pipeline"]
  fine-tuning-pipeline["Fine-tuning pipeline"] --> gpt-4-answer-generator["GPT-4 answer generator"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Climate document corpus Corporate climate disclosures and related documents gathered from the internet. In-house ingestion pipeline, Hybrid search retriever
In-house ingestion pipeline Document and data management layer that stores climate documents in a structured dataset. Climate document corpus, Postgres data store
Postgres data store Application data storage used by the Python-based system. In-house ingestion pipeline, Hybrid search retriever, Labeled dataset
Hybrid search retriever Combines vector similarity and BM25 keyword search to retrieve candidate source passages. Climate document corpus, Postgres data store, GPT-4 answer generator
GPT-4 answer generator Synthesizes answers about climate alignment from retrieved document context. Hybrid search retriever, Human analyst review
Human analyst review Expert review gate that verifies proposed answers before production. GPT-4 answer generator, Labeled dataset, Review-routing classifier
Labeled dataset Accumulated correct and incorrect examples from reviewed answers. Human analyst review, Review-routing classifier, Topic classifier, Fine-tuning pipeline
Review-routing classifier Flags which model outputs need human review and which can proceed directly. Labeled dataset, Human analyst review
Topic classifier Classifies climate-document topics to improve throughput and document ranking. Labeled dataset, Topic-based ranking
Topic-based ranking Planned ranking layer for selecting documents by topic relevance. Topic classifier, Hybrid search retriever
Fine-tuning pipeline Future path for smaller models trained from accumulated labels. Labeled dataset, GPT-4 answer generator

Data flow

Corporate climate documents are ingested from the internet into ClimateAligned's document store. A user assessment question triggers hybrid retrieval over document chunks using vector similarity and BM25 keyword search. Retrieved passages and the question are passed to GPT-4, which synthesizes proposed answers with source evidence. Human analysts review the proposed answers before production and their decisions become labeled examples. Once enough labels accumulated, classifiers route uncertain outputs to review, classify topics, and support future fine-tuning or smaller-model work.

3 supporting quotes
“we're consuming documents from the internet and storing them in a structured dataset within our system.”article, stated
“Starting with GPT-4 and hybrid search (vector similarity + BM25 keyword search), they achieved 85% model accuracy”summary, stated
“Every reviewed answer becomes part of a labeled dataset of correct and incorrect examples.”summary, stated

Why this architecture

ClimateAligned used RAG because climate finance decisions require source accountability and up-to-date document evidence, not unsupported model knowledge. Human review kept production accuracy at the expected standard while still reducing analyst time. The labeled-data flywheel then let the small team add routing, topic classification, and future fine-tuning without needing a perfect model on day one.

3 supporting quotes
“Because this is financial decisions that need to be made, accuracy is quite important, and trust in that accuracy is quite important”article, stated
“Even with the underlying model achieving only 85% accuracy, placing human review between model outputs and production means the actual product maintains that 99% accuracy standard.”summary, stated
“You can build a highly accurate production system around a moderately accurate model by adding appropriate review, validation, and routing mechanisms.”summary, stated

Models used: GPT-4 · Hybrid search · BM25 · Review-routing classifier · Topic classifier

Scale

Alternatives considered (roads not taken)

Manual expert review for every assessment from scratch

Rejected because: It took roughly 2 hours per company and limited throughput.

“a task traditionally performed manually by expert analysts requiring approximately 2 hours per company assessment.”summary, stated

Let GPT-4 answers go directly to production

Rejected because: The core model was only 85% accurate, below the 99% production standard expected for financial decisions.

“Even with the underlying model achieving only 85% accuracy, placing human review between model outputs and production means the actual product maintains that 99% accuracy standard.”summary, stated

Use only semantic vector search

Rejected because: The initial production retrieval used hybrid search, combining semantic similarity with BM25 keyword matching.

“hybrid search (vector similarity + BM25 keyword search)”summary, stated

Continue relying only on OpenAI APIs

Rejected because: The team wanted to reduce costs and improve reliability by fine-tuning smaller models and moving away from OpenAI APIs.

“moving away from OpenAI APIs to reduce costs and improve reliability.”summary, stated

Patterns

rag-retrieval (retrieval)

ClimateAligned retrieves source passages from climate documents and asks GPT-4 to synthesize grounded answers.

“We're going to be answering it using a RAG-based system. We're going to take you through that journey of doing that over the last year at an early-stage startup.”article, stated

hybrid-search (retrieval)

The retrieval layer combines vector similarity with BM25 keyword search.

“Starting with GPT-4 and hybrid search (vector similarity + BM25 keyword search), they achieved 85% model accuracy while maintaining 99% production accuracy through human-in-the-loop review.”summary, stated

human-in-the-loop-gate (governance)

Human analysts review model outputs before production to maintain financial-grade accuracy.

“placing human review between model outputs and production means the actual product maintains that 99% accuracy standard.”summary, stated

data-quality-loop (quality)

Every human-reviewed answer becomes labeled data for validation, routing, and future fine-tuning.

“Every reviewed answer becomes part of a labeled dataset of correct and incorrect examples.”summary, stated

confidence-thresholding (routing)

A classifier routes uncertain or likely incorrect outputs to human review while allowing confident outputs to proceed.

“built a classifier to flag which model outputs require human review and which can proceed directly to production.”summary, stated

fine-tuning-adaptation (model-improvement)

The 10,000-example labeled dataset creates the option to fine-tune smaller models and reduce dependency on OpenAI APIs.

“The speaker identifies key dataset size thresholds: around 500 examples enables validation testing and few-shot learning experiments, while 10,000 examples (their current scale) opens the door to model fine-tuning.”summary, stated

Tradeoffs

Use human review after model generation

Chose 99% production accuracy with analyst review over Fully automated 85%-accurate model outputs.

Rationale: Financial decisions required trust and accountability above raw model accuracy.

“accuracy is quite important, and trust in that accuracy is quite important, both for our users and for ourselves from a liability perspective.”article, stated

Start with GPT-4 and OpenAI APIs

Chose Fast initial value from a frontier model over Training smaller internal models immediately.

Rationale: GPT-4 let the small team create useful outputs and labeled data before investing in fine-tuning.

“Starting with GPT-4 and hybrid search (vector similarity + BM25 keyword search), they achieved 85% model accuracy while maintaining 99% production accuracy through human-in-the-loop review.”summary, stated

Route only uncertain outputs to humans after dataset growth

Chose Review-routing classifier over Full human review forever.

Rationale: Selective review created another order-of-magnitude throughput improvement while maintaining accuracy standards.

“This achieved roughly another order of magnitude improvement in throughput while maintaining production accuracy standards.”summary, stated

Gotchas (what bit them)

Model accuracy and product accuracy are different metrics.

Root cause: The core LLM can be only moderately accurate while the surrounding workflow catches errors before production.

Resolution: Add review, validation, and routing mechanisms around the model.

“The key insight here is that the overall system accuracy and the core model accuracy are different metrics. You can build a highly accurate production system around a moderately accurate model by adding appropriate review, validation, and routing mechanisms.”summary, stated

Chunking loses document-structure context.

Root cause: Half-page chunks can look linguistically similar even when they appear in different sections with different meaning.

Resolution: Work on topic-based document ranking and topic classification to recover context before retrieval.

“when chunking documents into approximately half-page segments for retrieval, important contextual information about where that chunk appears in the document structure is lost.”summary, stated

Startup resource constraints require staged value delivery.

Root cause: The team had limited runway, engineers, compute, and labeled data.

Resolution: Start with GPT-4 plus RAG and human review, then use accumulated labels for routing and future fine-tuning.

“successful LLMOps doesn't necessarily require large teams, massive compute budgets, or enormous datasets from the outset.”summary, stated

Evaluation

ClimateAligned evaluated the core model at 85% accuracy, production workflow at 99% accuracy through review, analyst productivity through time-per-company, and scaling progress through labeled dataset size and review-routing throughput.

Metrics: model accuracy · production accuracy · time per company assessment · labeled example count · review-routing throughput · human review rate

“achieved 85% model accuracy while maintaining 99% production accuracy through human-in-the-loop review.”summary, stated
“assessment time dropped from 2 hours to approximately 20 minutes per company.”summary, stated

Outcomes

Concrete anchors include a 10-person company with 3 technical staff, 85% model accuracy, 99% production accuracy, 2 hours to 20 minutes per company, 500 examples for validation/few-shot experiments, and 10,000 examples for fine-tuning.

“reduced analysis time from 2 hours to 20 minutes per company while building labeled datasets (reaching 10,000 examples)”summary, stated
“enabled automated review flagging and topic classification, resulting in additional order-of-magnitude throughput gains.”summary, stated