RAG-Based Industry Classification System for Financial Services

ramp · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services industry classification and business data quality

Ramp needed to replace fragmented, non-auditable industry classifications with a standardized system that could support compliance, risk assessment, sales targeting, portfolio monitoring, and product analytics.

Constraints:

• The old taxonomy combined third-party data, sales-entered information, and self-reported customer labels.
• Classifications were sometimes incorrect, overly broad, inconsistent across similar businesses, and hard to audit.
• The output had to be valid NAICS codes, not arbitrary LLM-generated categories.
• Ramp wanted control over model behavior, iteration speed, and costs in a business-critical workflow.

Architecture

Ramp built an in-house RAG industry classifier that migrates customer classifications to NAICS. The system embeds business queries and NAICS knowledge-base entries, retrieves likely code candidates with similarity scoring from ClickHouse-stored embeddings, and uses an LLM to rerank/select the final code from the filtered candidate set. Kafka logs intermediate results for diagnostics and iteration, while internal services run embedding and LLM prompt evaluations.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  business-record["Business record"] --> embedding-service["Embedding service"]
  business-record["Business record"] --> downstream-ramp-teams["Downstream Ramp teams"]
  naics-knowledge-base["NAICS knowledge base"] --> clickhouse-embedding-store["ClickHouse embedding store"]
  clickhouse-embedding-store["ClickHouse embedding store"] --> similarity-scorer["Similarity scorer"]
  embedding-service["Embedding service"] --> similarity-scorer["Similarity scorer"]
  similarity-scorer["Similarity scorer"] --> candidate-naics-recommendations["Candidate NAICS recommendations"]
  similarity-scorer["Similarity scorer"] --> kafka-diagnostic-log["Kafka diagnostic log"]
  candidate-naics-recommendations["Candidate NAICS recommendations"] --> llm-reranker["LLM reranker"]
  naics-knowledge-base["NAICS knowledge base"] --> llm-reranker["LLM reranker"]
  llm-reranker["LLM reranker"] --> final-naics-classification["Final NAICS classification"]
  naics-knowledge-base["NAICS knowledge base"] --> final-naics-classification["Final NAICS classification"]
  llm-reranker["LLM reranker"] --> kafka-diagnostic-log["Kafka diagnostic log"]
  kafka-diagnostic-log["Kafka diagnostic log"] --> evaluation-service["Evaluation service"]
  embedding-service["Embedding service"] --> evaluation-service["Evaluation service"]
  evaluation-service["Evaluation service"] --> llm-reranker["LLM reranker"]
  final-naics-classification["Final NAICS classification"] --> downstream-ramp-teams["Downstream Ramp teams"]

Components

Data flow

A business record enters the classifier. Internal embedding services encode the business query and compare it with precomputed NAICS embeddings stored in ClickHouse. Similarity scoring produces candidate recommendations. The LLM receives those candidates and chooses a final valid NAICS code rather than inventing a free-form category. Kafka captures intermediate retrieval and selection results for diagnosis, and downstream compliance, risk, sales, portfolio, and analytics workflows consume the standardized classification.

Why this architecture

Ramp used RAG because retrieval could narrow a large standardized taxonomy to plausible candidates, while an LLM could make the final semantic choice. Building the system in-house gave Ramp control over costs, behavior, diagnostics, and iteration, and constraining output to NAICS codes made the classifier auditable enough for compliance and risk workflows.

Models used: Embedding model · LLM reranker · RAG classifier

Scale

• Retrieval metrics improved by up to 60%.
  “up to 60% in retrieval metrics”summary, stated
• Final prediction accuracy improved by 5-15%.
  “5-15% in final prediction accuracy”summary, stated
• The classifier targets standardized NAICS codes.
  “standardized NAICS-based system powered by an in-house Retrieval-Augmented Generation (RAG) model.”summary, stated
• The old Professional Services category spanned law firms, dating apps, and consulting firms.
  “a category so broad it included law firms, dating apps, and consulting firms”summary, stated
• The classification supports at least four named business functions.
  “cutting across compliance requirements, portfolio monitoring, sales targeting, and product analytics.”summary, stated

Alternatives considered (roads not taken)

Fragmented homegrown taxonomy

Rejected because: It created incorrect, generic, inconsistent, and non-auditable categories.

Third-party classification solutions

Rejected because: Ramp wanted control over behavior, iteration speed, and costs for complex business data.

Open-ended LLM category generation

Rejected because: A valid-code candidate set made the output auditable and constrained.

Only final-output logging

Rejected because: Ramp needed intermediate results to diagnose and iterate on pathological cases.

Patterns

rag-retrieval (retrieval)

Ramp retrieves likely NAICS candidates from an embedded knowledge base before LLM selection.

retrieval-ranking (retrieval)

Similarity scoring and LLM reranking turn a broad taxonomy into ranked, filtered candidates.

structured-output-contract (interface)

The output contract is a valid NAICS code, not a free-form LLM label.

observability-tracing (operations)

Kafka captures intermediate classifier state for diagnostics and prompt iteration.

eval-driven-development (quality)

Ramp reports retrieval and final prediction accuracy improvements and runs internal embedding/LLM prompt evaluations.

Tradeoffs

Build a custom in-house model

Chose In-house RAG classifier over Third-party classification services.

Rationale: Ramp wanted control over complex data behavior, iteration speed, and costs.

Use retrieval plus reranking

Chose Embeddings-based retrieval followed by LLM-based re-ranking over Direct LLM classification.

Rationale: Retrieval constrains the LLM to valid taxonomy candidates while preserving semantic selection.

Standardize on NAICS

Chose External standardized taxonomy over Homegrown business categories.

Rationale: NAICS made industry data more consistent and communicable across teams and partners.

Gotchas (what bit them)

A stitched taxonomy created inconsistent business decisions.

Root cause: Different teams relied on third-party data, sales-entered data, and self-reported categories with no shared standard.

Resolution: Migrate industry classification to a NAICS-based system consumed across teams.

Broad categories were not actionable for risk or sales.

Root cause: A category like Professional Services grouped unrelated businesses.

Resolution: Use NAICS retrieval and LLM selection to produce more specific industry codes.

Open-ended classification would undermine auditability.

Root cause: LLMs can generate labels outside the official taxonomy.

Resolution: Constrain generation to candidates drawn from the NAICS knowledge base.

Pathological cases need intermediate diagnostics.

Root cause: Final labels alone do not show whether retrieval, candidate ranking, or LLM selection failed.

Resolution: Log intermediate results via Kafka for diagnostics and iteration.

Evaluation

Ramp evaluated the system with retrieval metrics, final prediction accuracy, internal embedding and prompt evaluations, and diagnostic analysis of intermediate retrieval/reranking results.

Metrics: retrieval metric improvement · final prediction accuracy · prompt evaluation results · pathological-case diagnostics · cross-team classification consistency

Outcomes

• Ramp replaced fragmented homegrown classifications with standardized NAICS codes.
• Retrieval metrics improved by up to 60%.
• Final prediction accuracy improved by 5-15%.
• Ramp gained more control over classification behavior, cost, and cross-team usage.

Concrete anchors include up to 60% retrieval-metric improvement, 5-15% final-prediction improvement, three RAG stages, ClickHouse retrieval storage, Kafka diagnostics, and downstream use across compliance, risk, sales, and analytics.