RAG-Based Industry Classification System for Financial Services

ramp · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services industry classification and taxonomy standardization

Ramp needed to replace a homegrown industry taxonomy that produced inconsistent, overly broad, non-auditable classifications and complex cross-taxonomy translations across risk, sales, compliance, and analytics workflows.

Constraints:

• Different teams used incompatible taxonomies including homegrown categories, SIC codes, and NAICS codes.
• The old system required many-to-many translation layers and lacked auditability.
• The classifier needed to predict six-digit NAICS codes while preserving useful hierarchy-level partial credit.
• Production outputs had to be valid, parseable, and diagnosable rather than opaque free-form LLM text.

Architecture

Ramp built an in-house RAG classifier for NAICS industry classification. The system calculates embeddings for business queries and NAICS descriptions, retrieves recommended codes with optimized embeddings and similarity scores, then uses a dual-prompt LLM flow: one prompt returns a small list of relevant codes and a second prompt selects the best single code with richer context. Structured outputs and valid-code guardrails keep responses parseable, ClickHouse stores knowledge-base embeddings, and Kafka logs intermediate steps for observability and prompt iteration.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  business-customer-record["Business customer record"] --> embedding-service["Embedding service"]
  business-customer-record["Business customer record"] --> downstream-ramp-workflows["Downstream Ramp workflows"]
  naics-knowledge-base["NAICS knowledge base"] --> clickhouse-embedding-store["ClickHouse embedding store"]
  clickhouse-embedding-store["ClickHouse embedding store"] --> similarity-scorer["Similarity scorer"]
  embedding-service["Embedding service"] --> similarity-scorer["Similarity scorer"]
  similarity-scorer["Similarity scorer"] --> recommendation-candidates["Recommendation candidates"]
  similarity-scorer["Similarity scorer"] --> kafka-diagnostic-log["Kafka diagnostic log"]
  recommendation-candidates["Recommendation candidates"] --> dual-prompt-llm-selector["Dual-prompt LLM selector"]
  naics-knowledge-base["NAICS knowledge base"] --> dual-prompt-llm-selector["Dual-prompt LLM selector"]
  dual-prompt-llm-selector["Dual-prompt LLM selector"] --> structured-output-parser["Structured output parser"]
  structured-output-parser["Structured output parser"] --> valid-code-guardrail["Valid-code guardrail"]
  naics-knowledge-base["NAICS knowledge base"] --> valid-code-guardrail["Valid-code guardrail"]
  valid-code-guardrail["Valid-code guardrail"] --> final-naics-classification["Final NAICS classification"]
  dual-prompt-llm-selector["Dual-prompt LLM selector"] --> kafka-diagnostic-log["Kafka diagnostic log"]
  kafka-diagnostic-log["Kafka diagnostic log"] --> evaluation-metrics["Evaluation metrics"]
  embedding-service["Embedding service"] --> evaluation-metrics["Evaluation metrics"]
  final-naics-classification["Final NAICS classification"] --> downstream-ramp-workflows["Downstream Ramp workflows"]
  evaluation-metrics["Evaluation metrics"] --> downstream-ramp-workflows["Downstream Ramp workflows"]

Components

Data flow

A business customer record is embedded and compared against precomputed NAICS description embeddings stored in ClickHouse. Similarity scoring generates recommendation candidates. The first LLM prompt filters to a small relevant code list; the second prompt chooses the best single NAICS code from that list and returns it in a structured format. Guardrails validate that returned codes exist. Kafka logs retrieval, prompt, and prediction intermediates so the team can diagnose whether errors come from retrieval, reranking, prompt behavior, or invalid outputs.

Why this architecture

Ramp chose RAG because it converts open-ended industry labeling into constrained selection over valid NAICS codes. The two-stage design separates retrieval quality from final prediction quality, making evaluation and optimization more targeted. Structured outputs, valid-code checks, Kafka logging, and LLM justifications make the classifier auditable enough for business-critical financial workflows.

Models used: Embedding model · Dual-prompt LLM selector · RAG classifier

Scale

• The system predicts six-digit NAICS codes.
  “The critical challenge was building a classification model capable of predicting six-digit NAICS codes for all Ramp businesses.”summary, stated
• Recommendation retrieval improved by 60%.
  “accuracy improvements of 60% in recommendation retrieval”summary, stated
• Final prediction accuracy improved by 5-15%.
  “5-15% in final prediction accuracy”summary, stated
• The old taxonomy created many-to-many translation layers across homegrown categories, SIC, and NAICS.
  “different teams using incompatible taxonomies (homegrown categories, SIC codes, NAICS codes) that required complex many-to-many translation layers.”summary, stated
• WizeHire became a specific six-digit NAICS code under the new system.
  “Under the new system, WizeHire would be classified as "561311 - Employment Placement Agencies,"”summary, stated

Alternatives considered (roads not taken)

Homegrown categories and translation layers

Rejected because: They produced multiple sources of truth and incompatible taxonomies across teams.

Third-party classification solutions

Rejected because: Ramp's complex proprietary data and need for control justified building in-house.

Single end-to-end evaluation metric

Rejected because: A multi-stage RAG system needed stage-specific metrics so retrieval and selection failures could be separated.

Treat every hallucinated code as equally bad

Rejected because: Some LLM codes not in recommendations were valid beneficial predictions, so Ramp filtered bad hallucinations rather than all novel valid codes.

Patterns

rag-retrieval (retrieval)

Ramp retrieves NAICS recommendations before LLM selection, grounding classification in a valid code knowledge base.

retrieval-ranking (retrieval)

The recommendation stage uses acc@k to ensure the correct code appears in top-k retrieved candidates before LLM selection.

structured-output-contract (interface)

Structured output and valid-code checks make the LLM prediction parseable and auditable.

guardrails-io-filtering (safety)

Ramp validates that returned NAICS codes exist, filtering harmful hallucinations while allowing valid beneficial predictions.

observability-tracing (operations)

Kafka logs every intermediate step so the team can see whether retrieval or reranking caused a failure.

Tradeoffs

Use a dual-prompt LLM selector

Chose First prompt narrows, second prompt selects one code over One-shot final prediction.

Rationale: The staged prompt flow balances broad coverage with focused final decision-making.

Build in-house

Chose Custom RAG over proprietary business data over Third-party classification solutions.

Rationale: Ramp needed control and ownership for complex proprietary data and classification behavior.

Use fuzzy accuracy for final prediction

Chose Hierarchy-aware partial-credit metric over Binary exact-match accuracy only.

Rationale: Partial hierarchy matches can still carry business value in NAICS classification.

Keep intermediate logs

Chose Kafka observability across retrieval and reranking over Black-box final labels.

Rationale: Intermediate logs enable targeted fixes when either stage fails.

Gotchas (what bit them)

Cross-taxonomy translation layers create organizational friction.

Root cause: Teams used homegrown categories, SIC, and NAICS codes with many-to-many mappings.

Resolution: Standardize on NAICS and use hierarchical codes for different granularity levels.

Retrieval quality bounds the LLM's final classification ceiling.

Root cause: If the correct code is not in the candidate set, the LLM cannot select it.

Resolution: Optimize the recommendation stage with acc@k before judging final prediction accuracy.

Strict exact-match accuracy hides useful partial correctness in hierarchical codes.

Root cause: NAICS codes have meaningful prefixes, so some wrong six-digit predictions are closer than others.

Resolution: Use fuzzy accuracy that awards partial credit for correct hierarchy prefixes.

Not every hallucination is operationally harmful.

Root cause: The LLM sometimes predicted valid codes absent from the recommendation list.

Resolution: Validate against real NAICS codes and filter only bad hallucinations.

Evaluation

Ramp decomposed evaluation by stage: acc@k for recommendation retrieval, fuzzy hierarchy-aware accuracy for final prediction, structured-output validation for parseability, valid-code checks for hallucination filtering, and Kafka diagnostics for error analysis.

Metrics: acc@k · fuzzy accuracy · recommendation retrieval accuracy · final prediction accuracy · valid-code rate · diagnosed retrieval vs reranking failures

Outcomes

• Ramp reports 60% accuracy improvement in recommendation retrieval.
• Ramp reports 5-15% improvement in final prediction accuracy.
• The system provides ownership, auditability, and interpretability for industry classification.
• Standardized NAICS classifications improved compliance, cross-team collaboration, and customer understanding.

Concrete anchors include six-digit NAICS codes, 60% recommendation-retrieval improvement, 5-15% final-prediction improvement, acc@k, fuzzy accuracy, ClickHouse, Kafka, and cross-functional use across sales, risk, analytics, and compliance.