RAG-Based Industry Classification System for Customer Segmentation

ramp · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer industry classification and segmentation

Ramp needed a consistent, auditable way to classify customers by industry for compliance, portfolio monitoring, sales targeting, and product analytics after homegrown taxonomies created inaccurate, overly generic, and inconsistent categories.

Constraints:

• Existing classifications came from third-party data, sales-entered information, and customer self-reporting.
• Different teams used inconsistent taxonomies and multiple sources of truth.
• The output needed to map to standardized NAICS codes rather than free-form categories.
• Business users needed interpretable classifications for risk, sales, marketing, and analytics decisions.

Architecture

Ramp replaced its homegrown classification patchwork with an in-house RAG classifier over NAICS codes. The system computes embeddings for business queries and the NAICS knowledge base, uses similarity scores to retrieve candidate recommendations from precomputed embeddings stored in ClickHouse, and asks an LLM to choose a final NAICS code from the filtered candidates. Internal services handle embeddings and prompt evaluation, Kafka logs intermediate results for diagnostics, and the valid-code knowledge base constrains output to an auditable taxonomy.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-business-record["Customer business record"] --> embedding-service["Embedding service"]
  customer-business-record["Customer business record"] --> downstream-business-teams["Downstream business teams"]
  naics-knowledge-base["NAICS knowledge base"] --> clickhouse-embedding-store["ClickHouse embedding store"]
  clickhouse-embedding-store["ClickHouse embedding store"] --> similarity-scorer["Similarity scorer"]
  embedding-service["Embedding service"] --> similarity-scorer["Similarity scorer"]
  similarity-scorer["Similarity scorer"] --> candidate-recommendations["Candidate recommendations"]
  candidate-recommendations["Candidate recommendations"] --> llm-selector["LLM selector"]
  naics-knowledge-base["NAICS knowledge base"] --> llm-selector["LLM selector"]
  llm-selector["LLM selector"] --> valid-naics-output["Valid NAICS output"]
  naics-knowledge-base["NAICS knowledge base"] --> valid-naics-output["Valid NAICS output"]
  valid-naics-output["Valid NAICS output"] --> downstream-business-teams["Downstream business teams"]
  similarity-scorer["Similarity scorer"] --> kafka-diagnostic-log["Kafka diagnostic log"]
  llm-selector["LLM selector"] --> kafka-diagnostic-log["Kafka diagnostic log"]
  kafka-diagnostic-log["Kafka diagnostic log"] --> prompt-evaluation-service["Prompt evaluation service"]
  embedding-service["Embedding service"] --> prompt-evaluation-service["Prompt evaluation service"]
  prompt-evaluation-service["Prompt evaluation service"] --> llm-selector["LLM selector"]

Components

Data flow

A customer or business record enters Ramp's classification flow. Internal services create embeddings for the query and compare them with precomputed NAICS knowledge-base embeddings stored in ClickHouse. Similarity search produces candidate NAICS recommendations. The LLM receives the filtered candidates and selects a final classification rather than generating an unconstrained label. Intermediate results flow to Kafka so the team can diagnose pathological cases and iterate prompts, while downstream compliance, risk, sales, marketing, and analytics teams consume the standardized industry data.

Why this architecture

Ramp chose in-house RAG because it needed the interpretability of a standardized taxonomy, the flexibility to control cost and behavior, and the validity guarantee of choosing from NAICS candidates instead of generating arbitrary labels. Retrieval narrows the taxonomy to plausible codes; the LLM handles semantic selection among those candidates; logging and prompt evaluation give the team a way to diagnose failures.

Models used: Embedding model · LLM selector · RAG classifier

Scale

• The system migrated industry classification to standardized NAICS codes.
  “migrated all industry classification to standardized NAICS codes”summary, stated
• The architecture has three main RAG stages.
  “The RAG architecture consists of three main stages”summary, stated
• The old Professional Services category was broad enough to include law firms, dating apps, and consulting firms.
  “a category so generic it could encompass law firms, dating apps, and consulting firms”summary, stated
• The classification supports at least five business functions.
  “including compliance, portfolio monitoring, sales targeting, and product analytics.”summary, stated

Alternatives considered (roads not taken)

Continue homegrown taxonomies

Rejected because: They produced inaccurate, generic, inconsistent, and non-auditable classifications.

Use stitched third-party, sales-entered, and self-reported labels

Rejected because: The combined sources produced multiple sources of truth across teams.

Let the LLM generate industry labels freely

Rejected because: Ramp needed outputs constrained to valid NAICS codes for interpretability and consistency.

Rely on third-party classification solutions

Rejected because: The in-house system gave Ramp full control over behavior, iteration, costs, and interpretability.

Patterns

rag-retrieval (retrieval)

Ramp retrieves candidate NAICS codes from embedded taxonomy knowledge before asking an LLM to select the final classification.

retrieval-ranking (retrieval)

Similarity scoring ranks candidate NAICS recommendations before LLM selection.

structured-output-contract (interface)

The model output is constrained to valid NAICS codes, turning classification into a multiple-choice selection task.

observability-tracing (operations)

Kafka logs intermediate retrieval and selection results for diagnostics and prompt iteration.

data-quality-loop (quality)

The system turns inconsistent taxonomy data into a standardized, auditable classification layer used across teams.

Tradeoffs

Build an in-house classifier

Chose Custom RAG over NAICS codes over External or stitched classification sources.

Rationale: Ramp gained control over model behavior, interpretability, and classification process consistency.

Use retrieval before LLM selection

Chose Embedding-based candidate retrieval plus LLM final prediction over Direct LLM classification across the whole taxonomy.

Rationale: Retrieval narrows the output space and makes the LLM choose from plausible valid codes.

Log intermediate results

Chose Kafka diagnostics over Only final classifications.

Rationale: Intermediate logging supports diagnosis of pathological cases and prompt iteration.

Gotchas (what bit them)

Homegrown taxonomies became multiple sources of truth.

Root cause: Ramp stitched together third-party data, sales-entered information, and self-reported customer labels.

Resolution: Move all industry classification to a standardized NAICS-based system.

Overly generic categories were not useful for business decisions.

Root cause: Labels like Professional Services covered very different business types.

Resolution: Use NAICS candidates and LLM selection to produce more specific, interpretable classes.

Unconstrained LLM classification would be hard to audit.

Root cause: Open-ended generation can produce invalid or inconsistent labels outside the taxonomy.

Resolution: Constrain the final decision to valid NAICS codes from retrieved candidates.

Evaluation

Ramp evaluated classification behavior through internal embedding and LLM prompt evaluations, diagnostic logging of intermediate results, and business validation of standardized NAICS outputs.

Metrics: classification quality · retrieval recommendation quality · prompt evaluation results · pathological-case diagnosis · cross-team consistency

Outcomes

• Ramp migrated customer industry classification to standardized NAICS codes.
• The system improved data quality and cross-team communication.
• Classifications became more interpretable and auditable than the previous homegrown taxonomy.
• The in-house system gave Ramp control over the classification process.

The case anchors scale through standardized NAICS migration, three RAG stages, ClickHouse precomputed embeddings, Kafka intermediate logging, and downstream usage across compliance, portfolio monitoring, sales targeting, and product analytics.