AI-Powered Merchant Classification Correction Agent

ramp · Finance · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial transaction merchant classification

Ramp needed to resolve customer-reported merchant classification errors that previously required support, finance, and engineering intervention and scaled with transaction volume.

Constraints:

• incorrect merchant classification can block customers through policy violations or wrong expense categories
• the card acceptor input is often vague and insufficient for perfect matching
• manual handling scales with transaction volume and is not sustainable
• the LLM must be prevented from unsafe merchant-record changes
• evaluation must scale beyond manual review while preserving a strong correctness signal

Architecture

Ramp built an AI agent that takes a customer correction request and constructs context from the transaction card acceptor, merchant category code, extracted receipt names, addresses, line items, and related transaction memos. The LLM uses embeddings, OLAP queries, multimodal RAG, and guardrails to decide whether the request is legitimate and what action is safe. If it acts, the merchant mapping is corrected; if it rejects, a second LLM rewrites the rejection reasoning into plain language for the user. Evaluation evolved from manual review to follow-up/rejection-rate signals and finally an LLM judge, including shadow-mode evaluation before rollout.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-correction-request["customer-correction-request"] --> context-builder["context-builder"]
  context-builder --> receipt-image-extraction["receipt-image-extraction"]
  context-builder --> olap-query-layer["olap-query-layer"]
  context-builder --> embedding-retrieval["embedding-retrieval"]
  context-builder --> classification-agent["classification-agent"]
  classification-agent --> guardrail-policy["guardrail-policy"]
  classification-agent --> merchant-database["merchant-database"]
  classification-agent --> rejection-explainer["rejection-explainer"]
  classification-agent --> evaluation-loop["evaluation-loop"]
  merchant-database --> evaluation-loop

Components

Data flow

A user reports an incorrect merchant classification from a transaction. Ramp builds context from the card acceptor name, MCC, receipts, memos, embeddings, OLAP queries, and related merchant data. The LLM decides whether the requested correction is legitimate and safe under guardrails. Accepted actions update merchant classification data; rejected requests carry reasoning that another LLM rewrites for the frontend. Evaluation data comes from manual samples, repeated requests, rejection rates, shadow mode, and an LLM judge that scores whether changes improved the transaction classification or whether rejections were reasonable.

"The LLM uses three parts of its context to guide this decision:

• Transaction card acceptor name and MCC.

• Extracted merchant names, addresses, and line items from related receipt images."

— article, implied "Transaction card acceptor name and MCC.

• Extracted merchant names, addresses, and line items from related receipt images.

• User-provided memos for related transactions."

Below is part of the rationale provided by our LLM for a user requesting to change a transaction from the "Four Points by Sheraton" hotel to the "Four Points Service Station" gas station."

Why this architecture

Ramp's agent needed more than text generation because merchant correction is an action against financial data. The context builder gives the LLM transaction, receipt, memo, and merchant signals that disambiguate vague card acceptors. Guardrails constrain actions that could corrupt merchant records. A second LLM improves rejected-request UX without changing the primary decision. The evaluation loop escalates from manual review to scalable LLM judging because production volume requires more than spot checks.

Models used: large language model · LLM judge · second LLM for rejection rewriting · embeddings · multimodal RAG

Scale

• Manual support and engineering teams handled only 3% of requests in 2023 and 1.5% in 2024.
  “Customer support and engineering teams were able to service only 3% of requests in 2023 and 1.5% in 2024.”article, stated
• The agent now handles close to 100% of requests.
  “Now:** Our agent handles close to 100% of requests”article, stated
• Requests are handled in under 10 seconds instead of taking hours.
  “Now:** Our agent handles requests in under 10 seconds”article, stated

  “Before:** Teams took hours to solve requests. This kept our customers blocked and cost Ramp hundreds of dollars.”article, stated
• The agent lets Ramp contemplate 100x transaction growth without 100x support staffing.
  “this agent unlocks our potential to more easily 100X our transactions without worrying about 100Xing our customer support staff.”article, stated
• The evaluation strategy needed to evaluate at least 100 requests per day.
  “It can evaluate at least 100 requests a day.”article, stated

Alternatives considered (roads not taken)

Manual customer-support, finance, and engineering handling

Rejected because: It consumed hours per request and scaled with transaction volume.

Let the LLM freely update merchant records

Rejected because: Unconstrained actions could corrupt important merchant data.

Manual evaluation for all production requests

Rejected because: Manual review was useful early but did not scale after rollout.

Show no explanation when the agent rejects a request

Rejected because: Users could not tell whether a request was rejected or failed, nor why it was rejected.

Patterns

tool-use-orchestration (orchestration)

The agent maps a user correction request to an action against merchant data after using contextual tools and guardrails.

rag-retrieval (retrieval)

Ramp builds multimodal context from receipts, memos, transaction fields, embeddings, and OLAP queries before the LLM decides.

guardrails-io-filtering (governance)

Guardrails limit what merchant-record changes the LLM may make based on request and merchant context.

llm-as-judge (evaluation)

Ramp uses a judge LLM to evaluate whether accepted actions improved the classification and whether rejections were reasonable.

canary-rollout (deployment)

Ramp ran shadow-mode evaluation before taking real actions for customers.

Tradeoffs

Automate merchant corrections with an agent.

Chose LLM agent with retrieval, OLAP, and guardrails over manual support/engineering process.

Rationale: Manual work took hours per request and could not scale with transaction volume.

Use rich multimodal context for legitimacy decisions.

Chose MCC, receipt image extraction, memos, embeddings, and OLAP over card acceptor name alone.

Rationale: The card acceptor is often too vague, while receipts and memos provide signals about the true merchant/category.

Use an LLM judge for scaled evaluation.

Chose judge model plus operational signals over manual review for every request.

Rationale: The judge provides stronger signal than simple rates and scales to production request volume.

Explain rejections with a second LLM.

Chose plain-language rejection rewriting over showing nothing or exposing raw reasoning.

Rationale: Users needed timely explanation when a request was rejected.

Gotchas (what bit them)

Card acceptor data can be too vague to classify merchants correctly.

Root cause: Payment processors provide limited transaction data, and similar names can refer to different merchants.

Resolution: Add receipt image extraction, memos, MCC, embeddings, OLAP queries, and LLM reasoning over broader context.

A correction agent can corrupt merchant records if unconstrained.

Root cause: An LLM action can propose unsafe changes unrelated to the legitimate transaction correction.

Resolution: Put guardrails around allowed changes according to request and merchant context.

Rejected requests confused users when no explanation was shown.

Root cause: Users could not distinguish rejection from failed update and did not know the reason.

Resolution: Rewrite primary LLM rejection reasoning into plain language for the frontend.

Manual review could not keep up after rollout.

Root cause: The system needed a correctness signal over at least 100 requests per day.

Resolution: Use repeated-request rate, rejection rate, shadow mode, and LLM judge evaluations to scale quality assessment.

Evaluation

Ramp evaluated the agent in four stages: manual review, repeated correction requests, rejection rates, and an LLM judge. Shadow mode let the team score what the agent would do before taking real action.

Metrics: manual-review correctness on selected transactions · follow-up correction request rate · request rejection rate · LLM judge improvement/reasonable-rejection score · at least 100 requests per day evaluation capacity · nearly 99% classification improvement according to the judge

Outcomes

• the agent handles close to 100% of correction requests
• customers are unblocked in under 10 seconds rather than hours
• manual support coverage rose from 3% in 2023 and 1.5% in 2024 to near-total agent handling
• judge evaluation reports nearly 99% improved transaction classifications
• the RAG-plus-LLM flow was extended to two other merchant mapping problems

Ramp frames the system as enabling 100x transaction growth without 100x customer-support staffing, with evaluation designed for at least 100 requests per day.