AI Agent for Automated Merchant Classification Correction

ramp · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Merchant classification correction

Ramp needed a scalable correction workflow for user-reported merchant classification errors that affected expense policies and analytics.

Constraints:

same source as transaction matching record
ambiguous merchant identity
accuracy expectations
cost pressure

Architecture

Ramp built a merchant-classification correction agent around user-supplied correction requests. A context builder combines card-acceptor fields, MCCs, receipt-derived merchant names/addresses/line items, user memos, and related merchants retrieved by embeddings plus name matching. The primary LLM maps the request to one of a small set of actions, guardrails reject hallucinated or unsafe actions and force retries, a second LLM can rewrite rejection reasoning for users, and evaluation progresses from manual review to follow-up/rejection signals, LLM judging, and shadow-mode testing.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  correction-request-ui[Correction request UI] --> context-builder[Context builder]
  context-builder --> related-merchant-retriever[Related merchant retriever]
  related-merchant-retriever --> merchant-database[Merchant database]
  context-builder --> primary-llm-decision-agent[Primary LLM decision agent]
  primary-llm-decision-agent --> guardrail-validator[Guardrail validator]
  guardrail-validator --> primary-llm-decision-agent
  primary-llm-decision-agent --> merchant-action-executor[Merchant action executor]
  merchant-action-executor --> merchant-database
  merchant-action-executor --> correction-request-ui
  primary-llm-decision-agent --> rejection-explanation-llm[Rejection explanation LLM]
  evaluation-harness[Evaluation harness] --> primary-llm-decision-agent

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Correction request UI`	Lets users submit the proposed merchant name, website, and category for a misclassified transaction.	`Context builder`, `Merchant action executor`
`Context builder`	Assembles card acceptor data, MCC, receipts, memos, user-supplied correction details, and related merchants for the LLM.	`Correction request UI`, `Related merchant retriever`, `Primary LLM decision agent`
`Related merchant retriever`	Fetches related existing merchants via transaction/merchant embeddings and requested-name matching instead of loading every merchant.	`Context builder`, `Merchant database`
`Merchant database`	Stores Ramp merchant records, mapped card acceptors, categories, websites, and logos.	`Related merchant retriever`, `Merchant action executor`
`Primary LLM decision agent`	Maps the request and context to create/update/reassign/reject decisions and produces reasoning.	`Context builder`, `Guardrail validator`, `Rejection explanation LLM`, `Merchant action executor`
`Guardrail validator`	Constrains the LLM to allowed actions and supplied targets, catches hallucinations, and triggers retry.	`Primary LLM decision agent`
`Merchant action executor`	Applies valid low-impact merchant updates and exposes changes to the front end.	`Primary LLM decision agent`, `Merchant database`, `Correction request UI`
`Rejection explanation LLM`	Rewrites rejection reasoning into clear user-facing language.	`Primary LLM decision agent`
`Evaluation harness`	Uses manual reviews, follow-up and rejection rates, LLM-as-judge, and shadow mode to evaluate agent success.	`Primary LLM decision agent`

Data flow

A user reports a wrong merchant by giving a proposed merchant name, website, and category. Ramp builds LLM context from transaction card-acceptor metadata, receipt/memo evidence, and related merchants fetched by vector and name search. The LLM decides whether to create, update, reassign, or reject; guardrails ensure the selected action and target are allowed, retrying hallucinated responses. If the action changes the merchant classification, the UI updates within seconds; if rejected, another LLM rewrites the reason for user-facing explanation. Evaluation uses manual review early, weak production signals, and an LLM judge at rollout scale.

5 supporting quotes

“When the user supplies a new merchant name, website, and category to fix a merchant classification, the agent's LLM must choose how to respond.”article, stated

“To map a request to an action, we must determine whether the user's request is reasonable. The LLM uses three parts of its context to guide this decision:”article, stated

“We pull merchants that are similar to the transaction using vector embedding similarity and merchants whose names roughly match the requested merchant name.”article, stated

“If the LLM hallucinates, we inform it of its mistake and have it retry until we get a valid response.”article, stated

“When our LLM takes an action that modifies the transaction's merchant classification in some way, the user will observe a change on the front end in seconds.”article, stated

Why this architecture

Ramp needed more than name matching because payment-processor data is cryptic and user context may be the only disambiguating signal. The architecture keeps the LLM useful by giving it targeted merchant context rather than every record, then boxing actions inside guardrails because changing merchant records incorrectly can affect policy enforcement and customer workflows.

3 supporting quotes

“User context is sometimes the only way to map a transaction to a merchant unambiguously. Card acceptor data alone is insufficient and can lead to our initial merchant classification being incorrect.”article, stated

“Giving it every Ramp merchant does not scale and would overload the LLM's context window. Instead, we use a RAG approach to fetch K related merchants.”article, stated

“We allow the LLM to take only one of a select set of actions because modifying Ramp's merchant records correctly is not straightforward.”article, stated

Models used: LLM · second LLM for rejection explanation · LLM judge

Scale

Manual correction time was hours, while the agent unblocks customers in less than 10 seconds.
“Instead of waiting for hours for manual human intervention, our customers are unblocked by our agent in less than 10 seconds.”article, stated
Manual teams handled only 3% of requests in 2023 and 1.5% in 2024, while the agent handles close to 100%.
“Before: Customer support and engineering teams were able to service only 3% of requests in 2023 and 1.5% in 2024. As Ramp grows, these percents will only go down.”article, stated
The LLM judge reports nearly 99% transaction-classification improvement and nearly two-thirds reasonable rejections.
“According to our judge, our agent improves nearly 99% of transaction classifications and nearly two thirds of rejections are reasonable.”article, stated
The LLM-judge evaluation phase was chosen because rollout required evaluating at least 100 requests per day.
“Once we were confident with our agent and began to roll out to more and more customers, we needed an evaluation strategy that satisfied two requirements:”article, stated

Alternatives considered (roads not taken)

Manual support/finance/engineering correction for most requests

Rejected because: The workload scales with card transactions and hours-long manual handling could not keep up with growth.

“The problem of fixing merchant classifications scales proportionally with the number of Ramp card transactions. A manual approach to handling these requests simply is not sustainable as we grow.”article, stated

Use only the user-submitted merchant information

Rejected because: The LLM would not know whether an existing merchant record already exists or whether the requested category is reasonable.

“Solving this problem with just the user's information would be extremely difficult.”article, stated

Put every Ramp merchant into the prompt

Rejected because: It would not scale and would overload the LLM context window, so Ramp retrieves K related merchants instead.

“Giving it every Ramp merchant does not scale and would overload the LLM's context window. Instead, we use a RAG approach to fetch K related merchants.”article, stated

Unconstrained free-form LLM actions

Rejected because: Guardrails require the LLM to choose only supplied actions/targets and retry when hallucinating.

“We require that the LLM always chooses one of the provided actions. This is crucial to ensure the guardrails discussed above work.”article, stated

Patterns

`human-in-the-loop-gate` (workflow)

Users supply corrected merchant information that becomes context for the automated decision.

“To resolve issues like the "entertainment" hotel, Ramp users are able to go to a transaction and request a merchant classification to be fixed by providing a new merchant name, website, and category.”article, stated

`cost-aware-routing` (operations)

The automation replaces expensive manual support with cents-per-request agent execution.

“1.5-3% manually), and achieves a 99% improvement rate according to LLM-based evaluation, while costing only cents per request versus hundreds of dollars for manual handling.”summary, stated

Tradeoffs

Whether to automate near all requests.

Chose agent handling of nearly all reports over servicing only a small percentage manually.

Rationale: Ramp traded manual precision work for an automated path with monitoring and guardrails.

“The system processes requests in under 10 seconds (compared to hours previously), handles nearly 100% of requests (up from 1.5-3% manually), and achieves a 99% improvement rate according to LLM-based evaluation, while costing only cents per request versus hundreds of dollars for manual handling.”summary, stated

Merchant context retrieval.

Chose two retrieval strategies: embedding similarity plus requested-name matching over only searching by card-acceptor embeddings.

Rationale: Card-acceptor names may differ from actual merchant names, so name matching covers cases embeddings alone would miss.

“It is critical that we use both merchants from searching embeddings and merchants that match the requested name because the card acceptor name—the basis of the embeddings—can be different from the merchant's actual name.”article, stated

Evaluation strategy at rollout scale.

Chose LLM-as-judge plus shadow-mode evaluation over manual review of every request.

Rationale: Manual review helped early, but rollout required at least 100 evaluations per day and stronger success signals.

“By using an LLM for evaluation, we can scale to a high number of requests while providing a stronger signal of our agent's success at acting on a user's request.”article, stated

Gotchas (what bit them)

Even knowledgeable humans can struggle with merchant matching.

Root cause: The same text label may correspond to multiple government or local merchants without enough context.

Resolution: Blend user context with model retrieval and guardrails so corrections are not based on name matching alone.

“There are various scenarios where merchant matching is quite difficult, even if a knowledgeable person was doing it by hand.”article, stated

Evaluation

Ramp reports monitoring plus LLM-based assessment of correction quality.

Metrics: 99% improvement rate · under 10 seconds · near 100% request coverage

Outcomes

Correction handling moved from hours and sparse coverage to near-universal low-latency automation.

Duplicate source URL with the transaction-matching record; framing here emphasizes the correction workflow.

“Furthermore, this agent unlocks our potential to more easily 100X our transactions without worrying about 100Xing our customer support staff.”article, implied