Building Trustworthy LLM-Powered Agents for Automated Expense Management

ramp · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: finance automation agents

Ramp wanted LLM-powered agents that could automate expense approval and related finance tasks while preserving user trust through transparency, user control, and reliable escalation.

Constraints:

• Finance workflows are high-trust even when individual decisions have low catastrophic risk.
• LLM outputs need visible reasoning and source grounding.
• Some expenses are ambiguous enough to require human review.
• Policies change and need to be editable by customers, not frozen inside prompts.
• Automation must respect customer-specific hard stops.

Architecture

Ramp's agent architecture centers on a policy agent with policy retrieval, cited reasoning, categorical outcomes, an uncertainty fallback, editable customer policy context, workflow-builder autonomy settings, deterministic rules, staged rollout from suggestions to autonomy, and evals that grow from edge cases and user-flagged failures.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  expense-event["expense event"] --> policy-agent["policy agent"]
  policy-agent --> policy-retrieval["policy retrieval"]
  policy-retrieval --> policy-editor["policy editor"]
  policy-agent --> decision-categories["decision categories"]
  decision-categories --> escape-hatch["escape hatch"]
  policy-agent --> citation-ui["citation UI"]
  citation-ui --> finance-operators["finance operators"]
  escape-hatch --> human-escalation["human escalation"]
  human-escalation --> finance-operators
  policy-agent --> workflow-autonomy-layer["workflow autonomy layer"]
  workflow-autonomy-layer --> deterministic-hard-stops["deterministic hard stops"]
  workflow-autonomy-layer --> finance-operators
  finance-operators --> collaborative-feedback-loop["collaborative feedback loop"]
  collaborative-feedback-loop --> policy-editor
  collaborative-feedback-loop --> eval-harness["eval harness"]
  eval-harness --> policy-agent

Components

Data flow

The policy agent evaluates expenses against in-platform policy context, renders reasoning with policy citations, chooses Approve/Reject/Needs review, applies workflow guardrails, escalates unsure cases, and uses policy edits plus eval failures to improve the system over time.

Why this architecture

Ramp treats trust as a system property: users need to see the model's reasoning, verify it against policy, configure autonomy, and watch uncertain cases defer rather than hallucinate. The architecture therefore mixes LLM decisions with citations, policy editing, deterministic rules, and evals.

Models used: LLMs

Scale

• Ramp reports that the policy agent fully handles more than 65% of approvals.
  “Since enabling the policy agent at Ramp, we’ve seen more than 65% of approvals be fully handled by the agent.”article, stated
• The system supports a conservative mode requiring review above $50.
  “The more conservative end of the autonomy slider, requiring human review on every expense above $50.”article, stated

Alternatives considered (roads not taken)

Represent uncertainty as a confidence percentage.

Rejected because: Ramp found LLM confidence scores uncalibrated and prone to repeated 70-80% values.

Let the LLM make every decision autonomously.

Rejected because: Ramp explicitly preserves human review for uncertain cases and customer-defined hard stops.

Roll out full autonomy immediately.

Rejected because: Ramp built a trust curve by starting with suggestions and gradually granting action rights.

Patterns

structured-output-contract (decisioning)

The agent buckets outcomes into Approve, Reject, and Needs review.

human-in-the-loop-gate (control)

The escape hatch and workflow settings keep humans in the loop for unsure or restricted cases.

“When the agent isn’t sure, we fall back to the pre-agent escalation process users are accustomed to.”article, stated

“These guardrails aren't just safety nets, they're how users tell the agent "I'll never be comfortable with you touching this.” The more conservative end of the autonomy slider, requiring human review on every expense above $50. !”article, stated

safety-policy-enforcement (guardrails)

Deterministic business rules bound the agent's autonomy.

eval-driven-development (evaluation)

Ramp treats evals as unit tests and adds production failures to the evaluation set.

Tradeoffs

Use a hybrid of LLM judgment and deterministic rules.

Chose LLM policy reasoning plus dollar/vendor/category hard stops over LLM-only decisioning.

Rationale: Some organizational rules are clearer and safer as deterministic boundaries.

Use editable policy context.

Chose Ramp-hosted policy editor over External PDF-only policy context.

Rationale: Policy edits improve both agent decisions and human clarity.

Progress from suggestions to autonomy.

Chose A staged trust curve over Immediate autonomous action.

Rationale: Users needed to observe patterns and mistakes before granting the agent real action authority.

Gotchas (what bit them)

Overconfident automation damages trust.

Root cause: Forcing LLMs to decide under uncertainty encourages hallucinated answers and unexpected actions.

Resolution: Allow the LLM to say it is unsure and route the case to human review.

Static policy context can stay ambiguous.

Root cause: Policies maintained outside the platform may not be updated when the agent reveals confusing language.

Resolution: Bring policy authoring into Ramp and let users edit the source context.

Raw user approval behavior can teach the wrong policy.

Root cause: Finance reviewers may approve reasonable but not fully compliant expenses.

Resolution: Review golden datasets against the information available in the system.

Evaluation

Ramp's evaluation process grows from quick tests to deeper edge-case coverage, converts user-flagged failures into test cases, and uses internally reviewed golden datasets to avoid affinity bias.

Metrics: More than 65% of approvals fully handled by the agent · Unsure reasons tracked over time · User-flagged errors added as eval candidates · Several golden datasets reviewed by Ramp

Outcomes

• Ramp reports that the policy agent fully handles more than 65% of approvals.
• The design operationalizes trust through reasoning, citations, uncertainty, autonomy controls, and collaborative policy context.
• The same agent principles extend across expense approval, merchant identification, and receipt parsing.

The source provides a 65% autonomous approval figure and example review threshold, but not detailed error-rate, latency, cost, or customer-distribution metrics.