Benchmarking and Optimizing AI Agents for Accounting Automation

ramp · Finance · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Accounting close automation

Ramp needed to improve an AI accounting agent for complex monthly book-close work without overfitting to slow design-partner feedback or single-customer environments.

Constraints:

• book-close tasks vary widely by customer and business type
• agent changes to prompts, tools, skills, or memory need fast comparable feedback
• memory must transfer reusable accounting process without carrying stale period facts
• skills can add context bloat and hurt performance
• users care about both accounting accuracy and runtime

Architecture

Ramp built an offline accounting benchmark around synthetic business worlds, close tasks, and accountant-written grading criteria. Stack combines a frontier model, agentic harness, tools, hand-written skills, and custom memory. The benchmark supports skill ablation, skill shrinking, memory-on/off roll-forward tests, model selection, task-category analysis, and accuracy-vs-runtime tradeoff analysis. It became the development loop for deciding which skills to keep, how to write memory, and which model/system configuration to ship.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  synthetic-worlds[Synthetic worlds] --> close-tasks[Close tasks]
  close-tasks --> ramp-stack-agent[Ramp Stack agent]
  ramp-stack-agent --> accounting-tools[Accounting tools]
  ramp-stack-agent --> skills-bundle[Skills bundle]
  ramp-stack-agent --> memory-system[Memory system]
  skills-bundle --> skill-ablation[Skill ablation]
  memory-system --> roll-forward-worlds[Roll-forward worlds]
  close-tasks --> grading-criteria[Grading criteria]
  synthetic-worlds --> accounting-benchmark[Accounting benchmark]
  skill-ablation --> accounting-benchmark
  roll-forward-worlds --> accounting-benchmark
  grading-criteria --> accounting-benchmark

Components

Data flow

Synthetic accounting worlds encode QuickBooks state and supporting files. Tasks ask the agent to reconcile, enter data, analyze variances, or build schedules/accruals. The agent uses tools, skills, and memory to produce answers; accountant-written criteria grade each run; benchmark results feed skill pruning, memory-writing strategy, model choice, and harness tuning.

Why this architecture

Ramp uses an offline benchmark because design-partner feedback is slow and customer-specific. Synthetic worlds plus accountant criteria let the team compare models, skills, memory, and harness changes across many accounting contexts before customers see them. Roll-forward worlds specifically test whether memory captures reusable process instead of stale facts.

Models used: GPT 5.5 · GPT 5.4 · Opus 4.7 · Opus 4.6 · Gemini 3.5 Flash · Gemini 3.1 Pro · Ramp Stack

Scale

• The benchmark slice contains 237 tasks and 3,469 grading criteria.
  “Across this slice, the benchmark contains 237 tasks and 3,469 grading criteria. | World | Type | Business | Key accounting complexity | Tasks | Grading criteria |”article, stated
• The benchmark includes eight worlds.
  “Below are the eight worlds included in the analysis. We include the count of tasks and the total number of grading criteria, remember, a given task will have multiple grading criteria.”article, stated
• The retained slim skill bundle improved six standard worlds by 2.3 percentage points.
  “Across the six standard worlds, the slim bundle was +2.3 percentage points versus the no-skills baseline in the current cleaned snapshot.”article, stated
• The spreadsheet skill shrank from 14k to 5k characters, a 64% reduction.
  “The bookkeeping spreadsheet skill went from 14k characters to 5k characters, a 64% reduction. The removed content moved into a separate, narrower module for the workflows that actually need it.”article, stated
• Memory improved roll-forward mean score from 56.0% to 61.5%.
  “On our roll-forward world, memory improved mean score from 56.0% to 61.5%, a +5.5 percentage point gain.”article, stated

Alternatives considered (roads not taken)

Rely on design-partner feedback

Rejected because: The loop is slow, imprecise, and risks overfitting to one customer's environment.

Keep all available skills loaded

Rejected because: Many skills were neutral or negative and added unnecessary context bloat.

Let memory infer reusable process from task and answer alone

Rejected because: It produced plausible summaries but missed reproducible accounting steps.

Patterns

eval-driven-development (product-development)

The benchmark drives model choice, skill pruning, memory strategy, and harness tuning.

agent-memory (memory)

Stack stores reusable accounting-process steps for recurring close tasks and tests them in roll-forward worlds.

context-compaction (prompting)

Ramp prunes and shrinks skills so context contains relevant bookkeeping guidance rather than unrelated content.

model-routing (model-selection)

The benchmark compares frontier model options and system choices by accuracy and runtime.

Tradeoffs

Use synthetic benchmark worlds

Chose static offline accounting worlds and criteria over waiting for customer production repeats.

Rationale: This gives fast, comparable feedback across business types and task categories.

Write step-oriented memory

Chose source-file mappings, schedule structures, GL accounts, calculation methods, and validation checks over free-form answer summaries.

Rationale: Reusable process transfers better than plausible old-answer summaries.

Balance accuracy and runtime

Chose Pareto-frontier model/system analysis over optimizing accuracy alone.

Rationale: Accounting users also experience the agent's end-to-end task runtime.

Gotchas (what bit them)

Skill bundles can hurt through context bloat.

Root cause: Agent systems accumulate too many long skills, many of which are neutral or negative.

Resolution: Ramp uses skill ablation and shrinkage to retain only useful skills.

Memory can smuggle stale facts into future periods.

Root cause: A memory may preserve an old task's fact pattern rather than a general reusable workflow.

Resolution: Ramp moves toward step-oriented memories and filters period-specific details.

A single task or world can mislead optimization.

Root cause: Skill effects vary by world and task category.

Resolution: The benchmark measures across many worlds, tasks, and criteria.

Evaluation

Ramp uses accountant-written criteria over synthetic worlds, five independent runs per task, Pass@1/Pass@5/Pass^5/mean criteria accuracy, task-category analysis, memory on/off roll-forward tests, skill ablations, and runtime-vs-accuracy comparisons.

Metrics: 237 tasks and 3,469 grading criteria · Pass@1, Pass@5, Pass^5, and Mean Criteria Accuracy · five independent runs per task · memory 56.0% to 61.5% · fastest GPT 5.4 around 4 minutes per task and slowest Gemini 3.5 Flash just over 20 minutes

Outcomes

• Ramp Stack outperformed raw frontier models in the reported benchmark comparisons.
• Skill ablation reduced context bloat and produced a slim skill bundle with positive average impact.
• Step-oriented memory improved roll-forward performance and exposed stale-memory failure modes.
• The benchmark became a product-development engine rather than just a scoreboard.

The benchmark is rich: 8 worlds, 237 tasks, 3,469 criteria, five runs per task, and explicit runtime/accuracy tradeoffs.