Cost-Effective LLM Transaction Categorization for Business Banking

anna · Finance · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Business-banking transaction categorization

ANNA needed to categorize business transactions for accounting and tax purposes with LLMs while controlling a large AI-accountant cost center. The workload requires broad business context, a long accounting-rule prompt, and quality monitoring because cost optimizations such as batching, larger contexts, and prompt caching can affect hallucination and predictability.

Constraints:

The AI accountant represented almost two thirds of LLM costs at the time discussed.
Transaction categorization is not real-time, allowing delayed batch processing.
The system prompt was approximately 50,000 tokens according to the summary.
Longer contexts increased hallucinated invalid transaction IDs.
Prompt caching and offline predictions have provider-specific and unpredictable interactions.

Architecture

ANNA optimized LLM transaction categorization through offline predictions for delayed batch discounts, improved context utilization to reduce the number of LLM requests, prompt caching for long static rulebook content, and ongoing quality evaluation to detect hallucination or categorization regressions.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  transaction-categorization-workload["Transaction categorization workload"] --> llm-categorization-requests["LLM categorization requests"]
  accounting-rulebook-prompt["Accounting rulebook prompt"] --> prompt-caching-layer["Prompt caching layer"]
  prompt-caching-layer --> llm-categorization-requests
  accounting-rulebook-prompt --> llm-categorization-requests
  llm-categorization-requests --> offline-prediction-queue["Offline prediction queue"]
  llm-categorization-requests --> quality-evaluation-harness["Quality evaluation harness"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Transaction categorization workload`	Batch of business banking transactions that need accounting categories.	`LLM categorization requests`
`Accounting rulebook prompt`	Long static system context containing accounting and tax categorization rules.	`Prompt caching layer`, `LLM categorization requests`
`LLM categorization requests`	Requests that combine transactions, rulebook context, and output constraints.	`Transaction categorization workload`, `Accounting rulebook prompt`, `Offline prediction queue`, `Quality evaluation harness`
`Offline prediction queue`	Delayed provider batch mode used to reduce cost when real-time output is unnecessary.	`LLM categorization requests`
`Prompt caching layer`	Caches long static prompt portions to reduce repeated prompt-processing cost.	`Accounting rulebook prompt`, `LLM categorization requests`
`Quality evaluation harness`	Monitors categorization quality, hallucination share, and optimization side effects.	`LLM categorization requests`

Data flow

Transactions are grouped into LLM requests with the long accounting-rule context. Because categorization can wait, requests can be sent as offline predictions for discounted delayed processing. Static prompt portions are candidates for prompt caching, while batching and model upgrades are used to reduce calls. Quality checks monitor hallucinated IDs and categorization quality after each optimization.

“So this is a, not real time categorization, which is very important to the stock.”transcript, stated

“So offline predictions, is is the way is basically an agreement between you and ALM provider,”transcript, stated

Why this architecture

The architecture exploits the fact that tax categorization can tolerate delayed output: offline predictions reduce cost, context utilization reduces calls, and prompt caching targets static rulebook text. ANNA keeps evaluation in the loop because provider optimizations can change quality and hallucination behavior.

“we first added offline predictions, then, improve con contact utilization,”transcript, stated

“whatever strategy you use, there is a chance that that may impact your quality solution, quality of your solution.”transcript, stated

Models used: Claude 3.5 · Claude 3.7 · Gemini offline predictions · OpenAI prompt caching

Scale

ANNA's chatbot automated approximately 80% of customer requests.
“the AI automates about, approximately eighty percent of, requests, customers, have.”transcript, stated
The AI accountant represented almost two thirds of costs at the time of the talk.
“the majority, is taken by the AI accountant almost two thirds.”transcript, stated
The illustrated batch split 45 transactions into three LLM requests.
“So say we have a client, they have forty five transactions, and we split them into three requests.”transcript, stated
Offline predictions were expected to cut monthly costs by 50% in the first optimization version.
“monthly costs, expect to, expect to, expect to decrease by fifty percent,”transcript, stated

Alternatives considered (roads not taken)

Use LLMs for every transaction individually

Rejected because: Batching transactions into fewer requests dramatically decreases call count and cost.

“So that would dramatically decrease the number of calls we, do, LLM requests, and we'll, save money.”transcript, stated

Treat provider cost features as guaranteed savings

Rejected because: Prompt caching with offline predictions was difficult to predict and best-effort.

“Savings from on Entropix prompt caching combined with offline predictions are actually difficult to predict.”transcript, stated

Optimize cost without quality checks

Rejected because: Longer contexts and batch modes can affect quality and hallucination rates.

“We saw that, we got more hallucinations with a longer context.”transcript, stated

Patterns

`batch-offline-processing` (cost)

Transaction categorization is delayed through offline predictions for provider discounts.

“offline predictions, you're, getting more”transcript, implied

`semantic-caching` (cost)

Prompt caching targets the long static accounting rulebook portion of the request.

“we can apply the prompt caching to, a particular roboc, which is long,”transcript, stated

`eval-driven-development` (evaluation)

Quality evaluation remains necessary after every cost optimization strategy.

“we want to be sure that the quality is the highest as possible because,”transcript, stated

`cost-aware-routing` (cost)

ANNA combines batching, model/version choices, offline predictions, and caching based on cost-quality tradeoffs.

“upgrading models is a, like, cost optimization strategy in itself.”transcript, stated

Tradeoffs

Use offline predictions for categorization

Chose delayed discounted batch mode over real-time categorization.

Rationale: Tax categorization does not need immediate answers, so delayed processing can save cost.

“to wait up to twenty four hours to get your predictions. And for that, you will get a discount.”transcript, stated

Increase transactions per request

Chose larger batches/context over more smaller calls.

Rationale: Fewer LLM requests save money but must be balanced against hallucination share.

“So that would dramatically decrease the number of calls we, do, LLM requests, and we'll, save money.”transcript, stated

Gotchas (what bit them)

Longer context increased hallucinations.

Root cause: Packing more transactions/context into requests changed model behavior.

Resolution: Measure hallucination share and choose a tradeoff between fewer requests and invalid IDs.

“found the trade off between hallucination share and, and the and the number of transactions”transcript, stated

Prompt caching savings were hard to forecast with offline predictions.

Root cause: Provider documentation treats cache use as best-effort.

Resolution: Treat savings projections as empirical rather than guaranteed.

“they say prompt caching works on a best effort basis,”transcript, stated

Evaluation

ANNA monitors solution quality during optimization, including hallucination share, categorization quality, and the cost/quality effects of offline predictions, prompt caching, and model/context changes.

Metrics: Monthly cost · Hallucination share · Invalid transaction IDs · Categorization quality · LLM request count · Prompt-cache savings