Running LLM Agents in Production for Accounting Automation

digits · Finance · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Accounting automation with production LLM agents

Digits needed to run LLM agents for bookkeeping and accounting workflows such as vendor hydration, onboarding simplification, and natural-language financial questions, while preserving production permissions, reliability, observability, memory, guardrails, and non-Python backend constraints.

Constraints:

• Production backends run in Kotlin and Golang, so Python agent-framework decorators were not directly usable.
• Agents needed to reuse existing permissioned APIs rather than reimplementing backend systems.
• Tool-heavy workflows can become chaotic without observability into prompts, tool calls, latency, cost, and failures.
• Provider outages require LLM proxying and fallback models to maintain service-level indicators.
• Agent memory must compress and preserve preferences rather than merely concatenate conversation logs.

Architecture

Digits evolved from a simple 100-200 line agent loop into a production agent platform. Agents start with an objective, call tools, and answer user or background tasks, but production adds an LLM proxy for model switching and fallbacks, curated API-to-tool schemas over existing permissioned services, Phoenix-based observability, memory services for short- and long-term preferences, guardrails and LLM-as-judge checks, and human review when confidence is low.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Accounting workflow request"] --> B["Agent loop"]
  B --> C["LLM proxy"]
  B --> D["API tool adapter"]
  D --> E["Permissioned backend APIs"]
  B --> F["Memory service"]
  B --> G["Guardrail evaluator"]
  B --> H["Phoenix observability"]
  G --> I["Human review path"]
  H --> I
  I --> A

Components

Data flow

A vendor, onboarding, or financial-question task enters an agent loop. The agent routes model calls through an LLM proxy, plans and calls curated API-backed tools, observes prompts/tool calls/latency/cost through Phoenix, optionally consults memory, and passes outputs through guardrails or human review. High-confidence classifications can ship automatically, while lower-confidence ones are surfaced to users for correction and learning.

Why this architecture

Digits kept the agent loop simple but surrounded it with production infrastructure. Existing APIs preserved permissions, a proxy handled provider choice and outages, observability made multi-tool behavior debuggable, memory compressed learned preferences, and guardrails/human review prevented low-confidence or unsafe outputs from silently reaching users.

Models used: LLMs · LLM-as-judge · Phoenix · Memory tools

Scale

• The simplest agent loop is only about 100-200 lines, but the production architecture is substantially larger.
  “you could probably do this in a 100 lines of code, maybe 200, and you get to a really good outcome.”transcript, stated
• Digits has been developing machine-learning solutions for seven years.
  “So we have been developing um our machine learning solutions for the last seven years and there were a lot of options.”transcript, stated
• About 90% of requests are asynchronous and 10% are synchronous chat-like requests.
  “currently we we see maybe 90% of the requests are asynchronous and 10% of the requests are actually people here coming online chatting with the use case.”transcript, stated
• The classification workflow reports a 96% acceptance rate with about 3% bubbling to end users.
  “we have an average of like 96% um acceptance rate on those things and only like 3% of uh questions actually bubble up to the end user”transcript, stated

Alternatives considered (roads not taken)

Use a full agent framework directly

Rejected because: Framework dependencies increase audit and security complexity, and Python decorators did not fit Digits' Kotlin/Golang production stack.

Manually define every tool schema

Rejected because: Manual tool definitions do not scale beyond a handful of tools.

Expose all backend RPCs as tools

Rejected because: Raw RPC definitions were noisy, so Digits curated the API surface and converted selected APIs to JSON schema.

Store raw LLM outputs as memory

Rejected because: Concatenated outputs are storage, not compressed preference memory.

Patterns

tool-use-orchestration (orchestration)

Digits connects agents to curated existing APIs through tool schemas so agents can act inside permissioned accounting systems.

fallback-degradation (reliability)

An LLM proxy allows model switching and fallback when a provider is down, preserving agent service levels.

agent-memory (memory)

Digits distinguishes memory from raw storage and uses compressed short-term and long-term preference summaries.

observability-tracing (operations)

Phoenix traces prompts, model calls, tool calls, latency, cost, and guardrail outcomes for chaotic multi-tool agent runs.

guardrails-io-filtering (safety-quality)

Digits uses LLM-as-judge checks and policy guardrails before showing outputs to users.

human-in-the-loop-gate (governance)

Low-confidence classifications are surfaced to users for confirmation or correction instead of being shipped silently.

Tradeoffs

Agent framework adoption

Chose direct implementation around existing APIs over heavy framework dependency stack.

Rationale: The core loop is small, while framework dependencies and Python assumptions were costly for Digits' audited Kotlin/Golang environment.

Tool integration surface

Chose curated API-to-JSON-schema tools over all raw RPC definitions or fully manual tools.

Rationale: Raw RPCs were noisy and manual schemas did not scale, so a curated introspected surface balanced safety and maintainability.

Planning model complexity

Chose higher-complexity models for task planning over cheap models for every step.

Rationale: Better planning prevents wrong tool calls and restarts, reducing execution time enough to offset higher planning cost.

Gotchas (what bit them)

Agent frameworks were not a free win in a secure non-Python production stack.

Root cause: Framework dependency chains increase audit burden, and Digits runs Kotlin/Golang rather than Python in production.

Resolution: Implement the core loop directly and reuse existing APIs for tools.

Manual tool definitions do not scale.

Root cause: Writing objective, schema, inputs, and outputs for every tool becomes time-consuming as tool count grows.

Resolution: Curate the API list and introspect selected APIs into JSON schemas agents can call.

Tool-heavy agents become hard to debug quickly.

Root cause: One task can fire multiple LLM requests and tool calls behind the scenes.

Resolution: Add agent observability with prompt/tool-call traces, latency, cost, and guardrail outcomes.

Raw transcript-style storage does not create useful memory.

Root cause: Concatenating model outputs preserves text but does not compress preferences or make them reusable.

Resolution: Use memory tools that summarize short-term interactions and preserve long-term preferences in databases.

Evaluation

Digits evaluates production agents through observability traces, guardrail checks, confidence-based human review, acceptance rates, and async/sync request mix. The talk reports a 96% acceptance rate and about 3% user review rate for a classification case.

Metrics: 96% average acceptance rate for the classification case · 3% of questions bubble up to end users for review · 90% asynchronous requests and 10% synchronous chat-like requests · Prompt, tool-call, latency, cost, and guardrail traces in Phoenix

Outcomes

• Digits operates agents for vendor hydration, onboarding simplification, and financial-book questions.
• The production architecture reuses existing permissioned APIs rather than rebuilding backend services.
• The platform includes proxying, observability, memory, guardrails, and human review beyond the simple agent loop.
• A classification use case reports 96% acceptance with only about 3% requiring user review.
• Most agent work runs asynchronously behind the scenes, with a smaller synchronous chat path.

Digits reports seven years of ML infrastructure, approximately 90% asynchronous requests, 10% synchronous requests, a 96% acceptance rate in one classification case, and roughly 3% user review escalation.