Scaling AI Agents for Financial Advisory Services with Compliance and Observability

range · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Regulated financial advisory agents

Range needed to scale AI financial-advice agents in production while meeting SEC auditability requirements, maintaining reliability despite provider outages, keeping latency low, and continuously judging whether generated advice was grounded, compliant, and useful.

Constraints:

• Range is an SEC-registered investment advisor with data-retention, auditability, and disclosure obligations.
• Financial advice failures damage trust because users discuss complex financial topics with the agent.
• The system runs more than 15 production agents, each with different definitions of quality.
• Provider capacity issues require failover without causing cascading failures.
• Some workflows need predictable latency and structured outputs, while other tasks are open-ended enough for agents.

Architecture

Range built a Mastra-based production agent platform with Rye as the flagship advisory agent, specialized onboarding and support agents, deterministic document workflows, sub-agent executions, custom logging/tracing, resilient model failover, and asynchronous post-generation LLM-as-judge checks. The system combines workflows for predictable document parsing with agents for open-ended financial advice and tool use.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Range users"] --> B["Rye advisory agent"]
  A --> C["Onboarding workflows"]
  A --> D["Support diagnostic agents"]
  E["Mastra agent platform"] --> B
  E --> C
  E --> D
  E --> F["Logging and tracing system"]
  B --> G["Tax projection system"]
  B --> H["Resilient language model"]
  B --> I["Post-generation analysis"]
  C --> J["Structured output extraction"]
  C --> I
  H --> I
  F --> I
  I --> K["Human advisors and support staff"]
  D --> K
  K --> A

Components

Data flow

Users ask Rye for financial advice such as tax planning. Rye calls Range's internal financial tools, including a tax projection system backed by codified state and federal tax rules. Other onboarding workflows process documents through structured extraction steps, while execution hooks log steps, tool calls, latency, success/failure, tokens, and traces. After generation, asynchronous judge pipelines evaluate grounding, sentiment, corrections, and compliance policies without blocking the user-facing response.

Why this architecture

Range separates workflows from agents because deterministic document parsing benefits from predictable steps and latency, while open-ended financial advice benefits from agentic tool use. Compliance and observability are first-class because regulated advice requires explainability, and resilient model routing is needed because provider outages directly affect trust in financial conversations.

Models used: Rye AI agent · LLM-as-judge · Claude Opus · Gemini Pro · custom models · foundation models

Scale

• Range had more than 15 agents running in production.
  “we have over 15 agents right now running in production. Um all of them have kind of like different definition of like what good looks like.”transcript, stated
• The company has roughly 25 technical staff building on its AI platform.
  “we are pretty much we're a team of about 25 uh technical folks. And so, everyone is building agents.”transcript, stated
• Rye uses a tax projection system with all 50 state tax codes plus federal tax rules.
  “We've codified all 50 uh states and federal uh tax codes as well.”transcript, stated
• Typical human financial advisors are described as passing certification exams at about 70%.
  “Rye can pass with flying colors the typical human financial advisor passes with about a 70%.”transcript, stated

Alternatives considered (roads not taken)

Use agents for every process

Rejected because: Range found workflows more appropriate for deterministic, latency-sensitive document parsing and similar structured processes.

Naive failover to Bedrock

Rejected because: If everyone fails over to the same backup provider during an Anthropic incident, the backup can suffer cascading outages.

Keep custom model training in workflow steps

Rejected because: As foundation models improved and workflows became more agentic, newer model capability ate the custom scaffolding.

OCR plus Textract plus LLM for document parsing

Rejected because: Better models let Range skip the OCR step in many cases and work more directly with document images.

Patterns

tool-use-orchestration (orchestration)

Rye uses existing financial tools such as tax projection systems, while other agents and workflows call document parsing and support-diagnostic capabilities.

observability-tracing (observability)

Range hooks into the execution lifecycle to record step, tool-call, latency, success, failure, token, metadata, and trace information for both debugging and regulatory explainability.

fallback-degradation (reliability)

Range built a resilient language model abstraction that can route around provider or model failures while controlling failover behavior to avoid cascading outages.

llm-as-judge (evaluation)

Range uses asynchronous post-generation LLM judging to evaluate advice and agent outputs against grounding, sentiment, correction, compliance, and policy metrics.

safety-policy-enforcement (governance)

Range encodes regulatory policies and required financial disclosures into post-generation compliance checks.

structured-output-contract (data)

Range's onboarding pipeline converts uploaded documents into specific structured formats after extraction.

Tradeoffs

Workflow versus agent abstraction

Chose workflows for deterministic document parsing and agents for open-ended advisory work over using agents everywhere.

Rationale: Workflows give predictable latency and structure, while agents fit open-ended tasks with tool use and less rigid execution paths.

Build resilient model routing

Chose controlled failover across providers and models over simple Anthropic-to-Bedrock failover.

Rationale: Simple failover can concentrate load on the same backup and create cascading outages.

Post-generation evaluation

Chose asynchronous LLM-as-judge quality and compliance checks over blocking user-facing responses on all evaluation work.

Rationale: The asynchronous check gives live quality and compliance visibility without adding latency to the main response path.

Foundation models over custom workflow models

Chose newer foundation-model capabilities and AI-engineering abstractions over continuing custom model training for workflow injection points.

Rationale: Improved model capability reduced the need for custom scaffolding and moved effort up the stack.

Gotchas (what bit them)

Compliance logging requirements shape the architecture, not just the audit trail.

Root cause: SEC obligations require retention, auditability, metadata, and reasoning traces across agent execution.

Resolution: Range built custom logging, tracing, archiving, and lifecycle hooks around agents and workflows.

Backup-provider failover can cause cascading outages.

Root cause: Many systems fail over to the same backup service when the primary provider has capacity problems.

Resolution: Range built controlled failover through a resilient language model abstraction instead of a single naive backup route.

The wrong abstraction can add latency and unpredictability.

Root cause: Some tasks are structured workflows, while others are open-ended enough to justify agents.

Resolution: Range uses workflows for deterministic document parsing and agents where open-ended reasoning or tool use is needed.

Each production agent needs a different definition of quality.

Root cause: Tax advice, investment-news summaries, onboarding, and support diagnostics have different correctness and compliance criteria.

Resolution: Range built a post-generation pipeline where teams can plug in agent-specific policies and live metrics.

Evaluation

Range evaluates financial-agent behavior after generation with LLM-as-judge and policy-specific checks, tracking grounding, sentiment, corrections, compliance policies, disclosures, latency, success, failure, token usage, and live metrics per agent.

Metrics: Grounding rate · User sentiment · Missing data or user corrections · Compliance and policy checks · Latency · Successes and failures · Token usage · Agent-specific real-time quality metrics

Outcomes

• Range deployed more than 15 agents in production.
• Rye can execute tax projections using Range's internal tax projection system.
• Range built compliance logging, archiving, and tracing around agent and workflow execution.
• Range built a resilient language model abstraction for provider and model failover.
• Range uses post-generation LLM-as-judge analysis for grounding, sentiment, corrections, compliance, and policy checks.
• Improved foundation models let Range drop OCR in many document-processing cases and reduce reliance on custom trained models.

Range reports more than 15 production agents, about 25 technical staff building agents, codified tax logic for all 50 states plus federal rules, and production quality metrics monitored in real time.