Building Production-Ready Agentic AI Systems in Financial Services

fitch_group · Finance · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production agentic AI and AgentOps for regulated financial-services workflows

Fitch Group needed practical ways to move agentic AI systems from proof of concept into production financial-services workflows without losing accountability, evaluation rigor, observability, business alignment, or numerical reliability.

Constraints:

• Financial-services systems are highly governed and require credibility, responsibility, and accountability.
• High-autonomy agent use cases are often not appropriate when output variance tolerance is low.
• LLM outputs are non-deterministic, while financial workflows often require deterministic checkpoints and explainable lineage.
• Production evaluation must be designed before the build, with baseline metrics tied to business value rather than generic productivity claims.
• Agent systems can involve multiple models, modalities, tools, memory, reflection, and concurrent agents, so logs and metrics compound quickly.
• Financial documents mix text, tables, images, and infographics, making extraction and summarization alignment difficult.

Architecture

The recommended production architecture layers agentic capabilities onto controlled financial workflows instead of replacing them with full autonomy. High-ROI use cases are selected using an 80/20 lens. Deterministic workflow agents call tools, memory, databases, and model components at defined steps; more autonomous reflection systems use evaluation agents, reflection agents, historical business rules, current workflow data, and smaller checkpoints rather than giving the LLM open-ended decision authority. Evaluation data, prompts, eval outputs, system prompts, business rules, and style prompts are versioned like APIs. Observability logs inputs, outputs, tool calls, token usage, response time, error rates, model failures, generation failures, drift, checkpoint state, and human-in-the-loop pattern reviews. LLM extraction is grounded with predictive models, knowledge graphs, causal analysis, SMEs, beta testers, and business-defined metrics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  business-problem-and-roi-definition[Business problem and ROI definition] --> data-prep-and-versioning-layer[Data prep and versioning layer]
  business-problem-and-roi-definition --> sme-and-stakeholder-review[SME and stakeholder review]
  data-prep-and-versioning-layer --> deterministic-workflow-agent[Deterministic workflow agent]
  data-prep-and-versioning-layer --> evaluation-framework[Evaluation framework]
  deterministic-workflow-agent --> llm-and-tool-calls[LLM and tool calls]
  llm-and-tool-calls --> checkpoint-logging[Checkpoint logging]
  llm-and-tool-calls --> evaluation-agent[Evaluation agent]
  evaluation-agent --> reflection-agent[Reflection agent]
  evaluation-agent --> business-rules[Business rules]
  reflection-agent --> business-rules[Business rules]
  reflection-agent --> checkpoint-logging[Checkpoint logging]
  deterministic-workflow-agent --> predictive-model-grounding[Predictive model grounding]
  predictive-model-grounding --> evaluation-framework[Evaluation framework]
  checkpoint-logging --> observability-and-monitoring[Observability and monitoring]
  checkpoint-logging --> evaluation-framework[Evaluation framework]
  observability-and-monitoring --> human-in-the-loop-review[Human-in-the-loop review]
  human-in-the-loop-review --> sme-and-stakeholder-review[SME and stakeholder review]
  evaluation-framework --> qa-and-beta-testing[QA and beta testing]
  qa-and-beta-testing --> sme-and-stakeholder-review[SME and stakeholder review]

Components

Data flow

A business stakeholder and technical team define the user problem, expected output, ROI, risk tolerance, and success metrics. Data is prepared and versioned as part of the data prep tax. The workflow then routes data through deterministic process steps, LLM/tool/memory calls, and optional evaluation/reflection agents. Each checkpoint logs input, output, tool calls, and responses. Evaluation agents assess outputs against business rules, reflection agents critique or optimize within those rules, and financial outputs are grounded with predictive models, knowledge graphs, causal analysis, SMEs, and human pattern review. QA and beta testers push the system before production. Production observability monitors reliability, latency, throughput, token usage, drift, output quality, task verification, and business outcomes.

Why this architecture

The architecture is intentionally hybrid and checkpointed because Fitch operates in a high-governance financial context where full autonomy is neither practical nor desirable. The system should augment analysts, preserve deterministic process structure where useful, use LLMs for time-consuming extraction and conversational access, and ground results with older models, business rules, SMEs, and metrics.

Models used: LLMs · LLM-as-a-judge · Reflection agents · Predictive ML models · Knowledge graphs · Causal AI · LangGraph · Agent orchestration frameworks · Multi-modal models

Scale

• The source frames use-case prioritization with an 80/20 rule.
  “80% of the focus should be on, uh, all those use cases where like they say low hanging fruit”transcript, stated
• The speaker warns against spending six months prototyping a framework that may become obsolete.
  “rather than spending, I don't know, six months just prototyping. And by that time the framework has already been obsoleted”transcript, stated
• One target use case is avoiding analysts spending three hours reading 500-page PDFs.
  “going and reading a 500 uh, page PDF, like for and spending like three hours on it.”transcript, stated
• The reflection example coordinates a batch of three systems.
  “Say you're creating a batch of three systems and you want it to configure or like coordinate between each other”transcript, stated
• Example agent workflows may involve three agents and two tools across multiple model layers.
  “if you're initiating like a, say three different agents, it's calling two different tools”transcript, stated
• Human review cannot scale by having two or three people inspect thousands of document extractions.
  “There's no way you can scale it with like say two or three human in the loop reviewing everything, say thousands of document data that you have extracted.”transcript, stated
• Development examples include 25 or even 100 test datasets before QA/beta testing.
  “I have 25, uh, tested data sets or like, even a hundred. It works absolutely amazing. Let's put it into a, a prod.”transcript, stated
• A productivity example describes reducing five hours of work to one or one and a half hours.
  “The work that I used to do in like say, I don't know, just give an example, five hours now I do in like one or one and a half”transcript, stated

Alternatives considered (roads not taken)

Highest-autonomy agents for financial workflows

Rejected because: The speaker says finance is not ready for fully autonomous pipelines and requires accountability and governance.

Replace clean deterministic workflows with complex agent systems

Rejected because: The speaker says simpler RPA-like processes do not need unnecessary model porosity, but can be augmented with tools, memory, and data access.

Give reflection agents full decision authority

Rejected because: The speaker does not want to give all autonomy to reflection agents due to LLM bias and business-rule requirements.

Use LLMs for every step

Rejected because: Financial numerical tasks need hybrid systems that use LLMs where they help and preserve predictive models where they are safer.

Build in a technical silo

Rejected because: The speaker says products built without stakeholder and SME input will not take off or gain adoption.

Patterns

human-in-the-loop-gate (governance)

Financial agent systems remain augmentation tools, with human/SME review at strategic checkpoints and pattern-level oversight rather than total automation.

eval-driven-development (quality)

Teams define baseline business metrics, evaluation data, and custom metrics before building agent systems, then refine evaluation criteria as they observe failures.

llm-as-judge (reflection)

Reflection systems use LLM-as-a-judge style evaluation agents, but the judge is bounded by business rules and checkpoints rather than unconstrained autonomy.

observability-tracing (agentops)

Every step logs inputs, outputs, tool calls, responses, model usage, token usage, latency, errors, drift, and failure type to support root cause analysis.

prompt-versioning (change-control)

Prompts, eval outputs, system prompts, business rules, voice/style prompts, and data are versioned like APIs with test cases for each version.

data-quality-loop (data-readiness)

The data prep tax creates AI-ready data with lineage and versioning for both building systems and evaluating them.

multi-agent-orchestration (agent-architecture)

More autonomous use cases may coordinate evaluation and reflection agents, but the coordination is bounded by rules and checkpoints.

guardrails-io-filtering (scope-control)

Agents are given explicit scope, tools, constraints, and business rules so they do not produce uncontrolled outputs or overstep the task.

model-routing (hybrid-architecture)

The system design chooses where to use LLMs, where to use predictive models, and where to ground outputs with causal or knowledge-graph methods.

Tradeoffs

Use agentic AI as augmentation in finance

Chose Human-guided, checkpointed workflows over End-to-end high-autonomy financial agents.

Rationale: Finance has low tolerance for uncontrolled variance, so agents should handle extraction, querying, and acceleration while humans and deterministic systems retain accountability.

Preserve deterministic workflows where they are clean

Chose Workflow plus selected LLM/tool/memory augmentation over Replacing simple processes with more autonomous agent loops.

Rationale: Existing workflows may not need model porosity; the value comes from augmenting specific bottlenecks rather than adding complexity everywhere.

Ground LLM extraction with established models

Chose Hybrid LLM plus predictive/causal/knowledge-graph grounding over LLM-only financial reasoning.

Rationale: LLMs are useful for time-consuming extraction, but predictive models and causal analysis provide stronger grounding for quantitative financial outputs.

Start with comprehensive logging

Chose Over-logging at initial checkpoints over Minimal telemetry to reduce log volume.

Rationale: The speaker prefers too much initial information because failures are hard to diagnose and hidden logs can contain useful signals.

Scale human review by pattern detection

Chose Human-in-loop review of systematic patterns over Manual review of every extracted document.

Rationale: Thousands of documents cannot be reviewed item by item, so humans look for recurring failure indicators by document type or extraction pattern.

Gotchas (what bit them)

Framework novelty can distract from high-ROI production use cases.

Root cause: Agent frameworks and models change quickly, and spending months prototyping a fashionable architecture may miss business value.

Resolution: Use the 80/20 rule: prioritize high impact-to-effort use cases and define ROI before implementation.

Self-reflection can be biased if an LLM judges other LLM outputs without grounding.

Root cause: The speaker cites studies suggesting LLMs can favor LLM-written output over similar human-written content.

Resolution: Bound reflection with business rules, historical data, current workflow data, and smaller checkpoints.

LLMs are unsafe for direct numerical financial judgment without grounding.

Root cause: Large language models are predictive token systems and do not inherently understand quantitative numbers.

Resolution: Use LLMs for first-pass extraction where useful, then ground outputs with established predictive models and domain-specific analysis.

Dev-test success does not predict production reliability.

Root cause: Developers may test a limited set of datasets without business edge cases, noisy inputs, or beta-user pressure.

Resolution: Run QA, open selected beta testing, ask users to push the system, and iterate with observability checkpoints before production.

Logs can become overwhelming, but too little telemetry prevents root-cause analysis.

Root cause: Agent systems produce reasoning, API calls, memory access, planning steps, tool calls, and outputs across multiple checkpoints.

Resolution: Over-log initially at each checkpoint, then reduce logging volume after understanding the workflow.

Multimodal financial documents can lose meaning across extraction paths.

Root cause: Large documents mix text, tables that are actually images, and infographics, so separate extraction streams may not align.

Resolution: Observe and validate that text, table/image, and infographic extraction align into a coherent story without lost data.

Undefined scope causes agents to overproduce irrelevant work.

Root cause: If the prompt only asks for help, an agent may choose its own broad interpretation and generate excessive artifacts.

Resolution: Provide structured context, exact tasks, allowed tools, constraints, and evaluation hooks.

Evaluation

Evaluation starts before development with baseline business metrics, data/eval preparation, versioned prompts and rules, checkpoint-level logging, and custom metrics. The system tracks traceability, tool-call correctness, source/link quality, model usage, token usage, response time, error rates, model versus generation failures, drift, output quality, task verification, and human/SME pattern review. QA and beta testing precede production because limited dev test cases do not reveal production edge cases.

Metrics: ROI and impact-to-effort ratio · Dollar savings, developer time saved, and cycle completion time · Accuracy, relevance, and output usefulness · Traceability of model and tool calls · Tool-call correctness · Token usage · Response time · Error rate · Model failure versus generation failure · Drift detection · Contextual relevancy and custom metrics · Task verification failures such as premature termination and incomplete verification · Human/SME pattern findings · QA and beta-user edge-case discovery

Outcomes

• The talk reports deployed agentic AI patterns in financial services, including report generation, document processing, and analyst-facing conversational querying.
• The recommended operating model favors augmentation, not full autonomy, in high-governance finance workflows.
• Fitch-style systems use staged evaluation, logging, versioning, and business rules before production release.
• The architecture grounds LLM extraction with predictive models, knowledge graphs, causal AI, SMEs, and beta testing.
• The speaker frames success as measurable business improvement, such as reducing a five-hour task to one or one and a half hours.
• The case identifies new AgentOps concerns such as multi-agent coordination, premature termination, incomplete verification, and incorrect verification.

The transcript gives production-scale indicators rather than a single product metric: 500-page PDFs, three-hour analyst reading tasks, three-agent/two-tool examples, thousands of document extractions, 25-100 test datasets, five-hour work reduced to one or one and a half hours, and six-month framework obsolescence risk.