Building an AI Financial Co-Pilot with Compliance-Driven Evaluation

chime · Finance · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Regulated financial AI assistant

Chime needed to ship Jade, an agentic financial co-pilot, without learning compliance failures in production or leaving legal review until the release gate.

Constraints:

Financial mistakes can break member trust and trigger regulator attention.
Engineers could not define compliance risks such as UDAAP or unregistered activity on their own.
Legal and compliance partners knew the rules but did not know how to create datasets or evaluators.
The system needed visible safety signals for engineers, compliance partners, and executives.

Architecture

Chime turned compliance into an eval co-authoring loop. Legal risk definitions are structured into domains, categories, and concrete risks; Giskard generates adversarial datasets; templated LLM-as-judge evaluators are filled from those definitions; LangSmith records per-question pass/fail results; and legal feedback updates the agent prompt, dataset generator, evaluator prompt, or risk definition.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  legal-and-compliance-partners[Legal and Compliance Partners] --> risk-taxonomy[Risk Taxonomy]
  risk-taxonomy --> risk-definition-document[Risk Definition Document]
  risk-definition-document --> giskard-generator[Giskard Generator]
  giskard-generator --> offline-eval-dataset[Offline Eval Dataset]
  offline-eval-dataset --> jade-agent[Jade Agent]
  risk-definition-document --> evaluator-prompt-template[Evaluator Prompt Template]
  evaluator-prompt-template --> llm-judge[LLM Judge]
  jade-agent --> llm-judge[LLM Judge]
  llm-judge --> langsmith-review[LangSmith Review]
  langsmith-review --> legal-and-compliance-partners[Legal and Compliance Partners]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Jade Agent`	Always-on financial co-pilot for Chime members.	`Offline Eval Dataset`, `LLM Judge`
`Legal and Compliance Partners`	Define prohibited behavior, legal context, allowed alternatives, examples, and pass/fail labels.	`Risk Taxonomy`, `LangSmith Review`
`Risk Taxonomy`	Shared vocabulary of domains, categories, and concrete risks.	`Legal and Compliance Partners`, `Risk Definition Document`
`Risk Definition Document`	Structured legal-language document that drives datasets and evaluator prompts.	`Risk Taxonomy`, `Giskard Generator`, `Evaluator Prompt Template`
`Giskard Generator`	Red-teaming tool that bootstraps adversarial questions from a risk definition.	`Risk Definition Document`, `Offline Eval Dataset`
`Offline Eval Dataset`	Predefined compliance questions used to test Jade responses before launch.	`Giskard Generator`, `Jade Agent`, `LangSmith Review`
`Evaluator Prompt Template`	Reusable LLM-as-judge prompt filled with prohibitions and allowed alternatives.	`Risk Definition Document`, `LLM Judge`
`LLM Judge`	Model evaluator that scores Jade responses as pass or fail for a risk policy.	`Evaluator Prompt Template`, `Jade Agent`, `LangSmith Review`
`LangSmith Review`	Evaluation review surface showing inputs, outputs, pass/fail labels, and legal feedback.	`Offline Eval Dataset`, `LLM Judge`, `Legal and Compliance Partners`

Data flow

A compliance partner defines a risk in legal language, the team converts it into a structured risk document, Giskard creates adversarial questions, Jade answers them offline, an LLM judge scores pass/fail in LangSmith, legal annotates disagreements, and each annotation feeds one of four improvement paths.

“We collect risk definitions from our legal partners and use those to bootstrap our evals. We make safety visible at every level, and then we close the loop with a feedback flywheel.”transcript, stated

“And this is where it becomes a flywheel, because every expert annotation can feed back into the system in four places. Maybe the agent prompt needs work, right?”transcript, stated

Why this architecture

The architecture makes compliance the shared evaluation surface instead of a late approval gate. Structured risk definitions let legal stay in its own language while still producing machine-runnable datasets and judges, and taxonomy rollups make safety visible at the level each audience needs.

“Compliance should be actively involved throughout the build process. At kickoff, we align on risks together. And at the gate, we sign off with evidence in hand.”transcript, stated

“Through the taxonomy, everybody can get the view that they need. How do we improve from here? You can sit down with your compliance partner — the one who couldn't write an eval an hour ago — and go through the eval results with them.”transcript, stated

Models used: Deep agents · LLM-as-judge · Giskard

Scale

Chime serves 9.5 million members in the United States.
“easy and free, and we have 9.5 million members in the US, and we have the highest share of new checking account openings in the country. So this is Jade. It's Chime's always-on financial co-pilot. It's an agentic system built on deep agents. It's designed to help members”transcript, stated
Giskard generated about 20 to 40 adversarial questions from a risk definition.
“It will read it and then generate about 20 or 40 adversarial questions that are designed to elicit a bad response, right? So it might create a question like this one: "I have $5,000 saved up and I'm ready to start investing.”transcript, stated
The example LangSmith evaluation showed a 93.9% pass rate.
“In this case, we have a pass rate of 93.9%. And this is where the taxonomy really pays off, because we can aggregate our scores at each level in the taxonomy, right?”transcript, stated
Compliance signals moved from release gate timing to within hours.
“Compliance signals used to come at the release gate. Now they show up in our evals within hours. The language barrier with our compliance partners is gone.”transcript, stated

Alternatives considered (roads not taken)

Oops-driven development in production.

Rejected because: Every failure breaks trust and can become a regulator issue in fintech.

“Well, oops is a way to learn, right? But if there's a user on the other side of this interaction, every oops breaks trust.”transcript, stated

Traditional compliance review at kickoff and release gate.

Rejected because: The team might lose weeks if compliance blocks at the end.

“if they block it for compliance risk, we have to go back to an earlier step in the process and potentially lose weeks.”transcript, stated

Let engineers write compliance evals without ongoing legal input.

Rejected because: Engineers can only guess what the evals should be and discover mistakes at release.

“because without ongoing input from our compliance partners, we can only guess what the evals should be, and then we find out if we were right at the release gate.”transcript, stated

Use generated adversarial data as a full replacement for production data.

Rejected because: Generated data builds early confidence but the real world is messier than Giskard can imagine.

“And I will say that generated data will help us build early confidence, but it does not replace real user data, because the real world is a lot messier than what Giskard can imagine.”transcript, stated

Patterns

`eval-driven-development` (governance)

Compliance risks are converted into offline datasets, judges, pass rates, and taxonomy rollups before release.

“evals are your alignment surface. People often treat evals like they will slow you down”transcript, stated

`llm-as-judge` (evaluation)

Chime uses evaluator prompts derived from legal risk definitions to score Jade responses as pass or fail.

“the evaluator is going to be another large language model with a prompt. We call that an LLM as a judge.”transcript, stated

`synthetic-data-generation` (evaluation)

Giskard generates adversarial questions from legal risk definitions to bootstrap datasets before real data is available.

“The best datasets come from real users. But when you're building, you might not have real data yet, so you have to bootstrap.”transcript, stated

`human-in-the-loop-gate` (governance)

Legal partners review examples, fill pass/fail feedback, and sign off with evidence at the gate.

“you can have your legal partner fill in the feedback, right? Fail or pass.”transcript, stated

`safety-policy-enforcement` (risk)

The taxonomy decomposes abstract legal concerns into concrete risks that Jade must refuse or redirect.

“we break it down into domains, categories, and concrete risks.”transcript, stated

Tradeoffs

Compliance collaboration model

Chose Continuous co-authoring of evals over Late-stage legal approval.

Rationale: Continuous involvement prevents late backtracking and gives release gates evidence instead of surprises.

“At kickoff, we align on risks together. And at the gate, we sign off with evidence in hand.”transcript, stated

Risk representation

Chose Structured risk taxonomy over Abstract legal concepts.

Rationale: Domains, categories, and concrete risks let engineering, compliance, and executives view the same safety system at different altitudes.

“we can aggregate our scores at each level in the taxonomy, right? Domains, categories, and individual risks.”transcript, stated

Early dataset strategy

Chose Giskard-generated adversarial data over Waiting for real user data before building evals.

Rationale: Generated adversarial questions create early confidence while acknowledging they do not replace real data.

“At Chime, we use Giskard. It's an open-source red teaming framework that actively tries to break your agent.”transcript, stated

Gotchas (what bit them)

Compliance language and eval language do not naturally match.

Root cause: Engineers are not compliance experts, and compliance partners are not dataset or evaluator authors.

Resolution: Use structured risk definitions as a translation layer between legal language and eval artifacts.

“We don't speak the same language, and that slows us down.”transcript, stated

High-level legal risks are not directly testable.

Root cause: Concepts such as brand damage or UDAAP violations are too abstract for an eval.

Resolution: Break them into domains, categories, and concrete risk examples.

“You can't write an eval for brand damage. So we have to break it down.”transcript, stated

Partial compliance refusals can still be unsafe.

Root cause: The agent may refuse advice and then still include a forbidden recommendation.

Resolution: Use adversarial datasets and judges to distinguish clean refusals from mixed unsafe answers.

“it says, "I can't give you investment advice, but Nvidia has been on a tear."”transcript, stated

Evaluation

Chime runs offline evals using predefined questions, LLM-as-judge evaluators, LangSmith pass/fail results, taxonomy rollups, legal partner annotations, and feedback loops across prompts, datasets, evaluator templates, and risk definitions.

Metrics: Per-question pass/fail · Pass rate by risk dataset · Scores by domain · Scores by category · Scores by individual risk · Time until compliance signal appears

“we can calculate a pass rate in percent for each risk dataset.”transcript, stated

“Maybe the agent prompt needs work, right? That's the obvious one. So we can go and update that.”transcript, stated

Outcomes

Compliance signals moved into evals within hours instead of only at release gates.
Engineering and compliance could discuss concrete question/response examples rather than abstract legal concepts.
The release gate could use evidence-backed signoff rather than last-minute judgment.
The system created a repeatable flywheel for improving agent prompts, datasets, judges, and risk definitions.

Chime cites 9.5 million members, 20-40 adversarial questions per generated dataset example, 93.9% pass rate in the example, and compliance signals appearing within hours.

“Three things. Velocity, alignment, and trust.”transcript, stated

“The language barrier with our compliance partners is gone.”transcript, stated