Building Enterprise AI Agents with Code-First Approach for Trust and Auditability

coinbase · Finance · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise agent platform engineering for regulated internal automation

Coinbase needed a repeatable way to build internal AI agents that automate decision-heavy enterprise workflows without losing the engineering controls required in financial services. The problem was not simply whether LLMs could draft or triage work; it was how to host, version, observe, evaluate, audit, and hand off those agents as production software services that touch business data and regulated legal processes.

Constraints:

• Enterprise agents must satisfy interpretability and auditability expectations for human and regulatory trust.
• Agents interact with business data and automate workflows that were previously human-only.
• The runtime had to fit Coinbase infrastructure and pipelines rather than remain an isolated prototype.
• Prompt-heavy low-code prototypes introduced context noise that made outputs harder to reproduce and individual steps harder to test.
• Legal and regulatory flows required records of data used, reasoning followed, and human approvals.

Architecture

Coinbase formed an Agentic AI Tiger Team to create paved roads for enterprise agents: code-first LangGraph/LangChain-style graphs hosted like services, with deterministic data nodes separated from probabilistic LLM nodes. Each graph starts from an SOP or job description, then runs through typed code, version control, traces, evaluation harnesses, human review, and immutable audit records. LangSmith provides the hosted observability layer, while reference implementations and onboarding materials let other engineers reuse the standardized stack instead of rediscovering hosting, testing, review, and audit patterns.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  agentic-ai-tiger-team[Agentic AI Tiger Team] --> agent-graph-standard[Agent graph standard]
  sop-job-description[SOP job description] --> agent-graph-standard
  agent-graph-standard --> code-first-agent-graph[Code-first agent graph]
  agent-graph-standard --> reference-implementations[Reference implementations]
  code-first-agent-graph --> deterministic-data-nodes[Deterministic data nodes]
  code-first-agent-graph --> llm-nodes[LLM nodes]
  code-first-agent-graph --> observability-platform[Observability platform]
  code-first-agent-graph --> human-review-workflow[Human review workflow]
  code-first-agent-graph --> audit-record[Audit record]
  deterministic-data-nodes --> unit-tests-and-ci[Unit tests and CI]
  llm-nodes --> evaluation-harness[Evaluation harness]
  llm-nodes --> observability-platform
  evaluation-harness --> llm-judge[LLM judge]
  observability-platform --> audit-record
  human-review-workflow --> audit-record
  reference-implementations --> engineering-teams[Engineering teams]

Components

Data flow

A team first writes the agent's SOP and identifies a workflow suitable for automation. The implementation becomes a code-first graph that separates deterministic data-fetch or transform nodes from LLM nodes. Deterministic nodes are unit-tested and gated through CI, while LLM nodes run against curated evaluation datasets and may be spot-checked by another LLM judge. During execution, every tool call, retrieval, decision, and output is traced, and an immutable audit record ties outputs to inputs, tools, reasoning path, and human approval.

Why this architecture

Coinbase chose code-first graphs because regulated enterprise agents need reproducibility, typed interfaces, version control, CI gates, tracing, and auditability. Low-code tools were useful for discovery, but production flows needed the separation and observability of software services. The platform also made human review and audit records first-class features rather than bolt-ons, which matched the legal and regulatory use cases that motivated the work.

Models used: LLMs · LangGraph · LangChain · LangSmith · LLM judge · AWS Bedrock AgentCore

Scale

• The initial tiger-team sprint ran for six weeks.
  “Over the course of a six-week period, the Enterprise Applications and Architecture team at Coinbase formed an Agentic AI Tiger Team”article, stated
• The team tested the platform across three internal domains: Institutional support, Onramp onboarding, and Listing legal review.
  “We focused on building out process automation agents in the Institutional support, Onramp onboarding, and Listing legal review spaces”article, stated
• The sprint produced two production automations and two more completed end-to-end in development.
  “By the end of the six weeks, two automations were in production with measurable time savings, two more were completed end‑to‑end in development”article, stated
• The production automations saved more than 25 hours per week.
  “put two into production with more than 25 hours saved per week”article, stated
• The paved-road approach reduced new agent build time from more than 12 weeks to under a week.
  “published the paved roads that cut new build time from 12+ weeks to under a week”article, stated
• More than half a dozen engineers could self-serve on the patterns after the initial work.
  “more than half a dozen engineers were able to self-serve on the patterns.”article, stated
• The hosted LangSmith implementation became a company-approved observability platform for all Coinbase agents.
  “we now have a consistent, company approved observability platform for all our agents.”article, stated

Alternatives considered (roads not taken)

Use low-code agent tools for core automations.

Rejected because: Coinbase found low-code helpful for prototyping but too noisy and difficult to reproduce, unit test, or gate in CI for long-running operational flows.

Treat agents as chat experiences rather than services.

Rejected because: The team needed typed interfaces, version control, observability, evaluation, and human controls, so they engineered agents as service graphs.

Rely on agent autonomy without explicit human review.

Rejected because: Coinbase intentionally designed handoff, feedback, and approval into the workflow because legal and business processes needed trust and long-tail tuning.

Add runtime complexity such as sidecars by default.

Rejected because: The team preferred the simplest viable Python-only runtime and reserved extra complexity for cases that actually demanded it.

Patterns

tool-use-orchestration (agent architecture)

Coinbase modeled enterprise automations as code-first agent graphs where tools, data nodes, LLM nodes, and human review are explicit service components rather than implicit prompt instructions.

observability-tracing (operations)

The platform traced tool calls, retrieval, decisions, outputs, artifacts, and version differences so teams could debug probabilistic and deterministic parts of agent runs.

eval-driven-development (quality)

Coinbase evaluated probabilistic LLM nodes with curated datasets, a second LLM judge, confidence scoring, and conventional tests for deterministic graph steps.

human-in-the-loop-gate (control)

Human approval and feedback were designed into the UX for workflows where agents draft with references or affect regulated business processes.

safety-policy-enforcement (governance)

The agent stack created immutable records of inputs, tool use, reasoning path, and approvals to satisfy compliance and trust requirements.

prompt-versioning (change management)

Coinbase's code-first approach made agents versionable through existing pipelines, with run diffs and artifact storage supporting controlled changes over time.

Tradeoffs

Choose a code-first agent graph for production automations.

Chose High-code LangGraph/LangChain-style graphs with typed interfaces and version control. over Low-code tool assembly for core automations..

Rationale: Low-code accelerated discovery, but high-code made outputs reproducible, steps testable, and production controls attachable.

Separate deterministic and probabilistic nodes in the graph.

Chose Unit-tested data nodes and separately evaluated LLM nodes. over A single opaque prompt or conversation that mixes data transforms and model calls..

Rationale: The split made failures diagnosable and runs reproducible while allowing specialized evaluation for LLM behavior.

Make human review part of the system.

Chose Designed handoff and feedback loops for approval and tuning. over Treating human review as an ad hoc workaround or removing it to maximize autonomy..

Rationale: Human feedback was necessary for trust, long-tail tuning, and regulated workflows, even if it introduces process friction.

Prefer a simple runtime until complexity is justified.

Chose Python-only builds for speed and simplicity. over Adding sidecars or other runtime complexity by default..

Rationale: A simpler runtime reduced early development friction while leaving room to add complexity for use cases that demanded it.

Invest in shared observability infrastructure.

Chose A hosted LangSmith implementation adopted as the company-approved agent observability platform. over Per-agent tracing and debugging setups..

Rationale: A common platform gave teams a consistent way to trace, evaluate, and debug every agent.

Gotchas (what bit them)

Prompt-heavy low-code prototypes became harder to reproduce as tool count and instruction count grew.

Root cause: Loading multiple tools and instructions into one prompt introduced context noise and hid individual testable steps.

Resolution: Coinbase kept low-code for discovery but moved core automations to code-first graphs with typed interfaces and explicit nodes.

Enterprise agent outputs could not be trusted without complete audit trails.

Root cause: Agents touched business data and legal/regulatory workflows, so teams needed evidence of what data was used, how it was used, and who approved the result.

Resolution: The platform created an immutable audit record for every execution and reused that capability across agents.

An underspecified SOP made the agent design itself suspect.

Root cause: If a human new hire could not succeed with the procedure, an agent would not have enough operational guidance either.

Resolution: Coinbase wrote the agent's SOP before building the agent and used it to define good outputs, allowed sources, and human deferral rules.

Teams could not tune or debug probabilistic agent behavior without full traces.

Root cause: LLM steps are probabilistic, so ordinary unit tests do not provide enough production visibility into decisions and outputs.

Resolution: Coinbase made observability a requirement, traced all execution artifacts, and used evaluation harnesses for LLM nodes.

Making agents fully autonomous would have reduced scale friction but weakened trust for sensitive workflows.

Root cause: Long-tail tuning and regulated decisions required feedback and approval where work happens.

Resolution: The team treated human review as an intentional system component rather than a workaround.

Evaluation

Coinbase used different evaluation modes for different graph nodes: deterministic data-fetch and transform steps were unit-tested and gated, while LLM nodes ran against curated datasets, used a second LLM judge for spot-checks and confidence scoring, and relied on trace diffs and stored artifacts for version comparison.

Metrics: Two automations in production after six weeks. · Two additional automations completed end-to-end in development. · More than 25 hours saved per week. · New agent build time cut from 12+ weeks to under a week. · More than half a dozen engineers self-serving the patterns.

Outcomes

• Coinbase deployed two automations to production during the six-week sprint.
• Two more automations were completed end-to-end in development.
• The production automations saved more than 25 hours per week.
• The paved-road stack reduced new agent build time from more than 12 weeks to under a week.
• A hosted LangSmith implementation became the consistent company-approved observability platform for agents.

The work moved beyond a single prototype: it covered multiple internal business domains, produced reusable reference implementations and onboarding materials, enabled more than half a dozen engineers to self-serve, and established common observability for all Coinbase agents.