Production-Grade Multi-Agent AI Systems: Distributed Systems Patterns for Agent Coordination

databricks_/_various · Finance · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial-services multi-agent coordination

A credit-decisioning multi-agent system failed after scaling from one agent to five because the team treated coordination as an AI/prompting problem rather than a distributed-systems problem involving race conditions, state consistency, contracts, recovery, and observability.

Constraints:

The workflow involves regulated credit decisions where wrong approvals create business risk.
Five agents create many coordination paths and potential failure points.
The business needs exact traceability of which agent made what call, in what order, and with what data.
Failure recovery must support rollback/compensation rather than leaving partial transactions.

Architecture

The recommended architecture uses orchestration for credit workflows, immutable state snapshots with versioning, data contracts between agents, circuit breakers for failing agents, and saga compensation for rollback. On Databricks, LangGraph/Mosaic AI Agent Framework manages workflow order, Unity Catalog registers agents and schemas, Delta Lake stores append-only state versions, Model Serving/AI Gateway enforces retries/timeouts/rate limits, and MLflow logs transitions, traces, and failures.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  credit-application-request["Credit application request"] --> langgraph-orchestrator["LangGraph orchestrator"]
  langgraph-orchestrator["LangGraph orchestrator"] --> credit-score-agent["Credit score agent"]
  langgraph-orchestrator["LangGraph orchestrator"] --> income-verification-agent["Income verification agent"]
  langgraph-orchestrator["LangGraph orchestrator"] --> risk-assessment-agent["Risk assessment agent"]
  langgraph-orchestrator["LangGraph orchestrator"] --> fraud-detection-agent["Fraud detection agent"]
  langgraph-orchestrator["LangGraph orchestrator"] --> final-approval-agent["Final approval agent"]
  langgraph-orchestrator["LangGraph orchestrator"] --> saga-compensation-controller["Saga compensation controller"]
  credit-score-agent["Credit score agent"] --> delta-lake-state-log["Delta Lake state log"]
  income-verification-agent["Income verification agent"] --> delta-lake-state-log["Delta Lake state log"]
  risk-assessment-agent["Risk assessment agent"] --> delta-lake-state-log["Delta Lake state log"]
  fraud-detection-agent["Fraud detection agent"] --> delta-lake-state-log["Delta Lake state log"]
  final-approval-agent["Final approval agent"] --> delta-lake-state-log["Delta Lake state log"]
  unity-catalog-contracts["Unity Catalog contracts"] --> langgraph-orchestrator["LangGraph orchestrator"]
  unity-catalog-contracts["Unity Catalog contracts"] --> ai-gateway-serving-layer["AI Gateway serving layer"]
  ai-gateway-serving-layer["AI Gateway serving layer"] --> langgraph-orchestrator["LangGraph orchestrator"]
  ai-gateway-serving-layer["AI Gateway serving layer"] --> circuit-breaker-policy["Circuit breaker policy"]
  circuit-breaker-policy["Circuit breaker policy"] --> mlflow-observability["MLflow observability"]
  delta-lake-state-log["Delta Lake state log"] --> mlflow-observability["MLflow observability"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Credit application request`	Input requiring credit decisioning across several specialized checks.	`LangGraph orchestrator`
`LangGraph orchestrator`	Central workflow graph that orders agent calls, parallelism, state, retries, and compensation.	`Credit application request`, `Credit score agent`, `Income verification agent`, `Risk assessment agent`, `Fraud detection agent`, `Final approval agent`, `Saga compensation controller`
`Credit score agent`	Calculates or retrieves credit score and appends a versioned state update.	`LangGraph orchestrator`, `Delta Lake state log`
`Income verification agent`	Verifies income information as a workflow step.	`LangGraph orchestrator`, `Delta Lake state log`
`Risk assessment agent`	Assesses risk from current immutable state.	`LangGraph orchestrator`, `Delta Lake state log`
`Fraud detection agent`	Performs fraud checks as part of the decision flow.	`LangGraph orchestrator`, `Delta Lake state log`
`Final approval agent`	Produces final approval/review decision.	`LangGraph orchestrator`, `Delta Lake state log`
`Delta Lake state log`	Append-only store of immutable state versions and customer/workflow data.	`Credit score agent`, `Income verification agent`, `Risk assessment agent`, `Fraud detection agent`, `Final approval agent`, `MLflow observability`
`Unity Catalog contracts`	Central registry for agent functions/models and input-output schemas.	`LangGraph orchestrator`, `AI Gateway serving layer`
`AI Gateway serving layer`	Serving layer enforcing retries, timeouts, rate limits, and circuit breakers.	`Unity Catalog contracts`, `LangGraph orchestrator`, `Circuit breaker policy`
`Circuit breaker policy`	Fails fast and allows half-open recovery for repeated agent failures.	`AI Gateway serving layer`, `MLflow observability`
`Saga compensation controller`	Calls compensating actions in reverse order after partial workflow failure.	`LangGraph orchestrator`
`MLflow observability`	Logs traces, state transitions, failures, and circuit-breaker transitions.	`Delta Lake state log`, `Circuit breaker policy`

Data flow

A credit application enters a workflow orchestrator. The orchestrator calls credit-score, income-verification, risk-assessment, fraud-detection, and final-approval agents in controlled order. Each agent receives immutable versioned state, validates the input contract, appends a new state version, and hands it forward. Circuit breakers wrap calls to failing agents, while saga compensation walks backward through successful agents if a later step fails. State versions, contracts, and transitions are governed and observable through Databricks components.

3 supporting quotes

“the system consisted of a single agent performing credit score calculations, which worked flawlessly in demos and maintained zero issues for two weeks in production.”summary, stated

“when the team scaled to five agents encompassing credit score calculation, income verification, risk assessment, fraud detection, and final approval, critical failures emerged within three days of deployment.”summary, stated

“The orchestration layer uses LangGraph wired into Mosaic AI Agent Framework, handling multi-agent orchestration by managing the workflow graph and determining which agents to call in what order.”summary, stated

Why this architecture

Financial-services credit decisions require debuggability, rollback, and data lineage more than decentralized agent autonomy. Orchestration centralizes visibility, immutable state prevents races, contracts catch boundary errors early, circuit breakers prevent cascading failures, and sagas undo partial work.

3 supporting quotes

“In the financial services context, the presenter notes that orchestration is used almost exclusively because the ability to debug and roll back matters more than agent autonomy.”summary, stated

“This approach eliminates race conditions by preventing concurrent modification of the same record, with each agent appending a new version instead of updating shared state.”summary, stated

“This contract enforcement catches quality issues at the boundary between agents rather than three agents downstream when garbage output appears in a final report.”summary, stated

Models used: LangGraph · Mosaic AI Agent Framework · Unity Catalog functions/models · Databricks Model Serving · AI Gateway · MLflow

Scale

The presenter has 18 years of distributed data systems experience and two years deploying multi-agent AI systems.
“The presenter, Sandy, brings 18 years of experience building distributed data systems at AWS and Databricks, with two years specifically focused on deploying multi-agent AI systems in production.”summary, stated
The first single-agent version ran with zero issues for two weeks.
“maintained zero issues for two weeks in production.”summary, stated
Scaling to five agents produced failures within three days.
“when the team scaled to five agents encompassing credit score calculation, income verification, risk assessment, fraud detection, and final approval, critical failures emerged within three days of deployment.”summary, stated
Twenty percent of credit decisions had incorrect risk ratings.
“Twenty percent of credit decisions showed incorrect risk ratings”summary, stated
The stale-cache race involved a 500 millisecond interval and scores of 750 versus 680.
“the risk assessment agent read the same customer's score 500 milliseconds later, it retrieved a value of 680.”summary, stated
Five agents create at least ten coordination connections and roughly 25x complexity.
“Five agents create at least ten potential connections, and each connection represents a potential failure point, race condition, or state synchronization problem.”summary, stated

Alternatives considered (roads not taken)

Choreography for financial credit decisioning

Rejected because: Decentralized events are scalable but harder to debug; credit decisions need exact order/data traceability and rollback.

“When something goes wrong with a credit decision, the business needs to know exactly which agent made what call, in what order, and with what data.”summary, stated

Shared mutable state in a database/cache

Rejected because: Concurrent reads/writes caused stale cache and lost-update style failures.

“The common but problematic approach involves shared mutable state where multiple agents read and write to the same database records concurrently.”summary, stated

Trust database defaults for concurrency

Rejected because: Default isolation without explicit locks shipped race conditions.

“Many teams use default isolation levels without explicit locks and inadvertently ship race conditions to production.”summary, stated

Let agents exchange arbitrary data

Rejected because: Runtime contracts catch bad handoffs at the boundary instead of downstream.

“Agents cannot simply throw arbitrary data at each other and hope it works.”summary, stated

Patterns

`multi-agent-orchestration` (architecture)

Credit decisioning agents are coordinated through an orchestrator that controls workflow order and parallelism.

“Orchestration provides a centralized alternative where a workflow orchestrator calls each agent directly and manages all coordination.”summary, stated

`durable-execution` (state/recovery)

Immutable state versions and saga compensation allow replay, rollback, and recovery from partial failures.

“If Agent C fails, the system can roll back to version two. For debugging, developers can replay state evolution from version one through version N”summary, stated

`structured-output-contract` (agent handoff)

Agents register and validate explicit input/output schemas at handoff boundaries.

“The Databricks implementation registers these input-output schemas in Unity Catalog, ensuring every agent's contract is versioned and governed in a central location, providing discoverability and governance across the organization.”summary, stated

`observability-tracing` (operations)

MLflow logs circuit transitions, traces, failures, and state history for debugging.

“On Databricks, circuit breaker policies are enforced at the serving layer through Model Serving or AI Gateway, with open-closed transitions logged in MLflow for monitoring when agents start experiencing problems.”summary, stated

`fallback-degradation` (resilience)

Circuit breakers fail fast, half-open test after timeout, and allow reduced functionality or human alerting.

“This pattern prevents one agent's failure from bringing down the entire workflow, enabling graceful degradation”summary, stated

`safety-policy-enforcement` (governance)

Unity Catalog centralizes governance/versioning of agents and schemas for regulated workflows.

“Registering these assets in Unity Catalog makes them centrally discoverable within the organization, governed in one place, and versioned for production operations.”summary, stated

Tradeoffs

Coordination pattern

Chose Orchestration for financial services over Choreography/event autonomy.

Rationale: Debugging and rollback matter more than autonomy in credit decisions.

“orchestration is used almost exclusively because the ability to debug and roll back matters more than agent autonomy.”summary, stated

State mutation model

Chose Immutable append-only state snapshots over Shared mutable records with last-write-wins updates.

Rationale: Versioned state eliminates concurrent modification and makes lineage/replay possible.

“State is stored in the orchestrator database as an append-only log using insert operations rather than updates.”summary, stated

Failure handling

Chose Circuit breakers and saga compensation over Letting retries/timeouts accumulate or leaving partial work.

Rationale: Fail-fast breakers contain cascading failures, and compensation returns the workflow to a clean state.

“If an agent fails mid-workflow, the orchestrator walks backward through the list of executed agents and calls compensate on each one in reverse order.”summary, stated

Gotchas (what bit them)

AI quality was not the failure point.

Root cause: The architecture ignored cache coherency, race conditions, and state synchronization.

Resolution: Treat multi-agent systems as distributed systems with explicit state/version/contract/recovery patterns.

“the problem was not with the model quality, prompt engineering, or AI capabilities, but rather with the architecture failing to account for distributed systems concerns”summary, stated

Cache invalidation created wrong risk decisions.

Root cause: Writes reached PostgreSQL but the cache was not invalidated before risk assessment read stale data.

Resolution: Use immutable state snapshots and controlled orchestrator handoffs rather than shared mutable cache reads.

“While writes to PostgreSQL succeeded, the cache was not being invalidated, causing the risk agent to read stale data from the cache”summary, stated

Choreography is hard to debug without observability.

Root cause: Decentralized agents publish/consume events without one central source of truth.

Resolution: Use choreography only with strong event tracing, or use orchestration for complex regulated workflows.

“The case study emphasizes that choreography requires bulletproof observability with strong event tracing capabilities.”summary, stated

Data contracts catch bad agent handoffs too late if absent.

Root cause: Arbitrary output from one agent can contaminate later agents.

Resolution: Validate schemas and confidence thresholds at every handoff.

“If the confidence score is below 0.7, the analysis agent rejects the handoff. This contract enforcement catches quality issues at the boundary between agents rather than three agents downstream when garbage output appears in a final report.”summary, stated

Evaluation

The system is evaluated operationally through traceability, contract validation, state replay, circuit-breaker monitoring, and correctness of credit decisions rather than only model metrics.

Metrics: incorrect risk rating rate · failure count · open/closed/half-open circuit state · contract validation failures · state version lineage · rollback/compensation completion

“Twenty percent of credit decisions showed incorrect risk ratings, and customers who should have been flagged for review were being approved, causing significant business panic.”summary, stated

“The implementation tracks failure count and state (open, closed, half-open), checking state before attempting calls.”summary, stated

Outcomes

The incident revealed race-condition and cache-invalidation risks in a five-agent credit workflow.
The proposed Databricks architecture supports 24/7 multi-agent workflows across billions of transactions with rollback and observability.
Distributed-systems patterns become the central design language for production multi-agent AI.

The case reports one-to-five agent scaling, 20% wrong risk ratings, two-day root-cause analysis, 500ms stale-cache timing, five agents/ten connections, and billion-transaction production ambitions.

“The root cause analysis, which took two days to identify, revealed a classic distributed systems race condition involving cache invalidation.”summary, stated

“resulting in systems capable of running 24/7 across billions of transactions with proper failure handling and rollback capabilities.”summary, stated