Evolving Quality Control AI Agents with LangGraph

rexera · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real estate transaction quality control

Rexera needed AI quality-control checks for thousands of real-estate workflows per day, but single-prompt LLM checks and later CrewAI agents struggled with complex multi-step workflows, wrong decision paths, and false positives or false negatives.

Constraints:

Architecture

Rexera evolved its QC architecture from single-prompt LLM checks to CrewAI multi-agent checks and then to LangGraph. The final LangGraph design uses tree-like branching and cycles to route each transaction through custom decision paths, such as a rush-order branch versus standard-order checks, while measuring false positives, false negatives, execution speed, and cost efficiency.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  real-estate-workflow-input["Real estate workflow input"] --> quality-control-application["Quality Control application"]
  quality-control-application --> single-prompt-llm-checks["Single-prompt LLM checks"]
  quality-control-application --> crewai-agents["CrewAI agents"]
  quality-control-application --> langgraph-decision-tree["LangGraph decision tree"]
  single-prompt-llm-checks --> metrics-evaluator["Metrics evaluator"]
  crewai-agents --> metrics-evaluator
  langgraph-decision-tree --> rush-order-branch["Rush-order branch"]
  langgraph-decision-tree --> standard-order-branch["Standard-order branch"]
  langgraph-decision-tree --> metrics-evaluator
  rush-order-branch --> qc-finding-output["QC finding output"]
  standard-order-branch --> qc-finding-output
  qc-finding-output --> quality-control-application
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Real estate workflow input Provides transaction data, documents, SLA state, client communications, and counterparty interactions for review. Quality Control application
Quality Control application Reviews thousands of workflows daily and coordinates AI checks. Single-prompt LLM checks, CrewAI agents, LangGraph decision tree
Single-prompt LLM checks Initial QC checks for document accuracy, SLA compliance, and other workflow issues. Quality Control application, Metrics evaluator
CrewAI agents Multi-agent iteration where different agents oversee different transaction-process areas. Quality Control application, Metrics evaluator
LangGraph decision tree Custom branching and cyclic control flow that directs checks based on scenario-specific requirements. Rush-order branch, Standard-order branch, Metrics evaluator
Rush-order branch Checks whether requested rush orders were acknowledged, executed, and marked correctly. LangGraph decision tree, QC finding output
Standard-order branch Runs regular processing checks for non-rush workflows. LangGraph decision tree, QC finding output
Metrics evaluator Tracks false positives, false negatives, execution speed, and cost efficiency across architecture iterations. Single-prompt LLM checks, CrewAI agents, LangGraph decision tree
QC finding output Reports accurate workflow issues to prevent delays or missed transaction problems. Quality Control application

Data flow

A real-estate workflow enters the QC application, task-specific checks inspect documents, SLA timing, communications, and transaction context, LangGraph routes the case through branches such as rush-order handling, the agent compares requested and recorded order states, and the QC finding flags inconsistencies before they cause delays.

“This application checks for errors across various stages of a real estate transaction, including data handling, client communication, and interactions with counterparties”article, stated
“When the application identifies a rush order, it's directed down the "Rush Order" branch of the tree. For standard orders, the application follows a different branch”article, stated

Why this architecture

Rexera moved to LangGraph because the CrewAI system improved headline error rates but still lacked precise control over decision paths. LangGraph let Rexera encode branching for complex workflow scenarios so the QC application could follow the right checks rather than letting agents wander into wrong paths.

“This lack of precise control meant that in complex scenarios, agents might veer off course, leading to false positives or false negatives.”article, stated
“Using LangGraph, Rexera created a tree-like structure for the Quality Control (QC) application that allows for cycles and branching.”article, stated

Models used: LLMs · LangChain · LangGraph · CrewAI · Single-prompt LLM checks · Quality Control application

Scale

Alternatives considered (roads not taken)

Single-prompt LLM checks.

Rejected because: They could not grasp the full workflow context or navigate multi-dimensional real-estate scenarios.

“Single-prompt LLMs struggled with the complexity of real estate workflows for several reasons, as they couldn’t grasp the full scope of a workflow”article, stated

CrewAI multi-agent checks.

Rejected because: CrewAI improved error rates, but the agents still sometimes chose wrong decision paths in complex workflows.

“While the AI agents were capable, they sometimes took the wrong path in decision-making, like a GPS system choosing a longer route.”article, stated

Use the same check path for rush and standard orders.

Rejected because: Rush orders require scenario-specific routing to check both acknowledgment/execution and whether the system recorded the rush status correctly.

“For standard orders, the application follows a different branch, focusing on regular processing checks.”article, stated

Patterns

multi-agent-orchestration (workflow-specialized agents)

Rexera tried CrewAI with different agents overseeing different parts of the real-estate transaction process.

“Rexera then tried a multi-agent approach using CrewAI, where each AI agent oversaw a different part of the transaction process.”article, stated

eval-driven-development (quality-control iteration)

Rexera compared each architecture using false positives, false negatives, execution speed, and cost efficiency.

“Rexera evaluates LLM check effectiveness using three key metrics across thousands of workflow runs:”article, stated

guardrails-io-filtering (deterministic decision routing)

LangGraph constrained checks through custom branches so the QC application followed scenario-specific paths instead of arbitrary agent routes.

“By following a custom decision path, it ensured both the acknowledgment of the rush and the proper handling of the order type were verified”article, stated

Tradeoffs

How to improve beyond simple LLM checks.

Chose CrewAI multi-agent decomposition over Single-prompt LLM checks.

Rationale: Dividing the process among agents reduced false positives and false negatives, even though it was not precise enough as the final architecture.

“This approach led to some improvements over single-prompt LLMs:”article, stated

How to gain precise decision control.

Chose LangGraph tree-like branching and cycles over CrewAI agents choosing paths dynamically.

Rationale: LangGraph gave Rexera control over scenario-specific paths and reduced randomness in QC decisions.

“introducing more deterministic decision-making and reducing the randomness of agents going down the wrong path”article, stated

How to handle rush-order QC logic.

Chose Dedicated rush-order branch over Standard-order processing path.

Rationale: Rush orders require checking acknowledged rush requests, execution, and whether the system marks the order type correctly.

“Order details indicate 'Rush Order: False' despite the client requesting and us acknowledging and executing a rush order.”article, stated

Gotchas (what bit them)

Single-prompt LLMs missed workflow complexity.

Root cause: They had limited context and could not navigate multi-dimensional real-estate scenarios.

Resolution: Move first to multi-agent checks and then to LangGraph-controlled decision paths.

“had limited context, and couldn’t navigate multi-dimensional scenarios properly.”article, stated

Multi-agent systems can still take the wrong route.

Root cause: CrewAI agents were capable but lacked precise control over decision-making in complex cases.

Resolution: Use LangGraph to custom-design scenario-specific paths with branches and cycles.

“This lack of precise control meant that in complex scenarios, agents might veer off course”article, stated

A QC agent can miss a discrepancy even when it recognizes the high-level outcome.

Root cause: CrewAI identified that the rush order was acted on but missed whether the order type was marked correctly.

Resolution: LangGraph's custom path checked both the rush acknowledgment/execution and the recorded order-type state.

“CrewAI identified the correct outcome of the rush order being executed but failed to notice a discrepancy in the order type”article, stated

Evaluation

Rexera evaluated QC architecture iterations using false positives, false negatives, execution speed, cost efficiency, and side-by-side outputs on rush-order scenarios.

Metrics: False positives · False negatives · Execution speed per transaction · Cost efficiency · Rush-order scenario accuracy

“Rexera evaluates LLM check effectiveness using three key metrics across thousands of workflow runs:”article, stated
“Comparing these outputs for the same rush order scenario, the following example breaks down the QC findings with each new architecture:”article, stated

Outcomes

Rexera reports thousands of daily workflow reviews in a $50 billion real-estate transaction market, with published error-rate reductions across the single-prompt, CrewAI, and LangGraph iterations.

“This shift has led to greater consistency in results, ensuring workflows are completed accurately and efficiently.”article, stated
“The final migration to LangGraph enabled custom decision paths with tree-like branching structures, further reducing false positives to 2% and false negatives to 2%”summary, stated