Building Production-Grade AI Agents with Distributed Architecture and Error Recovery

parcha · Finance · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Compliance and operations agents

Parcha needed production-grade agents for KYB/KYC-style compliance workflows, but its initial LangChain/websocket prototype suffered from unreliable communication, noisy scratchpads, weak memory retrieval, no recovery after multi-minute failures, hallucinated tools, and low reusability.

Constraints:

• Compliance workflows require multiple tools, OCR on documents, web research, API calls, evidence gathering, and final approve/deny/escalate assessments.
• Long-running tasks can take minutes, so losing a connection or failing late in the workflow is expensive.
• A single agent scratchpad becomes noisy as SOP steps and tool results accumulate.
• LLM stochasticity can pick nonexistent tools or bad tool inputs.
• New customer workflows need reusable building blocks rather than one-off agents and tools.

Architecture

Parcha evolved from a synchronous websocket/LangChain prototype into asynchronous long-running agents with pub/sub updates, REST/polling/SSE integration, Slack threads, and headless evaluation. Complex SOPs are decomposed into coordinator and worker agents: the coordinator plans from the master SOP, workers execute local task subsets with their own scratchpads, and the coordinator synthesizes final recommendations. Redis supplies relevant memory keys to tools without polluting prompts, RQ workers queue and execute jobs, typed tool exceptions are fed back for self-correction, and reusable document-extraction/tools interfaces reduce custom work per workflow.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  operator-or-workflow-trigger["Operator or workflow trigger"] --> async-agent-service["Async agent service"]
  async-agent-service --> status-update-channels["Status update channels"]
  status-update-channels --> customer-action-requests["Customer action requests"]
  customer-action-requests --> coordinator-agent["Coordinator agent"]
  async-agent-service --> rq-worker-queue["RQ worker queue"]
  async-agent-service --> coordinator-agent
  master-sop["Master SOP"] --> coordinator-agent
  coordinator-agent --> worker-agents["Worker agents"]
  worker-agents --> scoped-scratchpads["Scoped scratchpads"]
  worker-agents --> tool-interface["Tool interface"]
  worker-agents --> redis-memory-store["Redis memory store"]
  redis-memory-store --> tool-interface
  tool-interface --> domain-api-tools["Domain API tools"]
  tool-interface --> document-extraction-tool["Document extraction tool"]
  document-extraction-tool --> judgment-llm-call["Judgment LLM call"]
  tool-interface --> typed-exception-feedback["Typed exception feedback"]
  typed-exception-feedback --> worker-agents
  rq-worker-queue --> typed-exception-feedback
  worker-agents --> coordinator-agent
  coordinator-agent --> final-assessment-renderer["Final assessment renderer"]
  final-assessment-renderer --> webhook-automation["Webhook automation"]
  async-agent-service --> pear-benchmark-suite["PEAR benchmark suite"]
  microservice-agent-tools["Microservice agent tools"] --> async-agent-service
  microservice-agent-tools --> tool-interface

Components

Data flow

An operator, API caller, Slack workflow, or webhook triggers an agent job. The long-running agent service posts status through pub/sub, SSE, polling, or Slack thread replies rather than holding a fragile websocket open. A coordinator reads the master SOP and delegates subtasks such as identity verification, certificate validation, or watchlist checks to worker agents. Workers use scoped scratchpads, domain tools, Redis-backed memory keys, OCR/document extraction, and API integrations to gather evidence and conclusions. Tool errors return typed exception names/messages to the agent for self-correction or escalation. The coordinator combines worker evidence into a final approve/deny/escalate report and can be evaluated headlessly or integrated into customer workflows.

Why this architecture

Parcha's production architecture responds directly to failures in the prototype. Asynchronous services replace fragile websockets because customer interactions were mostly one-way job updates. Coordinator/worker agents replace a monolithic SOP scratchpad because context windows became noisy. Redis-backed memory replaces prompt stuffing because workers only need relevant keys. Typed exceptions and queues replace all-or-nothing runs because minute-three failures should recover without restarting the whole task.

Models used: Large Language Models · LangChain Agents · LLM extraction call · LLM judgment call

Scale

• The article describes roughly six months of development from demo toward production.
  “For the past six months at Parcha, we have been building enterprise-grade AI Agents”article, stated
• The first Parcha agents were already in production with initial design partners.
  “Now that we have the first set of Parcha agents in production for our initial design partners”article, stated
• Complex tasks can take minutes and include OCR over multi-page documents, web crawling, and multiple APIs.
  “Our agents may take minutes to perform a complex task.”article, stated

  “Some tasks require doing OCR in multi-page documents, web crawling to research a particular topic, or calling multiple domain-specific APIs”article, stated
• The original architecture could fail after three or four minutes and require a full rerun.
  “if, after 3-4 minutes, a step would fail, the connection to the customer would drop, and the agent would have to be rerun.”article, stated
• The scratchpad example describes a company with over 250 employees, hundreds of thousands of customers, and over $6 billion in assets.
  “It has over 250 employees and is headquartered in Chicago, Illinois.”article, stated

  “hundreds of thousands of customers and manages over $6 billion in assets.”article, stated
• Building the first set of agents took multiple weeks each before reusable building blocks were prioritized.
  “After taking multiple weeks to develop each of our first set of agents”article, stated

Alternatives considered (roads not taken)

Synchronous websocket conversations

Rejected because: Websockets caused reliability issues and did not match the mostly one-way job-completion interaction customers needed.

One monolithic agent with the full SOP in its scratchpad

Rejected because: As steps accumulated, the context window became noisy and the agent confused tools or skipped tasks.

Single-call document extraction and verification

Rejected because: Long documents include irrelevant material, so asking one call to extract and judge produced poor results.

Vector DB or prompt-based memory for worker sharing

Rejected because: The team wanted to avoid more stack complexity and scratchpad pollution, so it reused Redis and injected only relevant keys.

Fail the whole agent run on tool errors

Rejected because: Failures late in long tasks caused poor customer experience; typed exceptions let agents self-correct or escalate.

Patterns

async-message-orchestration (orchestration)

Parcha moved long-running agents to asynchronous processes with pub/sub, REST polling, SSE, Slack threads, RQ workers, and future webhooks.

multi-agent-orchestration (agent-architecture)

A coordinator agent decomposes a master SOP and delegates focused evidence-gathering subtasks to worker agents.

tool-use-orchestration (execution)

Agents use custom API integrations, document extraction, address verification, risk-profile, watchlist, and other domain tools to complete SOP steps.

agent-memory (memory)

Redis stores reusable memory keys that tools can pull as needed, keeping the worker scratchpad focused and reducing redundant tool execution.

prompt-chaining (reasoning)

Document workflows are split into an extraction call followed by a judgment/comparison call with a smaller prompt.

fallback-degradation (reliability)

Typed tool exceptions are routed back to agents so they can self-correct, retry, escalate, or move on instead of failing the whole task.

human-in-the-loop-gate (operations)

Agents can ask clarifying questions, wait for missing information, and produce approve/deny/escalate recommendations for human review.

eval-driven-development (evaluation)

Parcha planned PEAR benchmarks for planning, execution, accuracy, and reasoning, and can evaluate agents at scale as headless processes.

Tradeoffs

Run agents asynchronously.

Chose Long-running async jobs with pub/sub, polling, SSE, and Slack updates over Persistent websocket sessions.

Rationale: The interaction was mostly job submission plus progress updates, and async delivery made agents easier to integrate beyond a web UI.

Decompose SOPs into coordinator and worker agents.

Chose Coordinator-worker delegation over One complex SOP in a single agent scratchpad.

Rationale: Separate scratchpads keep context focused and let workers gather evidence for local tasks.

Separate extraction from judgment.

Chose Two LLM calls: extract relevant document facts, then compare over One LLM call over OCR text plus self-attested data.

Rationale: The second call sees less irrelevant text and improved quality without significantly increasing token count or execution time.

Reuse Redis for memory.

Chose Relevant memory keys in an existing in-memory store over Vector DBs or stuffing previous steps into the scratchpad.

Rationale: It avoided stack complexity and prompt pollution while giving workers reliable access to shared inputs.

Expose typed tool exceptions to the agent.

Chose Exception-name/message feedback and self-correction prompts over Immediate hard failure on any tool error.

Rationale: Many tool failures are recoverable if the agent knows the exact validation error or can decide to escalate.

Build reusable blocks and future microservices.

Chose Composable tool interfaces, document extraction blocks, and planned microservice/DAG orchestration over Tightly coupled per-agent tools.

Rationale: Reusable building blocks reduce time to adapt workflows for new compliance tasks and customers.

Gotchas (what bit them)

Websocket-based agent sessions were unreliable for multi-minute jobs.

Root cause: The original architecture held a connection open until task completion even though the interaction was mostly one-way status updates.

Resolution: Move to asynchronous long-running processes with pub/sub, SSE, polling, Slack threads, and REST-triggered jobs.

Global scratchpads polluted the context window.

Root cause: SOP text, tool outputs, and observations accumulated as agents completed many steps.

Resolution: Decompose work into coordinator/worker agents with separate scratchpads for focused subtasks.

Scratchpad memory led to redundant tool calls.

Root cause: Agents failed to find the right information in the scratchpad and reran tools to gather input for later steps.

Resolution: Store relevant values in Redis and pass only memory keys that tools may need.

Late tool failures forced full reruns.

Root cause: The initial configuration had no recovery path when OCR, web crawling, or APIs failed after minutes of work.

Resolution: Queue jobs, alert on failures, and feed typed exceptions back to the agent for correction or escalation.

LLM hallucinations can break tool workflows.

Root cause: Stochastic model behavior may select nonexistent tools or provide invalid inputs.

Resolution: Return explicit exception names/messages to the agent so it can self-correct or escalate.

Extraction and judgment in one call performed poorly on long documents.

Root cause: Documents contain lots of irrelevant content, so the model had to both locate facts and compare them inside a noisy context.

Resolution: Split document extraction from judgment and run the comparison with a smaller, cleaner prompt.

One-off tools slowed new customer workflows.

Root cause: Tools were tightly coupled to specific agents, so each new agent required new integrations.

Resolution: Invest in composable, extensible building blocks like a reusable document extraction tool.

Evaluation

Parcha evaluated through production design-partner workflows, shadow sessions over real SOPs, headless process runs, and planned PEAR benchmarks for Plan, Execute, Accuracy, and Reasoning. The article also uses operational outcomes such as catastrophic failure reduction, token savings, and context-noise reduction as evidence that the architecture improved.

Metrics: Task completion without catastrophic failure · Recovery after tool exceptions · Token savings from relevant Redis memory · Context-window noise reduction · Headless evaluation at scale · PEAR: Plan, Execute, Accuracy, Reasoning · Workflow adaptation effort for reusable tools

Outcomes

• Parcha moved its first agents into production with initial design partners.
• Async pub/sub simplified communication and opened REST, Slack, SSE, and headless execution modes.
• Coordinator-worker decomposition reduced scratchpad noise in complex SOPs.
• Two-step extraction/judgment improved document verification without significant token or runtime increase.
• Redis-backed memory saved tokens and reduced redundant tool execution.
• Typed exception feedback reduced catastrophic failures significantly.
• Reusable document extraction became a building block used in most workflows.

The source reports six months of work, first production agents with initial design partners, multi-minute tasks involving OCR/web crawling/APIs, 3-4 minute late failures in the old design, multiple weeks per early agent, and example KYB data with 250+ employees, hundreds of thousands of customers, and over $6 billion in assets.