Building Production-Ready AI Agents for Enterprise Operations

parcha · Finance · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Fintech operations and compliance automation

Parcha needed to automate the last mile of fintech operations workflows such as onboarding, KYC, underwriting, risk checks, and compliance reviews, where SOPs existed but human judgment still handled edge cases and manual integrations.

Constraints:

• Compliance failures can mean failing open in a high-risk scenario, so hallucination is unacceptable.
• Customer SOPs are more complex than demos, with many nuances, edge cases, and integrations.
• Long context windows describing every scenario can confuse LLMs after a few thousand tokens.
• Agent workflows require multiple LLM calls, so latency and model steerability both matter.
• Customers need transparency and trust before moving from assisted review to autonomous API operation.
• Early-stage product delivery needs enough customization for white-glove design partners while still evolving toward a scalable platform.

Architecture

Parcha builds verticalized AI agents that execute existing human-written SOPs instead of inventing open-ended plans. A supervisor agent orchestrates worker agents that each handle a smaller procedure slice and expose themselves as text-in/text-out tools. Those workers call integrations for address verification, state registration lookup, identity checks, and other operational systems. The team moved the core agent logic off LangChain abstractions into a roughly 500-line custom implementation while keeping LangChain for LLM interoperability and tooling. Claude is used for fast, stable, steerable multi-step agent behavior. Deployment begins as a transparent Chrome extension with reasoning and step traces, then graduates toward autonomous API endpoints after back-testing, sandbox use, and customer prompt editing.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-sops[Customer SOPs] --> editable-agent-configuration[Editable agent configuration]
  customer-sops --> supervisor-agent[Supervisor agent]
  editable-agent-configuration --> supervisor-agent
  supervisor-agent --> worker-agents[Worker agents]
  worker-agents --> customer-integrations[Customer integrations]
  custom-agent-runtime[Custom agent runtime] --> supervisor-agent
  custom-agent-runtime --> worker-agents
  langchain-interface-layer[LangChain interface layer] --> custom-agent-runtime
  langchain-interface-layer --> claude-model[Claude model]
  back-testing-pipeline[Back-testing pipeline] --> editable-agent-configuration
  back-testing-pipeline --> customer-integrations
  supervisor-agent --> chrome-extension[Chrome extension]
  chrome-extension --> human-operator[Human operator]
  human-operator --> editable-agent-configuration
  supervisor-agent --> api-endpoint[API endpoint]
  back-testing-pipeline --> api-endpoint

Components

Data flow

A customer provides SOPs, policies, tools, integrations, and historical back-testing data. Parcha converts the SOP into an editable agent workflow. The supervisor agent receives the plan and sees worker agents as tools. For each step, it chooses the relevant worker/tool, passes text input, receives text output, and combines worker results into a final recommendation. Worker agents call customer integrations such as address, state registration, and identity-verification systems. The Chrome extension shows operators each step and the reasoning, while back-testing runs many cases in parallel against historical data. Once accuracy and trust are sufficient, the same workflow can be exposed as an API endpoint for autonomous batch processing.

Why this architecture

Parcha chose verticalized, SOP-driven agents because fintech operations already have explicit procedures and integrations, which narrows the problem enough for current LLMs. Decomposing one long context into worker agents reduces hallucination and confusion. Simplifying the runtime to about 500 lines made behavior debuggable, while the Chrome extension and back-testing pipeline let customers build trust before autonomous operation.

Models used: Claude · GPT-4 · GPT-3.5 · LLMs · LangChain · Worker agents · Supervisor agent

Scale

• Parcha started in March 2023 and had raised pre-seed and seed funding by the interview.
  “we just started in March I kind of put the initial MVP together hacked together on top of Lang chain”transcript, stated

  “that's when we raised a seed round we raised a preed and then a seed kind of all together”transcript, stated
• Fintech workflows include onboarding, transaction risk checks, credit-card underwriting, and manual compliance review.
  “onboarding customers to doing risk checks on transactions to underwriting for credit cards”transcript, stated
• The agent input procedures can span 10, 20, or 30 procedures.
  “it could be like 10 20 30 procedures and we give that to the agent and the agent executes on the plan”transcript, stated
• The early demo handled an eight-step onboarding process with one or two integrations.
  “our first demo we demoed a fairly simple eight-step process for onboarding a customer it had like maybe one or two Integrations”transcript, stated
• The problematic context size for one-agent SOPs was around 2,000 to 3,000 tokens.
  “once you start getting to like 2 or 3,000 tokens where You' describe like every possible scenario”transcript, stated
• The decomposed architecture example moved from one agent to five smaller agents plus a supervisor.
  “instead of having one agent that could go and solve this sop we would have five agents that would solve small parts of it”transcript, stated
• The custom core runtime was reduced to roughly 500 lines of code.
  “he got it down to 500 lines of code and what that meant was we could really easily reason about what was happening in the system”transcript, stated
• Parcha targets 90% accuracy on back-testing before handing agents to customers, with customers tuning toward 99%.
  “we back test with their data once we get to 90% accuracy based on back testing data we give it to them”transcript, stated

  “we're going to be able to do it to like maybe that 90% accuracy but to get to 99% accuracy”transcript, stated
• Operators can run about a hundred agents in the cloud in parallel against back-testing data.
  “you can just run like a hundred of these in the cloud and just see what the output is on back testing data”transcript, stated

Alternatives considered (roads not taken)

General-purpose personal assistant agents

Rejected because: Parcha found open-ended agents harder to verify because they must invent their own plans and can get stuck in loops.

Single-agent SOP execution with all edge cases in one prompt

Rejected because: Large contexts made the model confused and increased hallucination risk, so Parcha decomposed the workflow into worker agents.

LangChain abstractions for the core agent runtime

Rejected because: LangChain helped demos but introduced abstraction layers that made production behavior harder to reason about.

OpenAI GPT-3.5 or GPT-4 as the production agent model

Rejected because: Parcha describes GPT-3.5 as fast but less steerable and GPT-4 as more steerable with tradeoffs, while Claude balanced speed, stability, and instruction following.

Parcha owning all prompt tuning to 99% accuracy

Rejected because: The product strategy is to reach a strong baseline and then let operations teams tune agents, mirroring how they already train humans from procedures.

Immediate autonomous API deployment

Rejected because: The initial Chrome extension makes every step and reasoning visible so customers can build trust before using autonomous endpoints.

Patterns

multi-agent-orchestration (agent-architecture)

Parcha decomposes complex SOPs into worker agents and a supervisor agent that treats those workers as tools and synthesizes their outputs.

tool-use-orchestration (agent-tools)

Agents select operational tools for each SOP step, such as address verification, state registration document reading, identity verification, and customer integrations.

prompt-chaining (workflow-control)

The agent carries out SOP steps in sequence, learning from each step and feeding results into subsequent steps rather than issuing one final prompt.

structured-output-contract (agent-runtime)

Parcha moved plan representation from comma-separated text to JSON lists with explicit step tracking so the agent knows where it is in the procedure.

eval-driven-development (quality)

Agents are back-tested against customer historical data until they reach a target accuracy, then customers test in sandbox and feed additional bugs and edge cases back into iteration.

human-in-the-loop-gate (deployment-safety)

Parcha starts with a Chrome extension that lets human operators watch each agent step and reasoning before shifting the same workflow toward autonomous API use.

model-routing (model-operations)

The team kept LangChain for LLM interoperability while selecting Claude as the primary model for its speed, stability, and steerability.

guardrails-io-filtering (safety-and-governance)

The constrained SOP plan and decomposition act as guardrails: the agent makes judgments inside known procedures instead of inventing open-ended plans in high-risk compliance workflows.

Tradeoffs

Constrain agents to customer SOPs

Chose Existing human-written procedures as the plan source over Letting agents generate their own open-ended plans.

Rationale: Skipping plan generation made agents more controllable and better suited to high-risk compliance workflows.

Use a hierarchical multi-agent design

Chose Worker agents plus a supervisor agent over One large agent with the entire SOP in context.

Rationale: Decomposition reduced confusion from large prompts and scaled better to complex procedures with many edge cases.

Rewrite core agent logic outside LangChain

Chose A small custom runtime of about 500 lines over Keeping production behavior buried inside general-purpose framework abstractions.

Rationale: The custom runtime made bugs and execution paths easier to inspect while preserving LangChain for model interfaces.

Use Claude for agent execution

Chose Claude over GPT-3.5 or GPT-4 for this workload.

Rationale: The team wanted both low latency and reliable instruction following across multiple agent steps.

Put agent tuning into customer hands

Chose Editable agents and customer prompt engineering after a 90% baseline over Parcha alone pushing every workflow to 99% accuracy internally.

Rationale: Operations teams already maintain and teach human procedures, so letting them tune agents maps to their existing workflow and shortens the feedback loop.

Deploy first as a Chrome extension

Chose Transparent assisted workflow beside human operators over Immediate fully autonomous API endpoint.

Rationale: The extension exposes reasoning and steps, which helps establish trust and collect feedback before autonomy.

Gotchas (what bit them)

A compelling demo can understate production SOP complexity.

Root cause: The first version handled a simple eight-step flow, but real customer procedures had more pieces, nuances, integrations, and edge cases.

Resolution: Shadow customers, inspect real SOPs, and redesign the architecture around production complexity rather than demo assumptions.

One large SOP prompt made the agent confused and created compliance hallucination risk.

Root cause: Packing every scenario into a 2,000-3,000-token context pushed the LLM past the useful local maximum for reliable instruction following.

Resolution: Decompose the agent into simpler worker components and a supervisor that composes their results.

Framework abstraction layers made production agent behavior hard to debug.

Root cause: LangChain's flexible abstractions support many use cases, but those layers obscured the specific execution path Parcha needed to reason about.

Resolution: Rewrite the core agent logic into a small custom codebase while keeping the framework where it was useful for model/tool interoperability.

Agent bugs are often model-behavior problems rather than ordinary deterministic bugs.

Root cause: The LLM may simply not do what the team wants in a case, forcing new prompting, decomposition, or product approaches instead of a local code fix.

Resolution: Optimize for iteration velocity, customer feedback, and accumulated organizational learning rather than expecting one static implementation to be correct.

Customers ask for autonomous APIs before the team believes trust is earned.

Root cause: The capability is impressive enough that customers quickly want batch automation, but compliance use cases still need transparency and hardening.

Resolution: Begin with a Chrome extension that exposes every step and reasoning, then move to API endpoints once accuracy and trust are established.

Evaluation

Parcha evaluates agents with customer historical data, a 90% back-testing threshold before customer exposure, sandbox/test-environment runs, customer-discovered edge cases, and ongoing benchmarking tools. The Chrome extension also serves as an explainability surface so customers can inspect every step and reasoning before trusting autonomous operation.

Metrics: 90% accuracy on back-testing data before customer handoff · Target progression toward 99% accuracy through customer tuning · Back-testing output over roughly a hundred cloud-parallel agent runs · Customer sandbox feedback · Edge-case bugs found by design partners · Per-step reasoning visibility in the Chrome extension

Outcomes

• Parcha moved from a LangChain MVP to a simpler custom multi-agent core that the team could reason about and debug.
• The architecture evolved from a simple eight-step demo to decomposed worker agents plus a supervisor agent for real customer SOP complexity.
• Agents are back-tested to a 90% threshold before customers receive them, then customers run in sandbox and provide additional edge cases.
• Customers quickly requested autonomous API endpoints after seeing the Chrome-extension version, indicating trust and demand for batch operation.
• The team framed verticalized agents as tractable in current LLM technology, while general-purpose agents remain much harder.

The case is early-stage but concrete: Parcha began in March 2023, raised pre-seed and seed rounds, had a small team, rewrote core agent logic to around 500 lines, targets 90% back-testing accuracy before customer use, can run about a hundred agents in parallel on historical cases, and aims to scale from hand-built design-partner deployments toward sales-led and eventually self-serve agent creation.