Production Agents: Routing, Testing and Browser Automation Case Studies

various · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production agent routing, testing, and browser automation

The panelists needed practical ways to make early LLM agents usable in production when agents had too many tools, weak real-time steering, unreliable browser execution, hard-to-test nondeterministic behavior, slow responses, and ambiguous failure modes.

Constraints:

• Users need to redirect agents while they are running without approving every single action.
• Agents with many CRUD/API tools run out of context and choose tools poorly.
• Browser automation must compress page state, handle icons/images, and avoid harmful actions on live websites.
• Repository agents should return useful partial progress when full automation is unsafe or unsupported.
• Production agent speed is constrained by slow LLM calls, critic loops, and sequential action execution.

Architecture

The case study presents three complementary production-agent designs. Sam's personal assistant uses a WebSocket-to-Redis feedback path and router/template agents to constrain tools around common workflows. Div's MultiOn browser agent uses compressed DOM parsing, OCR, user override, planned critics, authorization, evals, and benchmarks to run web tasks. Devin's GitHub bot indexes repositories, triggers feature-request agents from issue webhooks, and uses checkpointing so partial findings such as files, issues, or code owners are useful even when the agent cannot complete a change.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-conversation["User conversation"] --> top-level-conversational-agent["Top-level conversational agent"]
  top-level-conversational-agent["Top-level conversational agent"] --> user-conversation["User conversation"]
  user-conversation["User conversation"] --> feedback-websocket["Feedback websocket"]
  feedback-websocket["Feedback websocket"] --> user-conversation["User conversation"]
  feedback-websocket["Feedback websocket"] --> redis-feedback-store["Redis feedback store"]
  redis-feedback-store["Redis feedback store"] --> feedback-websocket["Feedback websocket"]
  redis-feedback-store["Redis feedback store"] --> template-agents["Template agents"]
  template-agents["Template agents"] --> redis-feedback-store["Redis feedback store"]
  top-level-conversational-agent["Top-level conversational agent"] --> router-agent["Router agent"]
  router-agent["Router agent"] --> top-level-conversational-agent["Top-level conversational agent"]
  top-level-conversational-agent["Top-level conversational agent"] --> template-agents["Template agents"]
  template-agents["Template agents"] --> top-level-conversational-agent["Top-level conversational agent"]
  router-agent["Router agent"] --> template-agents["Template agents"]
  template-agents["Template agents"] --> router-agent["Router agent"]
  browser-automation-agent["Browser automation agent"] --> compressed-dom-parser["Compressed DOM parser"]
  compressed-dom-parser["Compressed DOM parser"] --> browser-automation-agent["Browser automation agent"]
  browser-automation-agent["Browser automation agent"] --> ocr-augmentation["OCR augmentation"]
  ocr-augmentation["OCR augmentation"] --> browser-automation-agent["Browser automation agent"]
  browser-automation-agent["Browser automation agent"] --> user-override-controls["User override controls"]
  user-override-controls["User override controls"] --> browser-automation-agent["Browser automation agent"]
  browser-automation-agent["Browser automation agent"] --> critic-and-evaluation-layer["Critic and evaluation layer"]
  critic-and-evaluation-layer["Critic and evaluation layer"] --> browser-automation-agent["Browser automation agent"]
  browser-automation-agent["Browser automation agent"] --> authorization-controls["Authorization controls"]
  authorization-controls["Authorization controls"] --> browser-automation-agent["Browser automation agent"]
  github-webhook-pipeline["GitHub webhook pipeline"] --> repository-index["Repository index"]
  repository-index["Repository index"] --> github-webhook-pipeline["GitHub webhook pipeline"]
  github-webhook-pipeline["GitHub webhook pipeline"] --> feature-request-agent["Feature request agent"]
  feature-request-agent["Feature request agent"] --> github-webhook-pipeline["GitHub webhook pipeline"]
  repository-index["Repository index"] --> feature-request-agent["Feature request agent"]
  feature-request-agent["Feature request agent"] --> repository-index["Repository index"]
  feature-request-agent["Feature request agent"] --> checkpointing-layer["Checkpointing layer"]
  checkpointing-layer["Checkpointing layer"] --> feature-request-agent["Feature request agent"]

Components

Data flow

For the personal assistant, user messages go to a conversational agent, while live feedback travels through a WebSocket into Redis and is read before the next planning stage. The router chooses a product-flow template agent with a constrained tool set or a fallback dynamic agent. For browser automation, MultiOn receives a task, parses DOM into a compact representation, augments it with OCR, executes browser actions, and can be paused or overridden. For GitHub, a repository is indexed, issue events trigger a cloud-function job, and a feature-request agent navigates files through checkpointed steps.

Why this architecture

The architectures emphasize bounded workflows over unconstrained autonomy. Router/template agents reduce tool-choice entropy, live feedback and override controls keep humans able to steer agents, compressed DOM state keeps browser automation within context limits, and checkpoints keep code-navigation outputs useful when full completion is not safe.

Models used: GPT-4 · GPT-3.5 · LangChain ReAct-style agents · MultiOn browser agent

Scale

• Sam's personal assistant had been built for about six months.
  “a personal assistant built on top of llms for about uh six months”transcript, stated
• MultiOn claimed to cover about 90% of websites in under 2K tokens.
  “we are actually able to protect 90 of the websites in less than 2K tokens”transcript, stated
• The parser may work on about 95% of websites, leaving roughly 5% needing customization.
  “this could work say like 95 of the website but like for the remaining five percent of the website you might have to like customize the password”transcript, stated
• DOM parsing was described as under 10 milliseconds, while vision would add latency.
  “this works really fast like less than 10 milliseconds but if increase latency”transcript, stated
• The GitHub bot had a two-minute timeout from its reverse proxy and used aggressive retriggering/resume behavior.
  “the reverse proxy I'm currently using to run this um is I'm having timeouts after two minutes”transcript, stated

Alternatives considered (roads not taken)

Ask for confirmation after every action

Rejected because: The panel describes that as frustrating compared with optional mid-run feedback.

Give one agent all CRUD/API tools

Rejected because: The agent ran out of context and could not reliably guess the right tools.

Use full DOM or pure vision for browser state

Rejected because: Full web context exceeds model context limits and vision adds latency, so MultiOn used compressed DOM plus OCR.

Force the GitHub bot to complete a full implementation

Rejected because: The code agent should stop or return useful partials rather than making confusing wrong changes.

Patterns

human-in-the-loop-gate (interaction)

Users can steer the running agent via feedback or browser override controls rather than approve every action.

multi-agent-orchestration (agent-architecture)

The panel uses routers, template agents, planner agents, critics, and checkpointed focused agents to coordinate complex workflows.

tool-use-orchestration (agent-architecture)

Tool access is routed through workflow-specific templates rather than giving every tool to every agent.

context-compaction (context-management)

MultiOn compresses HTML/DOM state to fit browser context into a small token budget.

eval-driven-development (quality)

The browser automation team planned critic agents, evals, and benchmarks to determine which tasks were reliable enough.

durable-execution (reliability)

The GitHub bot uses checkpoints and resume/retrigger behavior to survive timeouts and produce useful intermediate results.

latency-optimization (performance)

The panel recommends streaming, caching, smaller models, and parallel actions to make slow agent execution usable.

Tradeoffs

Use opt-in feedback rather than mandatory confirmations.

Chose Real-time user feedback channel over Approve/continue prompt at every step.

Rationale: The feedback channel keeps UX fluid while letting users intervene when the agent heads down the wrong path.

Route to constrained template agents.

Chose Top-level router with product-flow agents over One agent guessing among all tools.

Rationale: Constrained agents reduce context load and apply workflow-specific instructions for common tasks.

Use DOM parsing with OCR instead of full vision control.

Chose Compressed DOM representation plus OCR for icons/images over Sending full DOM or relying only on image models.

Rationale: The DOM parser is fast and compact, while OCR covers visual elements; full-context vision would be slower and context-heavy.

Prefer partial checkpoint outputs to wrong complete automation.

Chose Checkpointed search/find/explain stages over Forcing the agent to complete the whole code change.

Rationale: Partial outputs still reduce engineering work, while wrong changes add confusion.

Gotchas (what bit them)

Real-time agent mistakes waste time when the user cannot steer execution.

Root cause: The agent's planning loop kept following old observations even when the user saw a better path.

Resolution: Read live user feedback from Redis before the next planning stage and inject it as a special feedback concept in the prompt.

Tool proliferation made agents unreliable.

Root cause: Giving one agent CRUD tools across many APIs exhausted context and made tool prediction brittle.

Resolution: Group tools and instructions into product-flow templates selected by a top-level router.

Browser control created operational risk on live websites.

Root cause: The agent can operate logged-in web sessions and make changes, including to systems like AWS.

Resolution: Add interpretability, user override, critic/eval loops, monitoring, and authorization controls for site-specific read/write actions.

Critic loops improve accuracy but hurt responsiveness.

Root cause: Every action critique adds latency in an already slow agent workflow.

Resolution: Choose different planning/criticism combinations by use case and optimize for snappy UX where needed.

A code agent can create confusion if it guesses beyond its implemented scope.

Root cause: The GitHub bot's bug agent was not fully implemented, and complete code changes are not always reliable.

Resolution: Safely ignore unsupported requests or return checkpointed partials like files, related issues, and code owners.

Evaluation

The panel evaluates production readiness through user steering quality, router/template performance, browser automation reliability on target tasks, checkpoint usefulness, and latency mitigations like streaming/caching/parallelism.

Metrics: MultiOn claimed 90% website coverage under 2K tokens · Remaining 5% of websites may require parser customization · DOM parser path described as less than 10ms · Two-minute timeout for the GitHub bot's reverse proxy · Streaming all action/action-input observations to the user

Outcomes

• The personal assistant gained a live steering mechanism and router/template architecture.
• MultiOn demonstrated web automation across DoorDash, Amazon, email, GitHub, AWS, mobile, and plugin contexts while identifying reliability controls still needed.
• The GitHub bot could index repositories, react to issue events, find relevant files, and produce useful implementation guidance or partial results.
• The panel produced a production-agent playbook centered on routing, constrained tools, checkpoints, streaming, monitoring, and human intervention.

The source reports a six-month personal-assistant effort, 90% website coverage under 2K tokens for MultiOn, roughly 5% of websites needing parser customization, less-than-10ms parser performance, and a two-minute timeout constraint in the GitHub bot.