Production Agents: Real-world Implementations of LLM-powered Autonomous Systems

various · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production autonomous agents across personal assistants, browser automation, and codebase navigation

Practitioners needed to make LLM agents useful in production despite rabbit-hole behavior, tool overload, unreliable browser control, slow response times, codebase-navigation limits, and the risk of doing the wrong thing instead of safely stopping.

Constraints:

• ReAct-style agents can go down unproductive paths while users watch without a clean way to steer them.
• Large tool sets cause context pressure and poor tool selection.
• Browser automation must work across many websites with messy DOMs, icons, images, and user authorization boundaries.
• Code-navigation agents need repository indexing, checkpoints, retries, and partial progress outputs rather than pretending to produce perfect code.
• Agents are powerful but slow, so production UX needs streaming, caching, parallelism, or smaller models.

Architecture

The panel presents three production agent architectures. Sam's personal assistant uses a ReAct loop with a WebSocket/Redis feedback channel and router/template agents that constrain tools by product flow. MultiOn uses a compressed DOM parser, OCR, browser controller, user override UI, planned critic, authorization rules, and evals for web automation. Devin's GitHub bot indexes repositories, receives issue events through a webhook/topic/cloud-function pipeline, runs a feature-request agent, and uses checkpointing so intermediate findings remain useful even when the agent cannot complete the full request.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  conversational-agent["Conversational agent"] --> feedback-websocket["Feedback websocket"]
  feedback-websocket["Feedback websocket"] --> conversational-agent["Conversational agent"]
  conversational-agent["Conversational agent"] --> router-agent["Router agent"]
  router-agent["Router agent"] --> conversational-agent["Conversational agent"]
  conversational-agent["Conversational agent"] --> template-task-agents["Template task agents"]
  template-task-agents["Template task agents"] --> conversational-agent["Conversational agent"]
  feedback-websocket["Feedback websocket"] --> redis-feedback-store["Redis feedback store"]
  redis-feedback-store["Redis feedback store"] --> feedback-websocket["Feedback websocket"]
  redis-feedback-store["Redis feedback store"] --> template-task-agents["Template task agents"]
  template-task-agents["Template task agents"] --> redis-feedback-store["Redis feedback store"]
  router-agent["Router agent"] --> template-task-agents["Template task agents"]
  template-task-agents["Template task agents"] --> router-agent["Router agent"]
  multion-browser-controller["MultiOn browser controller"] --> compressed-dom-parser["Compressed DOM parser"]
  compressed-dom-parser["Compressed DOM parser"] --> multion-browser-controller["MultiOn browser controller"]
  multion-browser-controller["MultiOn browser controller"] --> ocr-layer["OCR layer"]
  ocr-layer["OCR layer"] --> multion-browser-controller["MultiOn browser controller"]
  multion-browser-controller["MultiOn browser controller"] --> user-override-ui["User override UI"]
  user-override-ui["User override UI"] --> multion-browser-controller["MultiOn browser controller"]
  multion-browser-controller["MultiOn browser controller"] --> critic-agent["Critic agent"]
  critic-agent["Critic agent"] --> multion-browser-controller["MultiOn browser controller"]
  multion-browser-controller["MultiOn browser controller"] --> authorization-panel["Authorization panel"]
  authorization-panel["Authorization panel"] --> multion-browser-controller["MultiOn browser controller"]
  github-issue-event-pipeline["GitHub issue event pipeline"] --> repository-index["Repository index"]
  repository-index["Repository index"] --> github-issue-event-pipeline["GitHub issue event pipeline"]
  github-issue-event-pipeline["GitHub issue event pipeline"] --> feature-request-agent["Feature request agent"]
  feature-request-agent["Feature request agent"] --> github-issue-event-pipeline["GitHub issue event pipeline"]
  repository-index["Repository index"] --> feature-request-agent["Feature request agent"]
  feature-request-agent["Feature request agent"] --> repository-index["Repository index"]
  feature-request-agent["Feature request agent"] --> checkpoint-controller["Checkpoint controller"]
  checkpoint-controller["Checkpoint controller"] --> feature-request-agent["Feature request agent"]

Components

Data flow

A user request enters a conversational or web interface. For personal-assistant tasks, the top-level conversational agent routes the request to a template/background agent and watches a Redis-backed feedback stream before each planning step. For browser automation, the agent receives a task, parses the page DOM into a compressed representation, augments with OCR, controls the browser, and lets the user pause or override. For GitHub work, an issue event triggers a background feature-request agent over an indexed repository; the agent finds relevant files, emits interim checkpoints, and either suggests implementation steps or safely declines unsupported bug requests.

Why this architecture

The panelists converge on agents as systems rather than single prompts: routing constrains tool choice, live feedback keeps users in the loop without approving every step, browser automation needs compressed state plus override/authorization, and code agents need checkpoints so partial progress is valuable and unsupported requests can safely stop.

Models used: GPT-4 · GPT-3.5 · LangChain ReAct-style agents · custom browser agent models

Scale

• Sam's personal assistant had been under development for about six months.
  “a personal assistant built on top of llms for about uh six months”transcript, stated
• MultiOn claimed about 90% website coverage in under 2K tokens.
  “we are actually able to protect 90 of the websites in less than 2K tokens”transcript, stated
• The remaining roughly 5% of websites may need parser customization or other tricks.
  “this could work say like 95 of the website but like for the remaining five percent of the website you might have to like customize the password”transcript, stated
• The DOM parser path was described as less than 10 milliseconds, while vision models would add latency.
  “this works really fast like less than 10 milliseconds but if increase latency”transcript, stated
• The GitHub bot hit a two-minute reverse-proxy timeout and retriggered the agent aggressively.
  “the reverse proxy I'm currently using to run this um is I'm having timeouts after two minutes”transcript, stated

Alternatives considered (roads not taken)

Ask the user to approve every agent step

Rejected because: The personal-assistant team found constant confirmation frustrating and preferred optional live feedback.

Give one agent every CRUD tool for every API

Rejected because: The agent ran out of context and guessed tool sets poorly.

Use fully automatic browser control with no user override or authorization boundaries

Rejected because: Browser agents can make mistakes on real websites, so users need interpretability, pause/override controls, and site-level permissions.

Require the code agent to produce a complete correct PR every time

Rejected because: The GitHub bot explicitly treats partial findings as useful and prefers stopping to doing the wrong thing.

Patterns

human-in-the-loop-gate (interaction)

Sam's agent lets users intervene mid-execution through live feedback rather than forcing confirmation at every action.

multi-agent-orchestration (agent-architecture)

The examples use router agents, template agents, planner agents, critic agents, and chained focused agents to decompose production workflows.

tool-use-orchestration (agent-architecture)

Agents are constrained to workflow-specific tool sets, with router/template selection deciding which tools or sub-agents should execute.

context-compaction (context-management)

MultiOn compresses website DOM state into a compact representation so the browser agent can reason over web pages within context limits.

eval-driven-development (quality)

The browser-agent team planned critics, evals, and benchmarks to decide where automation is reliable enough for production tasks.

durable-execution (reliability)

The GitHub bot uses checkpoints, retries, and resumable partial outputs so long-running code-navigation work survives timeouts and remains useful.

latency-optimization (performance)

The panel discusses streaming, caching, parallel actions, smaller models, and avoiding critic loops where latency would hurt UX.

Tradeoffs

Use optional live feedback instead of step-by-step confirmations.

Chose User opt-in feedback while the agent runs over Mandatory approval at each action.

Rationale: This keeps the experience seamless while still allowing users to redirect the agent when it goes off course.

Use routers and workflow templates to constrain agents.

Chose Product-flow-specific template agents over One general agent with every API tool.

Rationale: Constrained tool sets and instructions avoided context overflow and helped agents perform common workflows without reinventing the wheel.

Use compressed DOM parsing before vision-heavy browser control.

Chose Custom DOM parser with OCR augmentation over Sending full page context or relying entirely on vision.

Rationale: Full web pages exceed context limits, while the custom parser is fast and compact; OCR fills in icons/images where text misses elements.

Prefer useful partial progress over wrong automation in codebase tasks.

Chose Checkpointed partial outputs and safe stopping over Always pushing toward full code changes.

Rationale: Relevant files, similar issues, or code-owner guidance can still reduce work, while wrong code creates confusion.

Use streaming and caching to offset agent latency.

Chose Streaming outputs, cached LLM calls, smaller/custom models, and parallel actions over Blocking user experience while one slow agent finishes.

Rationale: Agents are slow, and production users need visibility and responsiveness during long-running work.

Gotchas (what bit them)

Agents went down rabbit holes without mid-run user steering.

Root cause: The ReAct loop kept executing based on prior observations even when the user saw it pursuing the wrong path.

Resolution: Add a WebSocket/Redis feedback channel read before each planning stage and teach the prompt to prioritize recent user feedback.

Tool explosion caused poor tool selection.

Root cause: A single agent with CRUD operations across many APIs ran out of context and guessed badly.

Resolution: Constrain agents into product-flow templates with task-specific tools and instructions, plus a fallback dynamic agent.

Browser agents can make real-world mistakes on live websites.

Root cause: Autonomous web control can click or change state on arbitrary websites, including sensitive systems like AWS.

Resolution: MultiOn uses user interpretability, pause/override, planned critic agents, benchmarks, and site-level authorization controls.

Critic agents improve accuracy but add latency.

Root cause: Checking every browser action with another agent slows down an experience that users expect to be snappy.

Resolution: Use architecture combinations by use case and optimize between accuracy and speed.

Code agents should not fabricate complete work when only partial navigation is reliable.

Root cause: Repository navigation and code modification can exceed the model's ability or the implemented agent's scope.

Resolution: Use checkpoints and partial outputs such as relevant files, similar issues, or code owners; unsupported bug requests can be ignored rather than mishandled.

Evaluation

The panel evaluates agents through production usability: whether users can steer them, whether routing improves tool selection, whether browser tasks are reliable on target websites, whether checkpoints produce useful partial progress, and whether streaming/caching/parallelism make latency tolerable.

Metrics: MultiOn claimed roughly 90% website coverage in under 2K tokens · Remaining 5% of websites needing parser customization · DOM parser latency under 10ms · Two-minute timeout pressure on the GitHub bot · Examples of prompt 2.4-style live experiment improvements were discussed as a general production pattern

Outcomes

• The personal assistant gained a live feedback loop and router/template architecture for common workflows.
• MultiOn demonstrated browser automation across sites including DoorDash, Twitter, GitHub, AWS, and mobile/plugin contexts, while acknowledging reliability and authorization work.
• The GitHub bot indexed repositories, responded to issue events, and produced implementation guidance or partial checkpoints.
• The panel converged on production agents as systems with routing, monitoring, checkpointing, human feedback, streaming, and carefully scoped tools.

The panel covers a six-month personal-assistant effort, MultiOn's claimed 90% website coverage under 2K tokens, a two-minute timeout issue in the GitHub bot, and multiple production latency mitigations such as streaming, caching, smaller models, and parallel actions.