Building a Digital Workforce with Multi-Agent Systems and User-Centric Design

monday.com · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Work-management automation and agentic software products

monday.com wanted to automate work tasks inside its existing work operating system, but production agents had to earn user trust, operate on production board data, support human control, avoid brittle multi-agent decomposition, and scale from individual assistant actions toward a digital workforce handling millions of monthly requests.

Constraints:

• Users and companies have different risk appetites, so agent autonomy must be configurable rather than forced.
• Monday boards are production data, making direct agent writes risky without preview and control.
• Conversational agents face a huge unknown-action surface, with the speaker estimating that 99% of interactions may not have explicit handlers.
• Breaking a system into too many subagents compounds errors across agents.
• Agent workflows must fit existing Monday habits rather than requiring users to learn a new interface.

Architecture

monday.com built a digital workforce on LangGraph and LangSmith, using LangGraph as the central orchestration engine and LangSmith for monitoring. Around that core, monday.com added internal AI Blocks, a custom evaluation framework, and an AI Gateway that controls enabled inputs and outputs. The first digital worker, Monday Expert, uses a supervisor methodology with four agents: a supervisor, a data-retrieval agent for knowledge-base, board, and web-search context, a board-actions agent for Monday mutations, and an answer composer that tailors the final response. Product controls such as autonomy settings, previews, human-in-the-loop releases, undo, fallbacks, and external guardrails are treated as core architecture because trust is the adoption bottleneck.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  WorkOS["Monday Work OS"] --> Entry["Digital workforce entry points"]
  Entry --> Supervisor["Monday Expert supervisor"]
  Supervisor --> Retrieval["Data retrieval agent"]
  Supervisor --> Actions["Board actions agent"]
  Supervisor --> Composer["Answer composer"]
  Retrieval --> Sources["Knowledge base and web search"]
  Retrieval --> Boards["Board data and actions"]
  Actions --> Boards
  Composer --> Sources
  Supervisor --> Control["Preview and human control"]
  Actions --> Control
  LangGraph["LangGraph engine"] --> Supervisor
  LangGraph --> Blocks["AI Blocks"]
  LangGraph --> Gateway["AI Gateway"]
  Gateway --> Control
  LangGraph --> LangSmith["LangSmith monitoring"]
  LangSmith --> Evals["Evaluation framework"]
  Gateway --> Evals

Components

Data flow

A user works inside the Monday product and assigns a digital worker or asks the Monday Expert for help. The supervisor orchestrates a data-retrieval agent, board-actions agent, and answer composer. Data retrieval pulls from the knowledge base, board data, and web search; the board-actions agent prepares or executes changes; the answer composer uses conversation history, tone of voice, and user parameters to generate the response. The AI Gateway constrains inputs and outputs, the evaluation framework tests behavior, LangSmith monitors workflows, previews expose pending changes before saving, and undo lets the supervisor reverse actions based on feedback.

Why this architecture

monday.com chose an agent architecture that is deeply embedded in existing product workflows because adoption depended more on trust and user experience than on model capability alone. LangGraph handled orchestration primitives that the team did not want to rebuild, while still allowing customization. The supervisor pattern decomposed retrieval, action, and composition responsibilities, but the speaker also warned against unlimited decomposition because agent errors multiply. External guardrails, previews, evals, and gateway controls are used to make production board actions safe enough for users to accept.

Models used: LangGraph · LangSmith · Monday Expert · supervisor agent · data retrieval agent · board actions agent · answer composer · web search

Scale

• monday.com crossed $1 billion ARR in the year discussed.
  “we've actually crossed $1 billion ARR just this year”transcript, stated
• The platform processes around 1 billion work tasks per year.
  “we're actually processing around 1 billion tasks per year”transcript, stated
• monday.com reported 100% month-over-month AI-usage growth after launching its first AI feature.
  “We've been growing 100% month over month with our AI usage”transcript, stated
• The LangGraph-based agent system processes millions of requests per month.
  “we now process millions of requests per per month using Langraph. and it's proven to be super scalable”transcript, stated
• The speaker estimates conversational agents may only have explicit handling for about 1% of possible user interactions.
  “assume that 99% of user interactions you're not going to know how to handle”transcript, stated
• The speaker says productionizing AI can take a system from 80% to 99% quality over roughly another year.
  “it's one thing to bring AI to uh 80%. But then it takes another year to get to 99 99%”transcript, stated

Alternatives considered (roads not taken)

Build a new AI-only user experience

Rejected because: monday.com found that agents should fit existing product habits, such as assigning people or digital workers to tasks.

Let agents write directly to production boards

Rejected because: Users froze when the agent was about to push content to production board data, so monday.com introduced preview and control.

Put guardrails inside the LLM judgment loop

Rejected because: The speaker recommends building guardrails outside the LLM and gives Cursor's run limit as an example of an external guard.

Create a bespoke workflow for every rare business process

Rejected because: The earnings-report example would run once per quarter and might need rebuilding as models change, motivating dynamic orchestration from reusable agents.

Patterns

multi-agent-orchestration (architecture)

Monday Expert uses a supervisor methodology to coordinate specialized retrieval, board-action, and answer-composition agents.

human-in-the-loop-gate (ux)

Preview and human-loop controls let users inspect agent outputs before production board data is changed.

eval-driven-development (quality)

monday.com built an internal evaluation framework and treats evals as lasting IP because models and tooling change quickly.

observability-tracing (operations)

LangSmith is used alongside LangGraph for monitoring the agent ecosystem.

safety-policy-enforcement (safety)

An AI Gateway controls enabled inputs and outputs, while external guardrails constrain model-driven workflows outside the LLM itself.

Tradeoffs

Balance user control against agent autonomy.

Chose Configurable autonomy and user control over Fully autonomous agents acting around the clock by default.

Rationale: Different users and companies have different risk appetites, and monday.com saw adoption improve when control stayed in users' hands.

Use existing task-assignment UX for digital workers.

Chose Assign agents to tasks inside Monday over A separate AI product experience.

Rationale: Users already understand assigning people to work in Monday, so assigning digital workers avoids new habits.

Choose LangGraph as the orchestration foundation.

Chose LangGraph and LangSmith over Other tested agent frameworks or rebuilding orchestration primitives.

Rationale: LangGraph provided interrupts, checkpoints, memory, and human-loop support while staying customizable for Monday's needs.

Use reusable dynamic agents rather than bespoke one-off workflows for rare tasks.

Chose Finite agents dynamically orchestrated for many tasks over A purpose-built workflow for a once-per-quarter earnings process.

Rationale: A bespoke workflow may be obsolete by the next quarterly run, while dynamic agent composition can adapt across tasks.

Gotchas (what bit them)

Users froze when an agent was about to change production board data.

Root cause: The first interaction design allowed chat-driven board changes without enough visible preview for users to trust the save step.

Resolution: Add preview before saving so users can understand outputs and retain control.

Conversational agents cannot anticipate most user requests.

Root cause: The possible interaction space is effectively infinite compared with the small set of explicitly handled actions.

Resolution: Start by designing fallback behavior, such as searching the knowledge base and telling the user how to do unsupported actions themselves.

Production quality felt much farther away after shipping than it did in local development.

Root cause: Local confidence at roughly 80% quality did not cover the long tail of production behavior needed for a real product.

Resolution: Treat the 80-to-99% gap as a separate production-hardening effort with evals, human loops, and guardrails.

Too many specialized agents can reduce overall reliability through compound hallucination.

Root cause: Even individually good agents multiply their failure probabilities when chained together.

Resolution: Iterate on the right number of agents for the use case rather than decomposing every capability.

Evaluation

monday.com evaluates agents with a custom evaluation framework, LangSmith monitoring, production adoption behavior, preview/control adoption effects, fallback behavior for unknown requests, and production-hardening lessons from moving beyond local confidence.

Metrics: 100% month-over-month AI usage growth · Millions of LangGraph requests per month · Preview-driven adoption lift · Unknown-interaction fallback coverage · 80% to 99% production-quality gap · Compound reliability across chained agents

Outcomes

• monday.com launched a digital workforce after its first AI feature and reported 100% month-over-month growth in AI usage.
• The LangGraph-based agent ecosystem processes millions of requests per month.
• Preview controls increased adoption by giving users confidence before board changes were saved.
• The Monday Expert architecture established a supervisor-led, four-agent digital-worker pattern for retrieval, board actions, and response composition.
• The team is pursuing dynamic orchestration of reusable agents for future work processes.

The system is production-oriented and usage-backed: monday.com reports 1 billion work tasks per year, millions of LangGraph requests per month, and 100% month-over-month AI usage growth.