Implementing Multi-Agent RAG Architecture for Customer Care Automation

doctolib · Healthcare · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agentic customer-care automation for healthcare support workflows

Doctolib wanted to move beyond a basic RAG customer-care assistant so the system could handle multi-step support tasks, ask clarifying questions, fetch user/context data, execute back-end actions, and route between specialized domains without letting a single large prompt or general-purpose agent become unreliable.

Constraints:

• the work is a proof of concept rather than a deployed production system
• traditional RAG was limited for multi-step action-oriented support tasks
• LLM agents can invoke the wrong tool or call tools with incorrect parameters
• large prompts with tool descriptions, execution details, and message history suffer from positional attention bias
• unstructured data extraction is hard to perform consistently
• sensitive execution tools require user validation before action
• production deployment depends on design, feature, product, and ML platform teams
• LangGraph was new and had few real production examples

Architecture

Doctolib selected LangGraph to build a hierarchical multi-agent RAG architecture for customer care. A root assistant greets the user, converses until a clear Doctolib query emerges, and routes the query with an ML classifier to a specialized assistant. Each specialized assistant owns one or two use cases, with a smaller prompt, fewer tools, and focused context. Assistants can use data-fetching tools, the existing FAQ/RAG search tool, sensitive execution tools for back-end customer-support actions, and completion/cancellation tools that return the flow to the root assistant. Sensitive tools include a fact-check step to verify arguments before execution and require explicit user validation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  User["User"] --> Root_assistant["Root assistant"]
  Root_assistant --> ML_routing_classifier["ML routing classifier"]
  ML_routing_classifier --> Specialized_assistants["Specialized assistants"]
  Root_assistant --> Specialized_assistants
  Specialized_assistants --> Data_fetching_tools["Data fetching tools"]
  Specialized_assistants --> FAQ_search_tool["FAQ search tool"]
  Specialized_assistants --> Sensitive_execution_tools["Sensitive execution tools"]
  Specialized_assistants --> Task_completion_tools["Task completion tools"]
  Data_fetching_tools --> Doctolib_databases["Doctolib databases"]
  FAQ_search_tool --> FAQ_knowledge_base["FAQ knowledge base"]
  Sensitive_execution_tools --> Fact_check_step["Fact-check step"]
  Fact_check_step --> User_validation_gate["User validation gate"]
  User_validation_gate --> User
  User_validation_gate --> Customer_support_back_end["Customer support back end"]
  Task_completion_tools --> Root_assistant
  LangGraph_runtime["LangGraph runtime"] --> Root_assistant
  LangGraph_runtime --> Specialized_assistants
  LangGraph_runtime --> Deterministic_functions["Deterministic functions"]
  Doctolib_databases --> Data_referential["Data referential"]
  API_documentation["API documentation"] --> Data_referential
  API_documentation --> Sensitive_execution_tools
  Evaluation_tooling["Evaluation tooling"] --> LangGraph_runtime

Components

Data flow

A user asks a customer-care question. The root assistant interacts until the request is clear, then routes it to a specialized assistant using an ML classification model. The specialized assistant may fetch contextual data, search FAQs through the original RAG component, ask clarifying questions, and prepare a sensitive execution tool call. Before an action such as granting agenda access runs, the system checks tool arguments and asks the user for validation. When the task is completed or canceled, a completion tool loops the conversation back to the root assistant.

"The root assistant goals are :

• Interacting with the user until a clear query is mentioned regarding Doctolib"

Why this architecture

Doctolib chose specialized agents and LangGraph to reduce the pressure on any one LLM, constrain prompts and tool sets, and enable loops, conditional branches, and human validation around sensitive actions. The design is intended to make customer-care tasks more generalizable while reducing hallucination, tool-choice variance, and prompt-size problems.

Models used: LLM-based root assistant · LLM-based specialized assistants · ML classification model · LangGraph · LangChain

Scale

• The article is a 10-minute-read Part 2 case study published on Oct. 7, 2024.

  "10 min read

Oct 7, 2024"

— article, stated

• Each specialized assistant is scoped to one or two use cases.
  “each specialized assistant is meant to handle requests over a fixed scope (one or two use cases).”article, stated
• The first demonstrated use case is granting secretary access to a user's agenda.
  “The user wants to give his secretary access to his agenda.”article, stated
• The production path depends on multiple teams: design, feature teams, PM, and ML Platform.
  “we need a strong collaboration and synchronisation with the different teams involved (design, feature teams, PM, ML Platform…).”article, stated
• LangGraph is described as having few real production use cases at the time.
  “LangGaph is a new and moving library. It does not provide many real use cases that went into production.”article, stated

Alternatives considered (roads not taken)

Continue with a basic RAG-only system

Rejected because: Doctolib needed multi-step reasoning, tool use, API calls, follow-up questions, and action execution beyond retrieval and response generation.

Hard-coded bot flow

Rejected because: The team wanted agents to decide which actions to take and in what sequence instead of following a hard-coded next step.

One generalist agent handling all customer-care tasks

Rejected because: A broad agent would have larger prompts, more tools, and more behavior variance, so Doctolib split scope across specialized assistants.

Other agent frameworks such as CrewAI or AutoGen

Rejected because: Doctolib selected LangGraph for flexibility, security guardrails, simpler user interaction, clearer documentation, and LangChain ecosystem integration.

Execute sensitive customer-support actions without user validation

Rejected because: Back-end actions require explicit validation and argument checks before being run on a user's behalf.

Patterns

multi-agent-orchestration (agent-architecture)

Doctolib coordinates a root assistant with several specialized assistants, each scoped to limited use cases and tools.

rag-retrieval (knowledge-architecture)

The original RAG/FAQ search system remains available as one tool that specialized agents can invoke.

tool-use-orchestration (agent-architecture)

Specialized assistants choose between context-fetching tools, FAQ search, sensitive execution tools, and completion/cancellation tools.

human-in-the-loop-gate (governance)

Sensitive tools require explicit user validation before customer-support actions are executed.

safety-policy-enforcement (governance)

The system classifies sensitive tools and fact-checks tool arguments before allowing actions against back-end systems.

data-quality-loop (quality)

Doctolib treats data referentials, API documentation, OpenAPI specs, governance, and labeled data as core inputs for improving agent reliability.

eval-driven-development (quality)

The team identifies multi-component evaluation as critical for finding performance bottlenecks and error root causes before production deployment.

context-compaction (context-management)

The specialized-agent design reduces prompt size and tool scope to mitigate long-context attention problems.

Tradeoffs

LangGraph framework selection

Chose LangGraph on top of LangChain over CrewAI, AutoGen, or continuing without an agentic framework.

Rationale: Doctolib preferred LangGraph's flexibility, sensitive-tool guardrails, simpler user interaction, clearer documentation, and LangChain integration.

Specialized assistants

Chose multiple agents with fixed one-or-two-use-case scopes over one general assistant with all tools and instructions.

Rationale: Specialization limits prompt size, tool count, and freedom of action, reducing variance and improving reliability.

Sensitive tool validation

Chose fact-check and user validation before execution over agent-only execution of back-end actions.

Rationale: Customer-support actions can affect real user accounts, so sensitive tools need argument checks and explicit user approval.

Documentation as agent input

Chose API docs, OpenAPI specs, and data referentials as structured inputs over hand-written agent prompts disconnected from source-of-truth APIs.

Rationale: Correct action execution depends on accurate tool arguments, API semantics, and governed data descriptions.

POC before production

Chose continue iterating on robustness, evaluation, and cross-team dependencies over declaring the agentic architecture production-ready.

Rationale: The team acknowledges LangGraph maturity, evaluation complexity, robustness, and organizational dependencies still need work.

Gotchas (what bit them)

Agents do not consistently choose the right tool or parameters.

Root cause: LLM behavior is non-deterministic, especially when prompt size and degrees of freedom grow.

Resolution: Reduce the task expected from each LLM, specialize assistants, limit tools, and fact-check sensitive tool arguments.

Large agent prompts cause instruction-following degradation.

Root cause: Agent prompts include tool descriptions, execution details, message history, and other relevant information, creating lost-in-the-middle risk.

Resolution: Limit specialized assistant scope, prompt size, and tool count.

Unstructured context extraction is unreliable.

Root cause: Agents must extract useful information from unstructured data and interpret it to ask relevant questions.

Resolution: Create a clear data referential and labeled data to guide agents through smaller manageable tasks.

API and data documentation become part of the AI system.

Root cause: Agents need natural-language descriptions of product APIs, tool arguments, tables, and tasks to execute actions correctly.

Resolution: Use OpenAPI specifications, strong data governance, and high-quality code documentation as data sources for the agents.

Evaluating multi-agent systems is not a single-model metric problem.

Root cause: The system has many interdependent bricks that need separate evaluation to locate bottlenecks and root causes.

Resolution: Evaluate components separately and use tracing/evaluation frameworks such as Literal or LangSmith for root-cause analysis.

Productionizing the POC requires organizational change.

Root cause: Production deployment depends on cross-team synchronization, governed data assets, and a robust library/tooling foundation.

Resolution: Coordinate design, feature, PM, ML platform, data governance, and documentation work before production rollout.

Evaluation

Doctolib explicitly says evaluation is critical but complex because the system has many components that must be evaluated separately. The article does not report quantitative performance metrics; it identifies needed evaluation tooling such as Literal or LangSmith, plus component-level evaluation to find bottlenecks and root causes before production confidence.

Metrics: no quantitative production metrics reported · component-level evaluation required · tool-choice correctness · tool-argument correctness · root-cause analysis through Literal or LangSmith · robustness level required before production deployment · data referential freshness and governance

Outcomes

• Doctolib created a POC multi-agent customer-care architecture using LangGraph.
• The architecture demonstrates routing an agenda-rights request to a specialized assistant.
• The system can fetch context data, validate tool arguments, ask user confirmation, and execute an action on the user's behalf.
• The team identified agent behavior variance, prompt size, unstructured data, evaluation, and production deployment as remaining blockers.
• The article does not report production deployment, accuracy, support deflection, or user satisfaction metrics.

This is valuable precisely because it is honest POC evidence. The architecture is strong, but the extraction should not count it as proof that multi-agent customer-care automation is production-ready. Its durable lessons are specialization, sensitive-tool validation, and documentation/data governance as prerequisites for action-taking agents.