AI-Powered Customer Service Agent for Healthcare Navigation

alan · Healthcare · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Healthcare member support and reimbursement navigation

Alan needed to answer complex, personal healthcare and reimbursement questions instantly and empathetically for about 1 million members, while preserving escalation and domain-expert control.

Constraints:

Healthcare questions are personal, high-stakes, and more complex than password resets
Members need reliable guidance at scale
Privacy and security prevent exposing production conversations in demos

Architecture

Alan's healthcare support automation uses a classification agent to decide whether a member question has enough detail and which specialist should handle it, then routes to playbook-based specialist agents such as claims. Specialists run a ReAct loop, call tools for personalized member data, inject results back into prompts, redirect users into flows when needed, and are maintained through internal debug/configuration tooling owned by customer-service operators.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  member-question-interface["member-question-interface"]
  classification-agent["classification-agent"]
  specialist-playbook-agents["specialist-playbook-agents"]
  react-tool-loop["react-tool-loop"]
  member-data-tools["member-data-tools"]
  redirection-actions["redirection-actions"]
  debug-tool["debug-tool"]
  agent-configuration-tool["agent-configuration-tool"]
  synthetic-conversation-suite["synthetic-conversation-suite"]
  member-question-interface --> classification-agent
  classification-agent --> specialist-playbook-agents
  specialist-playbook-agents --> react-tool-loop
  specialist-playbook-agents --> redirection-actions
  react-tool-loop --> member-data-tools
  member-data-tools --> react-tool-loop
  redirection-actions --> member-question-interface
  debug-tool --> synthetic-conversation-suite
  debug-tool --> agent-configuration-tool
  agent-configuration-tool --> synthetic-conversation-suite

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`member-question-interface`	Entry point for member healthcare and reimbursement questions.	`classification-agent`
`classification-agent`	Routes questions by completeness and topic before handing off to specialists.	`specialist-playbook-agents`
`specialist-playbook-agents`	Domain-specific agents, including claims, that use playbooks plus deterministic controls.	`react-tool-loop`, `redirection-actions`
`react-tool-loop`	Reason-and-act cycle that selects tools, observes results, and decides whether enough context exists to answer.	`member-data-tools`
`member-data-tools`	Tools for claims, care events, member attributes, and other personalized healthcare data.	`react-tool-loop`
`redirection-actions`	Action mechanism for moving a member into flows such as reimbursement-document upload.	`member-question-interface`
`debug-tool`	Operator-facing trace and latency inspection tool for understanding agent answers and tool calls.	`synthetic-conversation-suite`, `agent-configuration-tool`
`agent-configuration-tool`	CI-like safe-change system for prompts, member attributes, redirections, tools, offline evals, and production pushes.	`synthetic-conversation-suite`
`synthetic-conversation-suite`	Test conversations and offline evaluations used by operators to verify agent changes.	—

Data flow

A member asks a healthcare question. A classification agent assesses detail and routes the request to a specialist agent. The specialist follows a playbook, reasons about the conversation state, calls tools for member-specific data such as care events, observes results, continues successive tool calls until enough context exists, then answers or redirects the member into a reimbursement flow. Debug and configuration tools expose traces, latency, prompts, tools, redirections, synthetic conversations, and evals to domain operators.

3 supporting quotes

“when we get the member question we have a classification agent. that agent has a clear objective is to determine if they have enough details”transcript, stated

“for each exper agent first we're going to analyze the conversations where do we stand? Can we call tools to make the answer personal?”transcript, stated

“it will iterate on the success successive tool calls until it has everything it needs to answer the member.”transcript, stated

Why this architecture

Alan shifted from deterministic workflows to playbooks because newer models could handle richer reasoning while still benefiting from targeted deterministic controls. The company also chose custom orchestration and heavy internal tooling so healthcare domain experts, not only engineers or data scientists, could debug, configure, and safely evolve agents at production speed.

3 supporting quotes

“we started with workflows every single agent uh that we had um well followed rules that were super deterministic”transcript, stated

“playbooks is uh a real advantage to harness the full power of AI and which doesn't prevent to include a few uh deterministics”transcript, stated

“customer service operators to maintain to improve agents over time and that's what I'm going to show you.”transcript, stated

Models used: LLMs

Scale

Alan supports about 1 million members today.
“Today we support 1 million number members and we plan to support 10 millions”transcript, stated
Alan plans to support 10 million members by 2030.
“we plan to support 10 millions uh in uh by by 2030.”transcript, stated
About 60% of reimbursements are processed under five minutes.
“most 60% of our reimbursements are performed under five minutes.”transcript, stated
The agent answers between 30% and 35% of member service questions.
“our agent is able to answer between 30 and 35% of the question for our members with a quality comparable to our care experts.”transcript, stated
Manager/orchestrator complexity was not justified for only about 4-5% of conversations.
“manage our eal and when you take a step back it's only for 4% four to 5% of our”transcript, stated

Alternatives considered (roads not taken)

Pure deterministic workflows

Rejected because: Alan found playbooks better matched newer model capability while still allowing deterministic controls inside a playbook.

“we started with workflows every single agent uh that we had um well followed rules that were super deterministic”transcript, stated

Off-the-shelf agent orchestrators

Rejected because: The team considered LangGraph and Pydantic AI but decided a custom orchestration framework fit better.

“we considered uh using offtheshelves uh agent orchestrators like langraph or uh padentic AI and uh we finally decided that it was better to build our own”transcript, stated

Adding more and more tools directly to agents

Rejected because: More tools made agents struggle with selecting the right call and arguments, so they removed parameters and combined common tools.

“to bring more and more tools the agent have started to sometimes struggle a bit to make the right call with the right”transcript, stated

Manager/orchestrator for every multi-topic question

Rejected because: The design solved a real challenge but added tooling and evaluation complexity for a small share of current conversations.

“different design with um a manager and orchestrator able to um call the different agents.”transcript, stated

“also brings a lot of complexity a lot of complexity about how we're going to manage our tooling how we're going to manage our eal”transcript, stated

Patterns

`multi-agent-orchestration` (member-support-routing)

A first classification agent determines whether it can hand off to a specialist agent such as the claims agent.

“The first part is when we get the member question we have a classification agent. that agent has a clear objective is to determine if they have enough details”transcript, stated

`tool-use-orchestration` (claims-reimbursement)

Specialist agents run a ReAct loop that decides which member/reimbursement tools to call, observes results, and generates the answer.

“So for that we use a react loop. So react is a name for reason and act. So for each exper agent first we're going to analyze the conversations”transcript, stated

`observability-tracing` (agent-debugging)

Alan exposes traces, latency for each API call, and every reasoning/tool step so operators and engineers can inspect agent behavior.

“what you see on this screen first is the trace. So it's basically the latency of each API call the agent makes.”transcript, stated

`eval-driven-development` (domain-expert-operations)

A CI-like configuration system lets customer-service operators test agent changes safely before pushing them to production.

“it's basically a CI like system that is that allows us to trace the different changes we make on our agents and also it allows customer service operators to test changes”transcript, stated

`human-in-the-loop-gate` (care-escalation)

The design keeps escalation to a care expert available whenever the automated conversation should not continue alone.

“Wanted to get escalation to be sure that at any moment we can escalate to a care expert.”transcript, stated

`canary-rollout` (coverage-expansion)

Alan started with one specialist agent, validated results, then expanded topic coverage step by step.

“we started really small with only one agent and we have been able to iterate really fast.”transcript, stated

Tradeoffs

How broad to make production orchestration.

Chose classification plus specialist agents for current production over manager/orchestrator for every multi-topic conversation.

Rationale: The orchestrator handled multi-topic questions, but added complexity that did not justify shipping for 4-5% of conversations.

“it's only for 4% four to 5% of our conversation right now so it's really a question that uh we're thinking about is it the right problem to solve”transcript, stated

Who should maintain agent knowledge.

Chose customer-service operators with tooling over engineers and data scientists acting as the prompt-engineering bottleneck.

Rationale: Domain experts know insurance content best, and tooling lets them improve agents safely without waiting on engineering for every change.

“we've invested a lot in our internal tooling to be sure that engineers or data scientists are not going to be um bottleneck”transcript, stated

Gotchas (what bit them)

Adding more tools made agents struggle with the right call and arguments.

Root cause: LLMs had to select tools and pass identifiers/arguments correctly as the tool surface grew.

Resolution: Alan removed as many function-call parameters as possible, combined frequently used tools, specified expected parameters, and added robust error handling.

“to bring more and more tools the agent have started to sometimes struggle a bit to make the right call with the right arguments.”transcript, stated

Architecture alone was not enough to keep agents improving.

Root cause: The useful input to the agent is insurance knowledge, which customer-service operators understand better than engineers alone.

Resolution: Alan built tooling so domain experts can maintain, improve, debug, and configure agents.

“architecture alone is not enough. Um, another aspect which is key and I really wanted to tell you a bit more about it tonight is to provide tools”transcript, stated

Manager/orchestrator design solved multi-topic questions but was too complex for current volume.

Root cause: Multi-topic conversations represented only about 4-5% of conversations, while orchestration complicated tooling and evaluation.

Resolution: Alan kept the design available experimentally but did not ship it to production yet.

“it also brings a lot of complexity a lot of complexity about how we're going to manage our tooling how we're going to manage our eal”transcript, stated

Evaluation

Alan evaluates quality against care experts, inspects traces and latency, reviews hundreds of conversations, tests configuration changes in safe branches, and compares tool-calling reliability over time.

Metrics: automation rate · quality comparable to care experts · latency per API call · tool-call correctness · conversation review volume · production rollout decision

“our agent is able to answer between 30 and 35% of the question for our members with a quality comparable to our care experts.”transcript, stated

“once you've had a look at 100 200 conversations that you've understood what are the strengths and weaknesses of your agent”transcript, stated

Outcomes

Alan automates about 30-35% of member questions with quality comparable to care experts.
The system supports multi-turn conversations, smart direction, personal answers, and escalation.
Domain experts can configure and debug agents through internal tooling.
Alan has not yet put the more complex manager/orchestrator design into production.

Alan supports about 1 million members and uses the agent for healthcare navigation and reimbursement/claims support.

“At Alan we support around 1 million members and we want to support even more”transcript, stated

“step by step we have been able to cover 30% uh 30 between 30 and 35% of uh the members questions”transcript, stated