Implementing RAG for Enhanced Customer Care at Scale

doctolib · Healthcare · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: RAG-enhanced customer care for healthcare platform support

Doctolib wanted to improve customer care beyond rules-based bots and public-knowledge LLM answers. Customer-care users needed answers grounded in Doctolib's frequently updated private FAQ content, with enough precision, latency, and UX quality to reduce cases for human agents while avoiding complex unsupported interactions that basic RAG could not handle.

Constraints:

• rules-based bots could not adapt to user context
• LLMs lacked access to recent and private Doctolib data
• standard RAG worked best for simple FAQ-style queries and struggled with complex user questions
• FAQ content changed daily and needed a refresh pipeline
• basic vector search was not precise enough without reranking
• latency initially reached about 1 minute and had to fall below 5 seconds
• the system could only provide descriptive answers and could not take back-office actions

Architecture

Doctolib built a production RAG pipeline for customer care. FAQ articles are updated daily, embedded into OpenSearch as vector chunks, and retrieved for user questions. The system uses GPT-4o hosted on Azure OpenAI for security and confidentiality, OpenSearch as the vector database, and a reranking component because vector search alone was not precise enough. A Ragas-based evaluation tool measures context precision, context recall, faithfulness, and answer relevancy, supported by annotated groundtruth records. UX work added an ML classifier that predicts whether the RAG system can answer a query, trading reach for precision by routing unanswerable questions away from automation. Latency was reduced from roughly 1 minute to under 5 seconds through code optimization, Azure OpenAI PTUs, smaller-model tests, and streaming.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Customer_care_user["Customer-care user"] --> Assistant_UI["Assistant UI"]
  Assistant_UI --> Answerability_classifier["Answerability classifier"]
  Answerability_classifier --> Human_customer_care_agents["Human customer-care agents"]
  Answerability_classifier --> Query_embedding_step["Query embedding step"]
  FAQ_articles["FAQ articles"] --> Daily_FAQ_refresh_pipeline["Daily FAQ refresh pipeline"]
  Daily_FAQ_refresh_pipeline --> Embedding_step["Embedding step"]
  Embedding_step --> OpenSearch_vector_database["OpenSearch vector database"]
  Query_embedding_step --> Embedding_step
  Query_embedding_step --> OpenSearch_vector_database
  OpenSearch_vector_database --> Reranking_brick["Reranking brick"]
  Reranking_brick --> Answer_generator["Answer generator"]
  Answer_generator --> Azure_OpenAI_GPT_4o["Azure OpenAI GPT-4o"]
  Azure_OpenAI_GPT_4o --> Provisioned_Throughput_Units["Provisioned Throughput Units"]
  Answer_generator --> Streaming_delivery["Streaming delivery"]
  Streaming_delivery --> Assistant_UI
  Ragas_evaluation_tool["Ragas evaluation tool"] --> Groundtruth_records["Groundtruth records"]
  Groundtruth_records --> Experiment_loop["Experiment loop"]
  Ragas_evaluation_tool --> Experiment_loop
  Experiment_loop --> Reranking_brick
  Experiment_loop --> Answer_generator
  Experiment_loop --> Daily_FAQ_refresh_pipeline

Components

Data flow

A daily data pipeline retrieves updated FAQ content, chunks and embeds it, and writes embeddings into OpenSearch. At runtime, a user asks a customer-care question through the assistant UI. An ML classifier predicts whether the query is answerable by the RAG system. If active, the query is embedded, FAQ chunks are retrieved from OpenSearch, reranked, and passed with the question to GPT-4o on Azure OpenAI to generate an answer grounded in context. Evaluation experiments use Ragas metrics and groundtruth records to tune retrieval, prompts, chunking, embedding models, LLM parameters, and reranking.

Why this architecture

RAG gave GPT-4o access to recent private FAQ knowledge that the base model lacked, while Azure OpenAI addressed healthcare security and confidentiality concerns. Reranking improved retrieval precision beyond vector search, Ragas and groundtruth records made optimization systematic, and an answerability classifier improved UX by reducing cases where the system attempted questions it could not answer.

As we embarked on the journey of implementing a RAG"

— article, stated

Models used: GPT-4o · Azure OpenAI · Machine Learning classifier model · embedding models · Ragas evaluation framework

Scale

• Doctolib was founded in 2013 and is a leading European e-health company.
  “_Founded in 2013, Doctolib is a leading e-health company in Europe.”article, stated
• The team included 40 data scientists working on AI products.
  “our team of 40 data scientists is working on developing groundbreaking AI products”article, stated
• The FAQ embedding pipeline refreshes every day.
  “retrieves the up-to-date FAQ and embeds it in the vector database everyday.”article, stated
• The RAG implementation reduced customer-care case deflection by 20%.
  “This improved our vanilla RAG. It has had a significant impact on customer care cases deflection with a 20% reduction.”article, stated
• Latency improved from about 1 minute to less than 5 seconds.
  “We had a long journey to go from a latency of 1 min to less than 5s. To do so, we tested several strategies”article, stated
• A consultation assistant was planned for release in Q4 2024.
  “our team of 40 data scientists is working on developing groundbreaking AI products (such as a consultation assistant which we will release in Q4 2024).”article, stated

Alternatives considered (roads not taken)

Rules-based bots

Rejected because: They could not adapt to users or context and required constant updates for new tasks.

Base LLM answers without retrieval

Rejected because: LLMs lacked recent private Doctolib knowledge, limiting accuracy for company-specific customer-care questions.

Vector search without reranking

Rejected because: The initial retrieval step alone did not produce good enough results.

Answer every query through RAG

Rejected because: The team added an ML classifier to avoid frustrating users when the system could not answer.

Stay with basic RAG for complex and action-oriented support

Rejected because: RAG could not handle complex queries, back-office actions, or seamless multi-turn conversation, motivating exploration of agents.

Patterns

rag-retrieval (knowledge-architecture)

Doctolib grounds GPT-4o answers in daily refreshed FAQ articles retrieved from OpenSearch.

retrieval-ranking (retrieval)

A reranking component improves initial vector retrieval before answer generation.

eval-driven-development (quality)

Doctolib built a Ragas-based evaluation tool and groundtruth records to guide systematic experiments.

confidence-thresholding (quality)

An ML classifier decides whether the system is likely to answer successfully before activating the RAG flow.

As we embarked on the journey of implementing a RAG"

— article, stated

latency-optimization (operations)

Doctolib reduced response latency from about 1 minute to under 5 seconds through code optimization, PTUs, smaller models, and streaming.

data-quality-loop (quality)

The FAQ embedding index is refreshed daily so the RAG system remains synchronized with current support knowledge.

privacy-preserving-processing (governance)

The team chose Azure OpenAI hosting for security and confidentiality in a healthcare context.

Tradeoffs

RAG before agentic architecture

Chose production RAG for FAQ-based customer care over jumping directly to multi-agent action-taking systems.

Rationale: RAG already transformed simple FAQ support, while agents were reserved for complex queries and action-oriented limitations.

Managed model hosting

Chose GPT-4o hosted on Azure OpenAI over unmanaged or less controlled model deployment.

Rationale: Healthcare support required security and confidentiality controls.

Classifier gate

Chose reduce reach to increase precision and impact over letting RAG attempt every user query.

Rationale: Routing unanswerable questions away from automation improved user experience even though fewer questions entered the automated flow.

"It diminished the reach of our tool but made it more precise and impactful.

As we embarked on the journey of implementing a RAG"

Latency engineering

Chose PTUs, code optimization, smaller-model tests, and streaming over accepting one-minute response times.

Rationale: Customer-care UX required a much faster response loop than the first working RAG prototype delivered.

Evaluation investment

Chose Ragas metrics plus annotated groundtruth over classic NLP metrics alone.

Rationale: RAG quality depends on retrieval relevance and answer grounding, which BLEU, ROUGE, and BERTScore do not fully capture.

Gotchas (what bit them)

Base LLMs could not answer private, current Doctolib questions accurately.

Root cause: They were trained on public data and lacked access to recent private company knowledge.

Resolution: Use RAG over daily updated FAQ articles.

Vector retrieval alone was not good enough.

Root cause: The initial retrieval step did not provide sufficient answer quality.

Resolution: Add a reranking brick and evaluate retrieval quality with Ragas metrics.

• Context recall which measure if all the relevant information"

— article, stated

RAG evaluation needed purpose-built metrics and groundtruth records.

Root cause: Classic NLP metrics did not verify semantic similarity or factual consistency with source documents.

Resolution: Deploy Ragas metrics and annotate perfect-answer groundtruth records.

Production latency was unacceptable at first.

Root cause: LLM-based RAG generation initially took about 1 minute per answer.

Resolution: Optimize code, use PTUs, test smaller models, and stream generation to reach under 5 seconds.

Trying to answer unanswerable queries hurt UX.

Root cause: The RAG system could not answer all customer-care questions, especially complex ones.

Resolution: Add an ML classifier to activate the system only when it predicts answerability.

RAG could not solve action-oriented or conversational support tasks.

Root cause: The system only produced descriptive answers and lacked seamless multi-turn conversation or back-office actions.

Resolution: Explore agentic frameworks as a second-stage architecture beyond the RAG ceiling.

Evaluation

Doctolib built a Ragas-based evaluation tool for component-wise and end-to-end RAG evaluation. The evaluation tracks context precision, context recall, faithfulness, and answer relevancy, and uses annotated groundtruth records as perfect-answer benchmarks. The team then ran experiments across hybrid search, prompt engineering, LLM parameters, chunking, embedding models, fine-tuning, and reranking.

Metrics: context precision · context recall · faithfulness · answer relevancy · 20% customer care case deflection reduction · latency reduced from 1 minute to under 5 seconds · answerability classifier precision/reach tradeoff

Outcomes

• Doctolib deployed a production RAG architecture for customer-care FAQ support.
• The system achieved a 20% reduction in customer-care case deflection.
• Latency improved from roughly 1 minute to less than 5 seconds.
• The team added reranking, evaluation tooling, groundtruth records, daily FAQ refresh, UX iteration, and an answerability classifier.
• The article explicitly identifies complex queries, non-descriptive actions, and multi-turn conversation as reasons to explore agents next.

This case should count as a strong production RAG system, not a completed agentic system. Its value for the pattern library is the disciplined path from vanilla RAG to production: retrieval freshness, reranking, RAG-specific evals, classifier gating, and latency work before escalating to agents.