Building and Optimizing a RAG-based Customer Service Chatbot

hdi · Insurance · 2022 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance

HDI needed a production RAG chatbot to help customer service agents answer insurance questions across complex internal document sources.

Constraints:

• retrieval needed to be fast and efficient
• chunking and indexing choices strongly affected recall and latency
• the team needed a practical MVP threshold instead of endless experiments
• feedback from actual users mattered after deployment

Architecture

HDI's customer-service chatbot is a modular RAG system for insurance agents, built around ingestion from SharePoints/databases, OpenSearch vector storage, specialized chunking, hybrid search, custom reciprocal-rank fusion, reranking, a chat interface, and feedback-driven experimentation. The team optimized chunking, embeddings, HNSW/vector indexing, and retrieval parameters through roughly 100-120 experiments before and after MVP.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-document-sources["enterprise-document-sources"]
  ingestion-and-chunking-pipeline["ingestion-and-chunking-pipeline"]
  opensearch-vector-store["opensearch-vector-store"]
  hybrid-search-retriever["hybrid-search-retriever"]
  reranking-layer["reranking-layer"]
  chat-answer-interface["chat-answer-interface"]
  feedback-loop["feedback-loop"]
  experiment-harness["experiment-harness"]
  enterprise-document-sources --> ingestion-and-chunking-pipeline
  ingestion-and-chunking-pipeline --> opensearch-vector-store
  ingestion-and-chunking-pipeline --> experiment-harness
  opensearch-vector-store --> hybrid-search-retriever
  hybrid-search-retriever --> reranking-layer
  hybrid-search-retriever --> chat-answer-interface
  reranking-layer --> chat-answer-interface
  chat-answer-interface --> feedback-loop
  feedback-loop --> experiment-harness
  experiment-harness --> ingestion-and-chunking-pipeline
  experiment-harness --> opensearch-vector-store
  experiment-harness --> hybrid-search-retriever

Components

Data flow

Insurance documents from SharePoints and databases are ingested into a knowledge base, chunked according to document/layout choices, embedded into a vector store, retrieved through OpenSearch plus custom hybrid search/reranking, and surfaced through a chat UI for customer agents. Positive/negative feedback and text comments feed back into experiments over chunking, indexing, embeddings, and retrieval settings.

Why this architecture

The architecture follows the operational need: insurance agents needed fast answers buried in long documents, but retrieval quality depended heavily on use-case-specific chunking, embedding choices, index settings, and hybrid retrieval. A modular RAG design let HDI swap model/endpoints, tune retrieval experimentally, and keep improving from production feedback rather than treating RAG as a one-shot build.

Scale

• The team ran roughly 100 experiments before moving to MVP.
  “We did round about 100 experiments before moving to MVP.”transcript, stated
• They mention about 30,000 possible combinations in the experiment setup.
  “you have around about 30,000 combination or maybe more than that but you need a lot of well at the start”transcript, stated
• The chunk corpus had not gone beyond about 200,000 chunks.
  “we personally uh have haven't gone beyond like 200,000 chunks.”transcript, stated
• The system reached around 6ms query latency and around 88% recall.
  “We do have a query latency of point. So it's around 6 milliseconds”transcript, stated

  “we optimize the index we are getting like around 88% recall rate”transcript, stated
• The chatbot was in production for over a year, had 800 current users, and was projected to reach 1,200 users by year end.
  “in production since over a year on one business line. We have already 350 people using that.”transcript, implied

  “we have around 800 users that are actually using that. And the future prospect is that we will expand that to around 1200 user by the end of this year.”transcript, stated

Alternatives considered (roads not taken)

Manual search across SharePoints and databases

Rejected because: Customer agents had to search multiple stores and scroll long documents to answer questions quickly.

Rigid one-provider RAG stack

Rejected because: The architecture was designed as modular Lego/Tetris blocks so teams could swap Bedrock, OpenAI, or SageMaker endpoints.

Vector-only retrieval

Rejected because: The team implemented custom hybrid search and reciprocal-rank-style fusion because hybrid support was not available or sufficient at the time.

Pick chunk sizes by intuition

Rejected because: The transcript repeatedly frames chunking and retrieval settings as use-case-specific empirical choices.

Patterns

rag-retrieval (retrieval)

The chatbot uses a RAG retrieval layer over insurance documents to support customer-service answers.

hybrid-search (retrieval)

HDI implemented custom hybrid search when native OpenSearch support was not sufficient at the time.

eval-driven-development (evaluation)

The team ran about 120 experiments and used recall to decide when to move into an MVP with test users.

Tradeoffs

Move to MVP after clearing an 85% recall target instead of continuing offline tuning indefinitely.

Chose test-user MVP with feedback over predicting all usage through more experiments.

Rationale: The transcript says the team moved above 85% recall to test users and actual feedback.

Gotchas (what bit them)

RAG tuning can become an endless search over chunking, algorithms, and parameters.

Root cause: There are many combinations of chunking strategies and indexing parameters.

Resolution: HDI prioritized what mattered for the use case and moved to MVP once recall cleared a target threshold.

Evaluation

The team measured recall, query latency, and user feedback, then continued incremental experiments after launch.

Metrics: 88% recall · 6ms query latency · positive/negative feedback

Outcomes

• The system is in production and had around 800 users at the time of the talk.

The deployment was expected to expand to around 1,200 users.