Implementing RAG for Call Center Operations with Hybrid Data Sources

manulife · Finance · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: RAG-assisted customer service for financial services call centers

Manulife needed customer service representatives to answer customer queries across heterogeneous enterprise data: structured policy databases, unstructured policy documents, and CSR notes. The system had to work with legacy call-center data, reduce manual lookup across multiple systems, produce grounded answers without hallucinations, keep latency acceptable during live calls, and monitor both automated confidence and human feedback after deployment.

Constraints:

• the call center had decades of operational legacy and mixed data formats
• data came from structured databases, unstructured PDFs, and CSR notes
• text-to-SQL would add LLM calls, latency, cost, and risk of incorrect query translations
• structured data required substantial upfront denormalization and concept grouping
• customer-facing call-center workflows needed faster answers while customers were on the phone
• claims of no hallucinations still depended on monitoring and evaluation methodology

Architecture

Manulife built a production RAG system that treats structured and unstructured data as semantically retrievable records. Unstructured PDFs are chunked into 400-token overlapping segments, embedded with OpenAI text-embedding-ada-002, and stored in Azure AI Search. Structured policy data is ingested daily, denormalized and grouped by business concepts, reduced from 50 million rows to 4.5 million rows, converted into JSON strings, embedded with the same model, and stored in a separate Azure AI Search index. At inference time, the system retrieves top-k chunks from both indexes, combines them into a prompt, generates a grounded answer with GPT-3.5, and uses an independent validation model such as Llama 3 or GPT-4 to assess quality with a confidence rating. Monitoring captures validation confidence, CSR feedback, and response times.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  CSR["CSR"] --> CSR_user_interface["CSR user interface"]
  CSR_user_interface --> Inference_service["Inference service"]
  SharePoint_policy_documents["SharePoint policy documents"] --> Azure_Data_Lake_Storage["Azure Data Lake Storage"]
  Azure_Data_Lake_Storage --> Unstructured_ingestion_pipeline["Unstructured ingestion pipeline"]
  CSR_notes["CSR notes"] --> Unstructured_ingestion_pipeline
  Structured_policy_database["Structured policy database"] --> Azure_Synapse_Lake["Azure Synapse Lake"]
  Azure_Synapse_Lake --> Structured_data_processor["Structured data processor"]
  Unstructured_ingestion_pipeline --> Embedding_model["Embedding model"]
  Structured_data_processor --> Embedding_model
  Embedding_model --> Unstructured_Azure_AI_Search_index["Unstructured Azure AI Search index"]
  Embedding_model --> Structured_Azure_AI_Search_index["Structured Azure AI Search index"]
  Inference_service --> Retriever["Retriever"]
  Retriever --> Unstructured_Azure_AI_Search_index
  Retriever --> Structured_Azure_AI_Search_index
  Retriever --> Prompt_builder["Prompt builder"]
  Prompt_builder --> GPT_3_5_answer_generator["GPT-3.5 answer generator"]
  GPT_3_5_answer_generator --> Validation_model["Validation model"]
  Validation_model --> CSR_user_interface
  Validation_model --> Monitoring_service["Monitoring service"]
  CSR_user_interface --> Monitoring_service
  Monitoring_service --> Prompt_and_pipeline_revision_loop["Prompt and pipeline revision loop"]
  Prompt_and_pipeline_revision_loop --> Structured_data_processor
  Prompt_and_pipeline_revision_loop --> Unstructured_ingestion_pipeline

Components

Data flow

Policy PDFs flow from SharePoint into Azure Data Lake Storage when updated, then are chunked, embedded, and indexed. Structured customer policy data flows daily into Azure Synapse Lake, is denormalized and aggregated by business concepts, converted to JSON strings, embedded, and indexed separately. A CSR query retrieves top-k records from both structured and unstructured Azure AI Search indexes, combines the retrieved chunks into a prompt, sends the prompt to GPT-3.5 for answer generation, and then sends the answer to an independent validation model for quality/confidence assessment before the UI exposes it and monitoring records feedback and latency.

"The system consolidates data from three main sources:

• General insurance policy documents for US states (unstructured PDFs) • CSR notes documenting previous customer interactions"

— summary, stated "During inference, the system:

• Retrieves the top k relevant chunks from both the structured and unstructured indexes based on the input query"

Why this architecture

Manulife avoided text-to-SQL because translating user questions into SQL would add model calls, cost, latency, and translation errors. Instead, it invested in upfront data engineering that made structured policy data retrievable like text by grouping business concepts and embedding JSON rows. The dual-index approach lets the same inference flow retrieve policy documents, CSR notes, and customer policy data, while the independent validation model adds quality control before answers reach CSRs.

Models used: OpenAI text-embedding-ada-002 · GPT-3.5 · Llama 3 · GPT-4

Scale

• The production application had been running since May 2024.
  “The production application has been running since May 2024, providing concrete evidence of the system's real-world viability.”summary, stated
• Unstructured chunks use 400-token segments with overlap.
  “text is divided into 400-token segments with overlaps between chunks.”summary, stated
• The embedding representation is 1536-dimensional.
  “vectorized into 1536-dimensional vectors using OpenAI's text-embedding-ada-002 model.”summary, stated
• Structured processing reduced 50 million rows to 4.5 million rows.
  “Reduced from 50 million rows to 4.5 million rows through this processing”summary, stated
• Average response time improved from 21.91 seconds to 7.33 seconds.
  “The initial launch had an average response time of 21.91 seconds, which was optimized down to 7.33 seconds. This approximately 66% reduction in latency”summary, stated
• The latency improvement was approximately 66%.
  “This approximately 66% reduction in latency is substantial for a call center application where CSRs need quick answers while customers are on the phone.”summary, stated

Alternatives considered (roads not taken)

Text-to-SQL over structured policy data

Rejected because: It would add LLM calls, latency, cost, and incorrect query translation risk.

Treat only unstructured documents as retrievable text

Rejected because: The call-center task also needed customer policy database information, so structured data had to enter the same semantic retrieval path.

"The system consolidates data from three main sources:

• General insurance policy documents for US states (unstructured PDFs)"

Use one model for both generation and validation

Rejected because: The design used an independent validation model to add a quality-control layer before answers reached users.

Accept initial launch latency

Rejected because: Call-center agents need fast answers while customers are on the phone, so the team optimized from 21.91 seconds to 7.33 seconds.

Assume no-hallucination claims need no monitoring

Rejected because: The summary cautions that no-hallucination claims depend on monitoring and methodology, so confidence ratings and human feedback remain important.

Patterns

rag-retrieval (knowledge-architecture)

The call-center application retrieves top-k chunks from structured and unstructured indexes before GPT-3.5 generates an answer.

structured-output-contract (quality)

Structured database rows are converted to JSON strings with table headers as keys, letting structured records participate in semantic retrieval.

llm-as-judge (quality)

A separate validation model rates generated answers for quality and confidence after GPT-3.5 answers.

latency-optimization (operations)

The architecture improved average latency from 21.91 seconds to 7.33 seconds partly by avoiding text-to-SQL and extra inference calls.

eval-driven-development (quality)

Monitoring captures validation confidence, CSR feedback, and response times, and errors drive data-pipeline and prompt revisions.

"The production system includes monitoring capabilities that capture:

• Confidence ratings from the validation model • Human feedback from CSRs using the system"

cost-aware-routing (operations)

The model architecture uses GPT-3.5 for generation while reserving GPT-4 or Llama 3 for validation, reflecting cost-conscious model placement.

Tradeoffs

Structured data representation

Chose embedded JSON rows grouped by business concepts over text-to-SQL over normalized relational tables.

Rationale: Semantic JSON retrieval avoided extra LLM calls and query translation errors, but required significant upfront denormalization and grouping work.

Dual-model quality control

Chose GPT-3.5 for generation plus Llama 3 or GPT-4 for validation over single-model generation with no independent reviewer.

Rationale: The independent validation model adds quality control while using more capable or expensive models strategically.

Unified vector retrieval

Chose same embedding model for structured JSON rows and unstructured document chunks over separate retrieval paradigms for each data type.

Rationale: A common vector-search approach simplified inference across mixed data sources.

Production latency optimization

Chose architectural optimization to reach 7.33 seconds over accepting the 21.91-second launch latency.

Rationale: A call-center context rewards speed, so a 66% latency reduction mattered even if 7.33 seconds might still feel slow in some settings.

Monitoring posture

Chose automated confidence plus CSR human feedback over offline accuracy claims alone.

Rationale: The system needed ongoing visibility into quality, user feedback, and response time in production.

• Confidence ratings from the validation model • Human feedback from CSRs using the system • Response times"

— summary, stated

Gotchas (what bit them)

Structured enterprise data did not fit standard document RAG.

Root cause: Customer policy data lived in relational tables rather than natural-language documents.

Resolution: Denormalize and group tables by business concepts, serialize rows to JSON, embed them, and index them separately.

Text-to-SQL would increase cost, latency, and translation risk.

Root cause: Natural-language-to-SQL adds an intermediate LLM translation step before retrieval.

Resolution: Use semantic retrieval over embedded JSON strings for structured data.

Latency was still high for a call-center workflow.

Root cause: Complex retrieval and generation over multiple data sources initially took 21.91 seconds on average.

Resolution: Optimize architecture and structured-data retrieval to reach a 7.33-second average response time.

Grounded-answer claims still need careful interpretation.

Root cause: A claim of no hallucinations depends on how monitoring, validation, and evaluation define and detect hallucination.

Resolution: Track confidence, human feedback, response times, and known errors from missing data or ambiguous contexts.

The structured-data approach requires heavy upfront data engineering.

Root cause: Business concepts must be modeled and 50 million rows denormalized and aggregated into a smaller semantic corpus.

Resolution: Invest domain knowledge and engineering work before retrieval, accepting that this may not fit every organization.

Evaluation

Manulife uses a separate validation model to rate generated answers, monitors confidence ratings, records CSR feedback, and tracks response time. The case reports that production answers have remained grounded without hallucinations since May 2024, while also acknowledging occasional errors from missing data or ambiguous contexts and the need for careful interpretation of no-hallucination claims.

Metrics: average response time reduced from 21.91 seconds to 7.33 seconds · approximately 66% latency reduction · validation-model confidence ratings · CSR human feedback · response times · reported grounded answers without hallucinations since May 2024

• Confidence ratings from the validation model • Human feedback from CSRs using the system • Response times"

Outcomes

• Manulife deployed the RAG call-center application in production since May 2024.
• The architecture unified structured policy data and unstructured policy/CSR text through vector retrieval.
• The system reduced average response time from 21.91 seconds to 7.33 seconds.
• The summary reports consistently grounded answers without hallucinations since May 2024, while noting this claim should be interpreted carefully.
• The case contributes a reusable pattern for embedding structured enterprise rows as JSON records instead of generating SQL.

This is summary-only because the PDF source is not cached, but the ZenML summary is unusually specific. The strongest reusable lesson is not generic RAG; it is the structured-data move: denormalize business concepts, serialize rows as JSON, embed them alongside document chunks, and validate generated answers independently.