Generative AI Customer Service Agent Assist with RAG Implementation

newday · Finance · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: customer-service agent assist

NewDay needed to help contact-center agents answer customer questions faster and more accurately across nearly 200 knowledge articles, while proving that a generative AI assistant could add business value despite legacy infrastructure, limited resources, cost constraints, and strict response-time requirements.

Constraints:

• NewDay serves around 4 million customers and handles 2.5 million contact-center calls annually
• customer-service agents search across nearly 200 knowledge articles
• the team had limited resources, competing priorities, and executive buy-in challenges
• legacy systems were not conducive to rapid experimentation
• the original fully fledged voice-assistant goal was too ambitious for the initial constraints
• NewDay required a maximum response time of 5 seconds
• the production solution needed a single source of truth for future generative AI solutions

Architecture

NewDay built NewAssist as a RAG-based customer-service agent-assist application. A Streamlit UI on Fargate with Cognito and Microsoft Entra ID SSO lets agents ask questions and give feedback. A knowledge-base processing component retrieves articles from a third-party CMS, parses widget-structured JSON, chunks content, creates embeddings, and stores them in OpenSearch Serverless. A suggestion-generation component retrieves relevant chunks and sends them to Anthropic Claude 3 Haiku through Amazon Bedrock. Feedback is logged to Snowflake, CloudWatch logs service requests, weekly business reviews drive experiments, and new versions are evaluated offline against a golden dataset before production deployment.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer_service_agent["Customer service agent"] --> streamlit_assistant_ui["Streamlit assistant UI"]
  cognito_and_entra_sso["Cognito and Entra SSO"] --> streamlit_assistant_ui
  third_party_knowledge_base["Third-party knowledge base"] --> knowledge_base_processing["Knowledge base processing"]
  knowledge_base_processing --> opensearch_serverless_vector_database["OpenSearch Serverless vector database"]
  opensearch_serverless_vector_database --> suggestion_generation_component["Suggestion generation component"]
  streamlit_assistant_ui --> suggestion_generation_component
  suggestion_generation_component --> amazon_bedrock["Amazon Bedrock"]
  amazon_bedrock --> anthropic_claude_3_haiku["Anthropic Claude 3 Haiku"]
  anthropic_claude_3_haiku --> suggestion_generation_component
  suggestion_generation_component --> streamlit_assistant_ui
  streamlit_assistant_ui --> snowflake_feedback_log["Snowflake feedback log"]
  snowflake_feedback_log --> business_expert_review["Business expert review"]
  business_expert_review --> experiment_loop["Experiment loop"]
  offline_evaluation_dataset["Offline evaluation dataset"] --> experiment_loop
  experiment_loop --> knowledge_base_processing
  experiment_loop --> suggestion_generation_component

Components

Data flow

Agents ask questions in the NewAssist UI and can rate answers with thumbs up/down plus comments. Knowledge articles are pulled from a third-party content management system, parsed with custom widget-aware logic, chunked, embedded, and stored in OpenSearch Serverless. At query time, the suggestion-generation component retrieves relevant chunks and passes them to Claude 3 Haiku through Amazon Bedrock. Questions, answers, and feedback are logged into Snowflake for dashboards and weekly expert review; pre-production versions are checked against an evaluation dataset before deployment.

Why this architecture

NewDay narrowed the first release from a voice assistant to a chatbot POC so the team could validate RAG against the existing knowledge base and build a reusable foundation. Serverless AWS services reduced infrastructure management and kept monthly running cost low, Claude 3 Haiku met the 5-second response-time requirement at acceptable cost, and a feedback/evaluation loop let the team improve accuracy from early sub-60% results to over 90%.

Models used: Anthropic Claude 3 Haiku · Amazon Bedrock

Scale

• NewDay provides credit access to around 4 million customers.
  “NewDay provides around 4 million customers access to credit responsibly”article, stated
• NewDay's contact center handles 2.5 million calls annually.
  “NewDay’s contact center handles 2.5 million calls annually”article, stated
• Customer Services alone has nearly 200 knowledge articles.
  “with nearly 200 knowledge articles in Customer Services alone”article, stated
• The POC started with a three-person cross-functional team, a target of 80% accuracy, and a golden dataset of more than 100 question-answer pairs.
  “a small cross-functional team of three experts set out to build the proof of concept with a target of 80% accuracy.”article, stated

  “A golden dataset of over 100 questions and correct answers for these questions was created”article, stated
• The team ran more than eight experiment loops over 10 weeks.
  “In just 10 weeks and over 8 experiment loops, the team honed the solution.”article, stated
• NewDay expanded the pilot to 10 experienced agents and then began rolling out to over 150 agents.
  “approval to expand experimentation to 10 customer service agents after just 3 months.”article, stated

  “NewDay is currently rolling out the solution to over 150 agents”article, stated
• Answer retrieval time improved from about 90 seconds to 4 seconds.
  “Previously, it would take them on average 90 seconds to retrieve an answer; the solution now retrieves an answer in 4 seconds.”article, stated

Alternatives considered (roads not taken)

Start with a fully fledged voice assistant

Rejected because: The team judged the voice-assistant ambition too large for the early constraints and pivoted to a chatbot POC to validate the knowledge base foundation.

Use generic PDF parsing for knowledge articles

Rejected because: PyPDF parsing ignored the article widget structure and left accuracy around 60%, while custom widget-aware parsing lifted accuracy to 73%.

Use newer more performant models by default

Rejected because: NewDay kept Claude 3 Haiku because it was cost-effective, produced satisfying results, and met the 5-second response-time requirement.

Assume users ask questions like the technical team expects

Rejected because: Real agents used internal acronyms such as dd and cst, which initially caused accuracy to drop.

Patterns

rag-retrieval (agent-assist grounding)

NewAssist retrieves relevant knowledge-article chunks from OpenSearch Serverless and passes them to Claude 3 Haiku as context for customer-service suggestions.

data-quality-loop (knowledge-base processing)

NewDay improved accuracy by studying the structure of its knowledge articles, replacing generic PDF parsing with custom widget-aware parsing, and improving content gaps found in feedback.

eval-driven-development (iterative release gating)

NewDay used Build-Measure-Learn cycles, a golden evaluation dataset, pre-production thresholds, weekly review of bad feedback, and experiment loops to improve accuracy before rollout.

observability-tracing (feedback and monitoring)

NewAssist logs questions, answers, feedback, and service requests into Snowflake and CloudWatch so business experts and AI engineers can diagnose weak responses.

human-in-the-loop-gate (expert review and pilot rollout)

Experienced agents and weekly business experts review bad answers and guide experiments before broader deployment.

cost-aware-routing (model selection)

NewDay stayed with Claude 3 Haiku because it balanced acceptable answer quality, cost, and the 5-second latency requirement.

Tradeoffs

Pivot from voice assistant to chatbot POC.

Chose A focused chatbot proof of concept over A fully fledged voice assistant as the first implementation.

Rationale: The team needed to validate the knowledge-management foundation and prove business value before attempting the larger voice-assistant ambition.

Use serverless infrastructure.

Chose Bedrock, Fargate, Lambda, API Gateway, and OpenSearch Serverless over Managing bespoke infrastructure for experimentation.

Rationale: Serverless reduced cost, supported real-time demand scaling, and let the small team focus on experiment validation instead of infrastructure operations.

Invest in bespoke data parsing.

Chose Widget-aware JSON parsing and polished content formatting over Manual PDF export plus generic PyPDF parsing.

Rationale: The custom parser took longer but raised accuracy from 60% to 73% without changing the AI component.

Keep Claude 3 Haiku despite newer models.

Chose Claude 3 Haiku through Amazon Bedrock over Switching to newer higher-performing models.

Rationale: Claude 3 Haiku remained the most cost-effective Bedrock model that met NewDay's result quality and 5-second latency requirement.

Validate with experienced agents before broader rollout.

Chose A 10-agent pilot with negative-feedback review over Immediate broad rollout after reaching the 80% POC milestone.

Rationale: Experienced agents were needed to identify incorrect responses and reveal real usage patterns before productionization.

Gotchas (what bit them)

A voice assistant was too ambitious for the first delivery step.

Root cause: The team faced limited resources, competing priorities, legacy infrastructure, and the need to prove business value.

Resolution: NewDay pivoted to a chatbot proof of concept focused on validating the knowledge-management foundation.

Generic document parsing left accuracy too low.

Root cause: PyPDF parsing ignored the widget-specific structure of NewDay's knowledge articles in the third-party CMS.

Resolution: The team studied each widget schema and wrote bespoke parsing logic to extract and polish relevant content.

Real agents used abbreviations the model did not understand.

Root cause: The technical team did not initially account for internal shorthand such as dd for direct debit and cst for customer.

Resolution: NewDay statically injected acronyms and abbreviations into the LLM prompt, slowly recovering accuracy above 80%.

Pilot users exposed failure modes the lab dataset missed.

Root cause: Agents asked questions differently and revealed knowledge-base gaps only after the tool was used in real workflows.

Resolution: NewDay expanded to experienced agents, reviewed negative responses, fixed knowledge-base gaps, trained on internal language, and coached the pilot team.

The model choice had to satisfy cost and latency, not only raw capability.

Root cause: More performant models became available, but the use case required a maximum 5-second response time and acceptable running cost.

Resolution: NewDay continued using Claude 3 Haiku because it delivered satisfying results within the latency and cost constraints.

Evaluation

NewDay used a golden dataset of more than 100 question-answer pairs, rapid Build-Measure-Learn experiment loops, offline pre-production evaluation thresholds, production feedback logging, weekly expert review of bad answers, and pilot-agent testing to measure and improve accuracy.

Metrics: target 80% accuracy · early accuracy below 60% · accuracy from 60% to 73% after custom parsing · accuracy over 80% after experiment loops · accuracy drop to 70% when real agents used acronyms · accuracy over 90% after scaling improvements · answer retrieval time from 90 seconds to 4 seconds · running cost under $400 per month · maximum response time of 5 seconds

Outcomes

• NewAssist won the hackathon and evolved into a production-ready RAG agent-assist solution.
• Accuracy improved from early sub-60% results to over 90%.
• A custom parser raised accuracy from 60% to 73% without changing the AI component.
• NewDay recovered accuracy after real-agent acronym usage caused a drop to 70%.
• The rollout expanded from 10 experienced agents to over 150 agents, with plans to broaden to Customer Operations departments such as Fraud and Collections.
• Average answer retrieval time dropped from 90 seconds to 4 seconds.

The case moved from a hackathon POC to production rollout in a 2.5-million-call-per-year contact center. NewDay reports a 10-agent pilot, over 150 agents in rollout, and eventual expansion across Customer Operations.