Enterprise RAG-Based Virtual Assistant with LLM Evaluation Pipeline

santalucía_seguros · Insurance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Insurance agent virtual assistant

Santalucía Seguros needed insurance agents to answer customer questions quickly from documentation scattered across locations and formats while maintaining response quality as documents, prompts, and RAG code changed.

Constraints:

Agents needed product, coverage, and procedure answers in real time.
Documentation came from multiple locations and formats.
The assistant had to be available through Microsoft Teams on mobile, tablet, and computer.
New document ingestion could not degrade previously released answer quality.
The architecture needed privacy, security, scalability, and model access control.

Architecture

Santalucía built a RAG virtual assistant on its Advanced Analytics Platform powered by Databricks and Microsoft Azure. Documentation is continuously ingested into vector stores; the RAG system runs as a pyfunc model in MLflow; LLMs are hosted through Databricks Model Serving endpoints with credentials in Azure Key Vault; and a CI/CD LLM-as-judge process scores answer quality against a professional-validated ground-truth set before releases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  microsoft-teams-interface["Microsoft Teams interface"] --> rag-virtual-assistant["RAG virtual assistant"]
  rag-virtual-assistant --> embedding-vector-stores["Embedding vector stores"]
  rag-virtual-assistant --> databricks-model-serving["Databricks Model Serving"]
  mlflow-pyfunc-model["MLflow pyfunc model"] --> databricks-model-serving["Databricks Model Serving"]
  databricks-model-serving --> azure-key-vault["Azure Key Vault"]
  databricks-model-serving --> ci-cd-evaluation-pipeline["CI/CD evaluation pipeline"]
  ci-cd-evaluation-pipeline --> ground-truth-question-set["Ground truth question set"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Microsoft Teams interface`	Agent-facing channel for natural-language questions on any device.	`RAG virtual assistant`
`RAG virtual assistant`	Answers questions about products, coverages, procedures, and documentation.	`Embedding vector stores`, `Databricks Model Serving`
`Embedding vector stores`	Indexes up-to-date documentation for rapid retrieval.	`RAG virtual assistant`
`MLflow pyfunc model`	Packages the RAG system as an LLMOps-managed model.	`Databricks Model Serving`
`Databricks Model Serving`	Hosts external and self-hosted LLM endpoints behind unified APIs.	`Azure Key Vault`, `CI/CD evaluation pipeline`
`Azure Key Vault`	Stores credentials for model endpoints securely.	`Databricks Model Serving`
`CI/CD evaluation pipeline`	Runs LLM-as-judge evaluations before release.	`Ground truth question set`
`Ground truth question set`	Professionally validated questions and expected answers used for release evaluation.	`CI/CD evaluation pipeline`

Data flow

Insurance documentation is ingested into embedding-based vector stores; agents ask natural-language questions in Microsoft Teams; Databricks-hosted LLM endpoints answer using the RAG model; CI/CD evaluates each release by having the VA answer ground-truth questions and an LLM judge compare answers for accuracy, relevance, and coherence.

“continuous ingestion of up-to-date documentation into embedding-based vector stores”article, stated

“The VA answers each question using the ground truth, and the judge assigns a score by comparing the expected answer with the one provided by the VA.”article, stated

Why this architecture

Santalucía chose Databricks/Azure RAG with model serving and LLM-as-judge release gates because insurance answers need speed, privacy, security, scalability, and quality guarantees as documents and prompts evolve.

“offering flexibility, privacy, security, and scalability.”article, stated

“we cannot afford to modify any part of the solution’s code without guaranteeing that it will not negatively impact the quality”article, stated

Models used: Databricks · Microsoft Azure · MLflow · Databricks Model Serving · Azure OpenAI API · GPT-4 · LLM-as-a-judge

Scale

Santalucía has supported families for over 100 years.
“a Spanish company that has supported families for over 100 years”article, stated
The assistant is available 24/7 on mobile devices, tablets, or computers.
“on any mobile device, tablet, or computer, in real-time, with 24/7 availability.”article, stated
Agents can get answers in seconds when customers ask coverage questions.
“whenever a customer asks about coverage they can get an answer in seconds.”article, stated

Alternatives considered (roads not taken)

Let users discover quality problems after release.

Rejected because: Santalucía needed the system itself to assess response quality before scaling to production.

“we cannot wait for users to evaluate the quality of the responses; instead, the system itself must be able to assess the quality before scaling to production.”article, stated

Call third-party LLM APIs directly from each application.

Rejected because: Databricks Model Serving provides a unified abstraction for credentials, permissions, and endpoints.

“provides an abstract layer that makes it easy for development teams to add new models, eliminating the need for third-party API integrations.”article, stated

Use static documentation without continuous ingestion.

Rejected because: The architecture continuously ingests up-to-date documentation into vector stores.

“This architecture enables the continuous ingestion of up-to-date documentation into embedding-based vector stores”article, stated

Patterns

`rag-retrieval` (retrieval)

The virtual assistant retrieves up-to-date documentation from embedding-based vector stores.

“Retrieval Augmented Generation (RAG) framework running on Santalucía’s Advanced Analytics Platform”article, stated

`llm-as-judge` (evaluation)

A high-capacity LLM judges VA answers against a ground-truth question set in CI/CD.

“Our proposed solution uses a high-capacity LLM as a judge within the CI/CD pipeline.”article, stated

`eval-driven-development` (evaluation)

Each release is evaluated before production to prevent regressions from new documents or prompt changes.

“Each time we ingest new documents into the VA, we must review the assistant's performance before releasing the updated version.”article, stated

`model-routing` (model-ops)

Databricks Model Serving abstracts external and self-hosted LLM endpoints behind one access layer.

“makes it easy to integrate external LLMs, such as GPT-4 or other models available in the [Databricks Marketplace]”article, stated

Tradeoffs

Expose the VA through Microsoft Teams.

Chose An existing collaboration tool with mobile/tablet/computer access. over A standalone assistant interface..

Rationale: Teams access makes the tool available in agents' daily workflow in real time.

“The VA is accessed within Microsoft Teams and is able to answer agent questions in natural language”article, stated

Use LLM-as-judge before production.

Chose Automated judge scoring against professional ground truth. over Waiting for user feedback after production release..

Rationale: The business cannot afford poor retrieval/generation quality in customer-service workflows.

“we don't need to wait for the user to inform us that the VA is malfunctioning”article, stated

Use Model Serving as an abstraction layer.

Chose Unified Databricks endpoints and git/CI/CD endpoint configuration. over Direct third-party API integrations per team..

Rationale: The abstraction centralizes credentials, permissions, token consumption, and security access.

“deploy new models on our platform through a git repository, where we define a configuration JSON”article, stated

Gotchas (what bit them)

New documents can degrade previously delivered RAG quality.

Root cause: Continuous ingestion changes retrieval context and answer behavior.

Resolution: Run ground-truth LLM-as-judge evaluations in CI/CD before releasing updates.

“seamless integration of newly ingested documents into the RAG system”article, stated

Prompt tweaks can accidentally harm existing answers.

Root cause: Small code or prompt changes alter response generation.

Resolution: Use a ground-truth regression set and LLM judge to detect quality regressions.

“detect whether any changes made to a prompt to enhance the user experience are impacting the quality of responses”article, stated

Direct model integrations create credential and security sprawl.

Root cause: Third-party and self-hosted models need credentials, endpoints, permissions, and token controls.

Resolution: Use Databricks Model Serving with Azure Key Vault and CI/CD deployment.

“management of token consumption, credentials, and security access.”article, stated

Evaluation

Santalucía evaluates each VA release with a professionally validated ground-truth set, LLM-as-judge natural-language criteria, and CI/CD scoring for accuracy, relevance, and coherence.

Metrics: Accuracy · Relevance · Coherence · Ground-truth Q/A coverage · Response quality before production · Response speed in seconds

“natural-language-based criteria for measuring accuracy, relevance, and coherence between expected answers and those provided by the VA.”article, stated

“with each release, the set of questions/answers in the ground truth increases model robustness.”article, stated

Outcomes

The VA gives insurance agents real-time answers in Microsoft Teams across devices.
The assistant reportedly exceeded user expectations.
The architecture supports continuous releases while protecting response quality.
Immediate answers are reported to improve customer satisfaction and accelerate sales.

Santalucía has operated for over 100 years; the VA is available 24/7 across devices for insurance-agent workflows.

“the Virtual Assistant has exceeded the expectations of users while demonstrating best practices for LLM deployment.”article, stated

“The speed of the response not only positively impacts customer satisfaction, it accelerates the sale of products”article, stated