RAG-based Chatbot for Utility Operations and Customer Service

xcel_energy · Energy · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Utility operations document review and customer-service knowledge access

Xcel Energy needed to speed up rate-case review, legal contract review, and earnings-call analysis over large regulated document corpora while preserving governance for sensitive utility data.

Constraints:

• Rate cases require review of complex documentation such as energy price reports and government regulations.
• The document corpus includes both public reports and sensitive legal or proprietary materials.
• The team needed to iterate across models and embeddings without rebuilding core infrastructure.
• Utility compliance and access-control requirements made centralized data governance essential.

Architecture

Xcel Energy built a Databricks Mosaic AI RAG chatbot. Unity Catalog governs structured and unstructured documents, Databricks notebooks and Apache Spark ingest and prepare large datasets, Vector Search stores embeddings, Foundation Model APIs route prompts to LLMs, and MLflow tracks experiments. The team iterated across Llama 2, DBRX, Mixtral 8x7b, and Claude Sonnet 3.5 via AWS Bedrock while using the platform to answer questions over rate cases, contracts, earnings calls, energy price reports, and regulations.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  utility-document-corpus["Utility document corpus"] --> databricks-ingestion-pipeline["Databricks ingestion pipeline"]
  utility-document-corpus["Utility document corpus"] --> unity-catalog["Unity Catalog"]
  databricks-ingestion-pipeline["Databricks ingestion pipeline"] --> unity-catalog["Unity Catalog"]
  databricks-ingestion-pipeline["Databricks ingestion pipeline"] --> vector-search["Vector Search"]
  unity-catalog["Unity Catalog"] --> rag-chatbot["RAG chatbot"]
  vector-search["Vector Search"] --> rag-chatbot["RAG chatbot"]
  rag-chatbot["RAG chatbot"] --> foundation-model-apis["Foundation Model APIs"]
  foundation-model-apis["Foundation Model APIs"] --> model-candidates["Model candidates"]
  foundation-model-apis["Foundation Model APIs"] --> aws-bedrock-claude-sonnet["AWS Bedrock Claude Sonnet"]
  model-candidates["Model candidates"] --> mlflow-experiment-tracking["MLflow experiment tracking"]
  rag-chatbot["RAG chatbot"] --> business-reviewers["Business reviewers"]
  business-reviewers["Business reviewers"] --> mlflow-experiment-tracking["MLflow experiment tracking"]

Components

Data flow

Utility documents from government websites, legal contracts, invoices, earnings calls, and internal sources are ingested with Databricks notebooks and Spark. Unity Catalog manages the governed corpus and access controls. Prepared text is embedded and stored in Vector Search. A user question reaches the RAG chatbot, which retrieves relevant chunks, passes the context through Foundation Model APIs to a selected LLM, returns an answer with document grounding, and records experiments and model iterations in MLflow.

Why this architecture

The Databricks stack let Xcel combine governed data access, scalable ingestion, vector retrieval, model-provider flexibility, and experiment tracking in one platform. That mattered because the same chatbot needed to work across sensitive legal documents, regulatory materials, and large PDF-heavy review workflows, while the team was still experimenting with model and embedding choices.

Models used: Llama 2 · DBRX · Mixtral 8x7b · Claude Sonnet 3.5 · Databricks Foundation Model APIs · AWS Bedrock

Scale

• Xcel serves about 3.4 million electricity customers.
  “serving approximately 3.4 million electricity customers”summary, stated
• Xcel serves about 1.9 million natural gas customers.
  “1.9 million natural gas customers across eight U.S. states”summary, stated
• The utility operates across eight U.S. states.
  “eight U.S. states (Colorado, Michigan, Minnesota, New Mexico, North Dakota, South Dakota, Texas, and Wisconsin).”summary, stated
• Rate-case review time dropped from up to 6 months to 2 weeks.
  “take 2 weeks instead of up to 6 months.”article, stated
• The team tested at least four model families before settling on Claude Sonnet 3.5.
  “multiple LLMs including Llama 2, DBRX, and Mixtral 8x7b before switching to Claude Sonnet 3.5”summary, stated

Alternatives considered (roads not taken)

Manual document search and review

Rejected because: Rate cases and earnings-call analysis required searching large document sets and could take months.

Infrastructure built from scratch

Rejected because: Databricks integrated with internal data sources and dashboard tools so the team could focus on quality.

Stay on the initial open-source model experiments

Rejected because: The platform allowed model iteration and the team ultimately moved to Claude Sonnet 3.5 via AWS Bedrock.

Ungoverned document access

Rejected because: Sensitive legal contracts and proprietary materials required fine-grained controls.

Patterns

rag-retrieval (retrieval)

The chatbot retrieves relevant chunks from governed utility documents and feeds them to an LLM for context-aware answers.

privacy-preserving-processing (governance)

Unity Catalog provides access control for public and sensitive legal/proprietary materials in the same RAG corpus.

model-routing (model-selection)

The platform let Xcel test multiple LLMs and switch the answer-generation path to Claude Sonnet 3.5.

eval-driven-development (quality)

MLflow and model/embedding experiments gave the team a way to compare retrieval and generation quality.

Tradeoffs

Use an integrated Databricks stack

Chose Unity Catalog, Vector Search, Foundation Model APIs, and MLflow over Custom-built RAG infrastructure.

Rationale: The integrated stack reduced setup work and let the team focus on retrieval quality and workflow impact.

Use RAG over document corpora

Chose Context-aware chatbot answers over Manual document search across PDFs and reports.

Rationale: RAG reduced time spent finding relevant regulatory and business information.

Keep model access flexible

Chose Foundation Model APIs and AWS Bedrock integration over A single fixed model backend.

Rationale: The team could evaluate Llama 2, DBRX, Mixtral, and Claude Sonnet 3.5 without redesigning the RAG app.

Gotchas (what bit them)

RAG over utility documents requires governance before retrieval quality matters.

Root cause: The corpus mixes public reports with sensitive legal contracts and internal materials.

Resolution: Use Unity Catalog for centralized management and fine-grained access controls.

Manual review workflows scale poorly over PDF-heavy regulatory material.

Root cause: Rate cases and earnings reports involve complex documentation and hundreds of pages of PDFs.

Resolution: Use retrieval-augmented answers to surface relevant context from the corpus.

Initial model choices may not remain best as requirements sharpen.

Root cause: Different embeddings and language models perform differently for governed document QA.

Resolution: Keep model access and experiment tracking flexible enough to switch providers.

Evaluation

Xcel used MLflow and Databricks platform capabilities to iterate across embeddings and language models, with business impact measured by review-time reduction.

Metrics: rate-case review time · model performance · embedding performance · time to value · data-governance compliance

Outcomes

• Rate-case review time reportedly fell from up to 6 months to 2 weeks.
• The chatbot supports rate-case reviews, legal contract reviews, and earnings-call analysis.
• The platform let the team iterate across embeddings and language models, including a switch to Claude Sonnet 3.5 through AWS Bedrock.
• Governed RAG access reduced manual search across large utility document sets.

Concrete anchors include 3.4 million electricity customers, 1.9 million natural gas customers, eight U.S. states, four named model families, and review-time reduction from 6 months to 2 weeks.