RAG-Powered Virtual Assistant for Retail Store Operations

co-op · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Retail store operations knowledge search

Co-op store colleagues needed faster access to policy and procedure guidance than keyword search could provide across more than 1,000 web-based guides and tens of thousands of weekly queries.

Constraints:

• Store colleagues often need answers while under pressure in busy retail environments.
• The existing keyword search required precise terms and still left colleagues searching lengthy documents.
• The application handled roughly 50,000 to 60,000 weekly questions.
• The team needed model evaluation, safeguards, governance, and deployment infrastructure before broader store rollout.

Architecture

Co-op built the 'How Do I?' RAG virtual assistant on the Databricks Data Intelligence Platform. Databricks Lakeflow Jobs automate daily extraction and embedding of Contentful documents; Databricks AI Search stores and retrieves vector embeddings; Agent Bricks Custom Agents powers the RAG assistant; MLflow supports model swapping and experimentation; an evaluation module fires hundreds of test questions across configurations; Model Serving deploys the selected model; and GPT-3.5 was selected after testing DBRX, Mistral, and OpenAI models for performance, speed, cost, and security.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Contentful policy documents"] --> B["Databricks Lakeflow Jobs"]
  B --> C["Databricks AI Search"]
  C --> D["Agent Bricks Custom Agents"]
  D --> G["GPT-3.5"]
  D --> F["Evaluation module"]
  E["MLflow"] --> F
  E --> H["Model Serving"]
  F --> G
  D --> H
  H --> I["Store colleague assistant"]
  I --> J["Store colleagues"]

Components

Data flow

Policy and procedure documents are extracted from Contentful daily, embedded, and stored in Databricks AI Search. Store colleagues ask operational questions through the How Do I assistant, Agent Bricks retrieves relevant document context, GPT-3.5 generates an answer, and the response is evaluated and iterated through Databricks/MLflow tooling before broader trial rollout.

Why this architecture

Co-op chose a RAG assistant because the pain was document discovery over a large store-operations corpus, not generic text generation. Databricks already supported their data work, and the platform let them automate document ingestion, experiment across models, evaluate configurations, and deploy while keeping governance and security in view.

Models used: OpenAI GPT-3.5 · DBRX · Mistral · OpenAI GPT models · Databricks Assistant

Scale

• Co-op is the UK's fifth largest food retailer with over 2,500 stores.
  “the UK’s fifth largest food retailer with over 2,500 stores.”article, stated
• The assistant targets a library of over 1,000 policy and procedure guides.
  “a comprehensive library of over 1,000 web-based guides covering all store policies and procedures.”article, stated
• The existing application saw around 50,000 to 60,000 weekly queries.
  “more effectively manage around 50,000 to 60,000 weekly queries”article, stated
• The query load included about 23,000 initial and 35,000 follow-up questions weekly.
  “about 23,000 initial queries and 35,000 follow-up questions weekly”article, stated

Alternatives considered (roads not taken)

Continue with keyword search over policy documents

Rejected because: Keyword search required precise terms and still made employees dig through long documents.

Choose a model without comparative evaluation

Rejected because: The team evaluated DBRX, Mistral, and GPT models across accuracy, response time, safeguards, speed, cost, and security before selecting GPT-3.5.

Deploy immediately to all stores

Rejected because: The project remained in proof-of-concept with selected store trials planned to collect feedback before full rollout.

Patterns

rag-retrieval (retrieval)

The How Do I assistant retrieves policy and process document context from Databricks AI Search before answering.

eval-driven-development (quality)

Co-op built an evaluation module and selected GPT-3.5 after testing hundreds of questions across configurations.

model-routing (model-operations)

MLflow and model experimentation let the team compare DBRX, Mistral, and GPT options before choosing GPT-3.5.

safety-policy-enforcement (risk)

Model evaluation included built-in safeguarding features and security tradeoffs before selection.

Tradeoffs

Use semantic RAG over keyword search.

Chose RAG virtual assistant with AI Search over Traditional keyword search over policy documents.

Rationale: The existing search flow slowed employees and pushed work to support centers.

Select GPT-3.5 despite all models scoring well.

Chose OpenAI GPT-3.5 over DBRX, Mistral, and other GPT model options for this proof of concept.

Rationale: GPT-3.5 had the best balance across performance, speed, cost, and security.

Trial before full deployment.

Chose Proof-of-concept and selected-store trial over Immediate full-scale rollout.

Rationale: The team wanted user feedback and performance adjustments before full implementation.

Gotchas (what bit them)

Keyword search forced store colleagues to know exact terms.

Root cause: The existing search engine could not reliably surface specific answers from long policy documents.

Resolution: Use RAG and semantic retrieval over embedded Contentful documents.

A better search interface still needed rigorous model selection.

Root cause: Different models varied across accuracy, latency, cost, security, and safeguards.

Resolution: Fire hundreds of test questions across configurations and select GPT-3.5 for the best balance.

Governance and secure handling needed to mature with the proof of concept.

Root cause: The assistant handles internal store policy/process information and is transitioning toward broader governance controls.

Resolution: Use Databricks support and Unity Catalog transition to improve governance and access controls.

Evaluation

Co-op used a Databricks evaluation module that fired hundreds of test questions across configurations to compare model accuracy, response time, and safeguards, then selected GPT-3.5 based on balanced performance, speed, cost, and security.

Metrics: Hundreds of test questions · Accuracy · Response times · Built-in safeguarding features · Performance, speed, cost, and security balance · Internal user feedback

Outcomes

• Internal tests found the application intuitive and faster than the previous setup.
• The system is expected to reduce support center load and encourage employee self-service.
• Co-op planned selected-store trials before possible full-scale deployment.
• The project opened the door to additional GenAI projects in legal document processing and personalized offers.

The reported operating scale is over 2,500 stores, over 1,000 policy/procedure guides, about 50,000-60,000 weekly questions, and a breakdown of about 23,000 initial and 35,000 follow-up weekly questions.