AI-Powered Co-pilot System for Digital Sales Agents

wayfair · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce digital sales support

Wayfair wanted to help digital sales agents answer customer chats faster and more consistently while keeping humans responsible for the final message sent to customers.

Constraints:

Architecture

Wayfair Agent Co-pilot constructs context-rich prompts from policy, product, and conversation data, generates recommended chat replies, keeps human sales agents in control, and monitors quality with metrics, feedback, logs, and a QA LLM.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  digital-sales-chat-interface["Digital sales chat interface"]
  prompt-constructor["Prompt constructor"]
  llm-response-generator["LLM response generator"]
  recommendation-panel["Recommendation panel"]
  human-sales-agent["Human sales agent"]
  qa-llm["QA LLM"]
  quality-dashboard["Quality dashboard"]
  feedback-signals["Feedback signals"]
  digital-sales-chat-interface --> prompt-constructor
  prompt-constructor --> llm-response-generator
  llm-response-generator --> recommendation-panel
  recommendation-panel --> human-sales-agent
  human-sales-agent --> digital-sales-chat-interface
  human-sales-agent --> feedback-signals
  qa-llm --> quality-dashboard
  quality-dashboard --> feedback-signals
  feedback-signals --> prompt-constructor
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Digital sales chat interface Where human agents handle customer conversations. Prompt constructor, Human sales agent
Prompt constructor Assembles task description, guidelines, policies, product information, and conversation history. LLM response generator
LLM response generator Generates live recommended chat responses. Recommendation panel
Recommendation panel Shows suggested response text to the sales agent. Human sales agent
Human sales agent Accepts, edits, or rejects recommendations before customer delivery. Digital sales chat interface, Feedback signals
QA LLM Evaluates Co-pilot responses and supports quality monitoring. Quality dashboard
Quality dashboard Tracks instruction adherence, factual accuracy, handle time, conversion, adoption, and edit distance. Feedback signals
Feedback signals Drive prompt and retrieval improvements. Prompt constructor

Data flow

The prompt includes task description, guidelines, policies, product information, and conversation history; the LLM recommendation is then presented to the human sales agent for sending or editing.

“The prompt consists of several crucial components: * Task Description: This clearly defines what the LLM should do”article, stated
“This response is then presented to the sales agent where it might be sent to the customer as is, or modified to further refine its accuracy and suitability for the conversation.”article, stated

Why this architecture

Quality monitoring is central because factual product or policy errors would undermine agent trust and customer service.

“We evaluate the accuracy of the information provided by Co-pilot, checking for factual errors or hallucinations. This includes verifying the correctness of product information, policy details, and other relevant data.”article, stated
“Our team regularly reviews agent feedback, analyzes conversation logs, and even employs a second LLM – a "QA LLM" – to assess the quality of Co-pilot's responses.”article, stated

Models used: large language model · QA LLM

Scale

Alternatives considered (roads not taken)

Fully autonomous customer-facing sales AI

Rejected because: Wayfair keeps final message responsibility with sales agents while Co-pilot supplies live recommendations.

“While agents are chatting with customers, Co-pilot provides live, contextually relevant chat response recommendations.”article, stated

No independent quality monitor for generated responses

Rejected because: Wayfair uses a QA LLM plus logs and feedback to assess response quality.

“Our team regularly reviews agent feedback, analyzes conversation logs, and even employs a second LLM – a "QA LLM" – to assess the quality of Co-pilot's responses.”article, stated

Prompting without product, policy, and conversation context

Rejected because: The prompt explicitly assembles task, guideline, policy, product, and conversation-history sections.

“The prompt consists of several crucial components: * Task Description: This clearly defines what the LLM should do”article, stated

Patterns

human-in-the-loop-gate (safety)

The LLM drafts response recommendations, but the sales agent decides whether to send or modify them.

“This response is then presented to the sales agent where it might be sent to the customer as is, or modified to further refine its accuracy and suitability for the conversation.”article, stated

prompt-chaining (orchestration)

The prompt is explicitly assembled from task, guideline, policy, product, and conversation-history sections.

“The prompt consists of several crucial components: * Task Description: This clearly defines what the LLM should do”article, stated

llm-as-judge (evaluation)

A second LLM evaluates Co-pilot response quality alongside human feedback and conversation-log review.

“Our team regularly reviews agent feedback, analyzes conversation logs, and even employs a second LLM – a "QA LLM" – to assess the quality of Co-pilot's responses.”article, stated

eval-driven-development (evaluation)

Wayfair tracks instruction adherence, factuality, edit reasons, adoption, conversion, and handle time to refine the assistant.

“We track how closely Co-pilot’s response adheres to the specific instructions provided in its prompt.”article, stated

Tradeoffs

How much autonomy to give customer-facing sales AI.

Chose copilot recommendations reviewed by agents over fully autonomous customer chat responses.

Rationale: Human agents preserve judgment and quality control while still getting speed and context benefits from AI.

“While agents are chatting with customers, Co-pilot provides live, contextually relevant chat response recommendations.”article, implied

How to measure whether the assistant is useful.

Chose business and behavior metrics together over only model-quality scores.

Rationale: Wayfair tracks handle time, conversion, adoption, and edit distance because value depends on workflow behavior as well as output quality.

“In addition to AHT, we are also tracking order conversion rate, adoption rate (both contact-level and response-level), and the edit distance between our recommended responses and the text sent by the agent”article, stated

Gotchas (what bit them)

Response quality failures have multiple forms.

Root cause: A suggestion can violate prompt instructions, include factual errors, miss product information, or require style edits.

Resolution: Track quality dimensions separately, including prompt instruction adherence, factuality, edit reasons, and message purpose.

“This metric is crucial for understanding the specific areas where Co-pilot is falling short, allowing us to prioritize improvements.”article, implied

Factual product and policy errors would undermine agent trust.

Root cause: Generated responses may hallucinate or misstate product information, policy details, or other customer-facing facts.

Resolution: Evaluate factuality and compare recommendations with product and policy data before relying on broad rollout.

“We evaluate the accuracy of the information provided by Co-pilot, checking for factual errors or hallucinations. This includes verifying the correctness of product information, policy details, and other relevant data.”article, stated

Evaluation

Wayfair evaluates Co-pilot through AHT, conversion, adoption, edit distance, instruction adherence, factuality, edit reasons, message purpose, agent feedback, logs, and a QA LLM.

Metrics: 10% reduction in handle time in initial tests · average handle time · order conversion rate · contact-level and response-level adoption · edit distance between suggested and sent responses

“Initial test results of Co-pilot are very encouraging and show a 10% reduction in handle time, allowing our customers to be served more quickly.”article, stated
“By continually monitoring these metrics and making adjustments, we will ensure Co-pilot remains valuable for both customers and sales agents”article, stated

Outcomes

Company-authored engineering blog; results are initial and focused on AHT rather than long-term customer conversion.

“We're exploring ways to enhance Co-pilot's contextual understanding by leveraging RAG techniques.”article, stated