Scaling an AI-Powered Conversational Shopping Assistant to 250 Million Users

rufus · E-commerce · 2025 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Conversational shopping assistant at Amazon scale

Amazon needed Rufus to answer shopping questions, personalize recommendations, use live retail data, and take shopping actions for hundreds of millions of customers while balancing accuracy, latency, cost, and global operational scale.

Constraints:

• Shopping questions require domain-specific accuracy, product context, real-time catalog data, and customer-specific order or preference context.
• Custom model iteration can take weeks or months, which slows feature expansion.
• Different query types have different reasoning, context-window, traffic, and latency requirements.
• Agentic features must integrate with durable traditional services for price alerts, auto-buy, order history, and catalog lookups.
• The assistant runs at Amazon scale, including peak shopping events and global marketplaces.

Architecture

Rufus started with a custom shopping-domain LLM served on AWS silicon, then shifted to a hybrid Bedrock architecture that combines the custom model with Amazon Nova and Claude Sonnet. Bedrock supplies model hosting, model-agnostic interfaces, global availability, evaluation, prompt caching, parallel tool calling, and tool-use support. Rufus routes query types across models, grounds answers with customer/catalog/web context, and integrates with persistent shopping services such as price alerts and auto-buy.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Rufus conversation interface"] --> B["Model selection layer"]
  B --> C["Amazon Bedrock"]
  B --> D["Custom shopping LLM"]
  C --> E["Amazon Nova and Claude Sonnet"]
  C --> F["Tool integration layer"]
  C --> H["Nova Web Grounding"]
  F --> G["Shopping action services"]
  G --> A

Components

Data flow

A shopping question enters Rufus, which selects an appropriate model through Bedrock or the custom stack, injects known context such as order history when available, lets the model call tools when it needs live product/catalog/account information, grounds with Nova Web Grounding when appropriate, and writes persistent shopping intents into services such as price alerts or auto-buy.

Why this architecture

Amazon kept the specialization benefits of its custom shopping model while using Bedrock to reduce infrastructure overhead, accelerate model adoption, and add tool-using agent capabilities across global shopping contexts.

Models used: custom in-house shopping LLM · Amazon Nova · Anthropic Claude Sonnet · foundation models on Amazon Bedrock

Scale

• More than 250 million customers used Rufus in the reported year.
  “More than 250 million customers have used Rufus this year.”article, stated
• Monthly Rufus users were up 140% year over year and interactions were up 210% year over year.
  “Monthly users are up 140% YoY and interactions are up 210% YoY.”article, stated
• Customers using Rufus during a shopping journey were 60% more likely to complete a purchase.
  “customers that use Rufus during a shopping journey are 60% more likely to complete a purchase.”article, stated
• Prime Day scale reached 80,000 AWS Inferentia and Trainium chips.
  “used 80,000 AWS Inferentia and Trainium chips”article, stated
• Auto-buy requests can remain active for six months and current users save an average of 20% per purchase.
  “Auto-buy requests remain active for six months, and customers currently using this feature are saving an average of 20% per purchase.”article, stated

Alternatives considered (roads not taken)

Use only off-the-shelf general-purpose models

Rejected because: They underperformed shopping evals or were too large, slow, and costly for Rufus's needs.

Continue only with the custom in-house model

Rejected because: Weeks- or months-long model iteration slowed advanced reasoning and feature expansion.

Model-only shopping actions

Rejected because: Persistent actions like auto-buy need traditional services to store and execute records safely.

Patterns

tool-use-orchestration (agent-architecture)

Rufus uses model tool calls to query live databases, order history, catalog data, price history, and shopping action services.

model-routing (model-ops)

Rufus selects among custom, Nova, Claude Sonnet, and other Bedrock models based on query complexity, latency, cost, and accuracy.

agent-memory (context)

Rufus uses account memory and shopping history to personalize future answers and actions.

rag-retrieval (retrieval)

Rufus grounds answers by retrieving live product, order, catalog, web, and account context when the model needs more information.

latency-optimization (optimization)

The team uses model choice, prompt caching, parallel tool calling, and Bedrock-managed serving to reduce response latency.

Tradeoffs

Model platform strategy

Chose hybrid custom LLM plus Bedrock-hosted foundation models over custom model only.

Rationale: The hybrid approach preserves shopping specialization while accelerating access to advanced reasoning, larger context windows, and managed global serving.

Context acquisition

Chose hybrid pre-supplied context plus tool retrieval over always asking the model to fetch more context.

Rationale: Supplying known context reduces inference calls and increases determinism, while tool calls remain available when more context is needed.

Shopping automation implementation

Chose agent creates persistent records, services execute them over model-only execution of purchase actions.

Rationale: Persistent service records give durable, auditable execution for price alerts and auto-buy.

Gotchas (what bit them)

Custom LLM iteration slowed feature expansion.

Root cause: Training cycles can take weeks or months while frontier models advance quickly.

Resolution: Adopt Bedrock for rapid model evaluation and integration while retaining the custom model where useful.

General-purpose models were not automatically good shopping assistants.

Root cause: They underperformed on shopping evals or imposed larger latency/cost footprints than needed.

Resolution: Use a custom shopping-domain LLM and route query types across specialized or managed models.

Agentic shopping actions need more than model output.

Root cause: Actions such as price alerts and auto-buy require persistent records, payment/shipping context, and traditional service execution.

Resolution: Let the agent create records while conventional services manage and execute them.

Evaluation

Rufus evaluates shopping assistants through domain-specific evals, LLM-as-a-judge answer accuracy, latency, engagement, conversation turns, and business outcomes.

Metrics: shopping eval performance · LLM-as-a-judge answer accuracy · latency · repeat customer engagement · conversation turns per interaction · purchase completion

Outcomes

• Rufus reached more than 250 million customers.
• Bedrock increased development velocity by more than 6x.
• Monthly users and interactions grew 140% and 210% year over year.
• Rufus users were 60% more likely to complete a purchase during shopping journeys.
• Agentic price alerts and auto-buy enable persistent shopping actions.

The system operates across Amazon shopping journeys, new marketplaces, global model hosting, and Prime Day-scale AWS silicon deployment.