Context Engineering for Production AI Assistants at Scale

spotify · E-commerce · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Commerce assistant context engineering at merchant scale

Shopify's Sidekick/Psychic assistant needed to serve millions of merchants while maintaining instruction following, latency, cost, safety, and continuity across long multi-tool conversations. More tokens, tools, and retained conversation history made the assistant slower, more expensive, and more likely to follow noisy or conflicting instructions.

Constraints:

• The assistant operates at the scale of millions of merchants.
• Tool messages can dominate token usage compared with user and system messages.
• Users may continue unrelated tasks in the same chat, so context trimming can damage perceived continuity.
• Memory must be useful for non-technical merchants without letting users write unsafe low-level rules directly.
• Safety and tool-use instructions must be close enough to the relevant data to affect the model.

Architecture

Shopify combines context pruning, a three-part memory system, and just-in-time instructions. Processed tool messages and old turns are removed when safe; explicit preferences, implicit user profiles, and episodic RAG memory provide continuity; and tool/user safety instructions are collocated with the data or message they govern.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  merchant-user["Merchant user"] --> psychic-assistant["Psychic assistant"]
  psychic-assistant --> tool-suite["Tool suite"]
  psychic-assistant --> context-pruning["Context pruning"]
  context-pruning --> memory-system["Memory system"]
  memory-system --> explicit-preferences["Explicit preferences"]
  memory-system --> implicit-user-profile["Implicit user profile"]
  memory-system --> episodic-memory-search["Episodic memory search"]
  episodic-memory-search --> just-in-time-instructions["Just-in-time instructions"]
  just-in-time-instructions --> psychic-assistant
  just-in-time-instructions --> tool-suite

Components

Data flow

A merchant request triggers the assistant's agentic loop and tool calls. After the model processes tool messages, bulky tool tokens can be removed. On new turns, explicit preferences and implicit profiles are injected into system context, episodic memory retrieves past conversations, and just-in-time instructions are packaged with tool messages or user-message wrappers.

Why this architecture

Shopify designed around context scarcity despite large model windows. Tool messages were 100x larger than user messages, old turns slowed the assistant and increased derailment risk, and collocating instructions with tool outputs improved attention while avoiding irrelevant global instructions.

Models used: Gemini · LLM system for profile synthesis · Psychic/Sidekick assistant model

Scale

• The assistant serves millions of merchants.
  “we serve millions of merchants.”transcript, stated
• The speaker says the assistant operates at the scale of millions.
  “real merchants and it operates at the scale of millions.”transcript, stated
• Tool messages can be 100x the token volume of user messages.
  “for each token in our user message we have 10x more tokens in the system message and we have 100x more tokens in the tool messages.”transcript, stated
• Turn 50 is used as an example where early-turn context is usually less relevant.
  “if you are in turn 50 like what happened in in the first turn is pretty much uh useless or yeah it's not really relevant for the current turn.”transcript, stated
• Just-in-time safety instructions reportedly improved internal instruction adherence by 5-10%.
  “with these safety instructions uh we have seen increase 5 to 10% erh in instruction adherence in some of our internal benchmarks”transcript, stated

Alternatives considered (roads not taken)

Rely on larger context windows

Rejected because: More tokens reduce instruction following, increase noisy/conflicting instructions, and raise cost and latency.

Keep all past tool messages and turns in context

Rejected because: Tool messages consume too much space and old turns make the system slow, expensive, and more likely to go off rails.

Let users directly type their own low-level preferences

Rejected because: Shopify's merchant users are not coding-tool users, so the LLM generates preferences to make them safer and better structured.

Graph RAG / graph databases for episodic memory

Rejected because: In production it added too much complexity for the answer quality benefit, so Shopify used hybrid search.

Patterns

context-compaction (context)

Processed tool messages and older conversation turns are removed from active context once they are no longer useful.

agent-memory (context)

The assistant uses explicit preferences, implicit user profiles, and episodic memory to restore continuity after context pruning.

rag-retrieval (retrieval)

Episodic memory uses RAG/hybrid search over past conversations and interactions to rerun or recall detailed prior work.

tool-use-orchestration (orchestration)

Tool instructions are packaged with tool outputs rather than kept permanently in the system message, reducing irrelevant instructions.

safety-policy-enforcement (governance)

User safety instructions are inserted just in time around user messages to prevent contradiction of core Sidekick/Psychic guidelines.

Tradeoffs

Context window management

Chose remove processed tool messages and old turns over retaining all chat and tool history.

Rationale: Tool messages dominate token usage and old turns slow the assistant down, but memory is needed to preserve continuity.

Preference capture

Chose LLM-generated explicit preferences over merchant-authored rules like coding-agent configuration files.

Rationale: Merchant users are not expected to write safe, effective rules directly, so the LLM generates preferences that are shown to users.

Episodic memory retrieval

Chose hybrid search over Graph RAG / graph databases.

Rationale: Graph approaches added too much production complexity for the benefit they brought.

Instruction placement

Chose just-in-time tool and safety instructions over permanent system-message instructions for every tool.

Rationale: Instructions near the relevant data improve attention and separation of concerns without occupying global context for unused tools.

Gotchas (what bit them)

Large context windows are not a scalable substitute for context engineering.

Root cause: More tokens lower instruction following and increase cost, latency, noisy instructions, and conflicting instructions.

Resolution: Actively manage context, remove processed tool messages, and use memory/JIT instructions to restore what matters.

Context pruning can create a bad user experience.

Root cause: Users expect earlier conversation details to remain available even after the system removes them from active context.

Resolution: Back pruning with explicit, implicit, and episodic memory.

Global tool instructions can distract the model or become irrelevant.

Root cause: Instructions in the system message may apply to tools the agent never calls and can interact oddly with past context.

Resolution: Collocate instructions with the tool output and add conditional instructions based on result type.

Evaluation

Shopify reports internal benchmark improvements from just-in-time safety instructions and monitors practical product metrics such as instruction adherence, jailbreak likelihood, latency, and token/cost pressure.

Metrics: 5-10% increase in instruction adherence · lower jailbreak likelihood · latency · token usage in user/system/tool messages

Outcomes

• The assistant can maintain merchant preferences and profiles across conversations without retaining all prior tokens.
• Just-in-time safety instructions improved instruction adherence in internal benchmarks by 5-10%.
• Hybrid episodic memory supports detailed recall of prior analytics-style interactions without using graph complexity.

The assistant is described as serving millions of Shopify merchants, with tool-message tokens roughly 100x user-message tokens.