Building and Evaluating Sidekick: A Production Agent for E-commerce Merchants

shopify · E-commerce · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce merchant assistant

Shopify needed Sidekick to help merchants operate stores inside Shopify admin, while scaling tool use, preserving a consistent merchant-facing voice, and evaluating safety and goal fulfillment for a production-facing agent.

Constraints:

Tool-specific instructions can collide when packed into one system prompt.
Merchant-facing responses need consistent tone and grounding.
Agents can perform analysis and draft actions, but merchants must sign off on changes.
LLM judges and reinforcement learning can be exploited if evaluation is weak.

Architecture

Sidekick lives inside Shopify admin as the merchant's single point of contact. Shopify moved tool-specific instructions into structured tool results through just-in-time instructions, explored specialized subagents hidden behind Sidekick, and built a merchant simulator plus LLM judges to evaluate safety, grounding, goal fulfillment, and merchant sentiment.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  merchant[Merchant] --> sidekick[Sidekick]
  sidekick --> tool-results[Tool Results]
  sidekick --> subagents[Subagents]
  subagents --> sidekick[Sidekick]
  sidekick --> merchant-simulator[Merchant Simulator]
  merchant-simulator --> llm-judges[LLM Judges]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Merchant`	Shopify user asking questions and approving changes.	`Sidekick`
`Sidekick`	Main merchant-facing agent and only direct point of contact.	`Tool Results`, `Subagents`, `Merchant Simulator`
`Tool Results`	Structured tool responses that carry just-in-time instructions.	`Sidekick`
`Subagents`	Specialized internal agents with domain-specific tools and prompts.	`Sidekick`
`Merchant Simulator`	Simulates merchant conversations for evaluation.	`Sidekick`, `LLM Judges`
`LLM Judges`	Score conversations against safety, goal fulfillment, grounding, and sentiment criteria.	`Merchant Simulator`

Data flow

A merchant asks Sidekick for help, Sidekick selects tools whose structured results include just-in-time instructions, delegates specialized work to subagents when useful, forms a merchant-facing response, and is evaluated by simulated merchant conversations and LLM judges.

“Sidekick is an assistant that lives within the Shopify admin, and merchants use it to get general help with their store”transcript, stated

“it's really They're just moving those instructions from the system prompt into the structured tool results. And this is a simplified version of our help tool.”transcript, stated

Why this architecture

Sidekick keeps the merchant-facing voice centralized while moving tool-specific instructions out of the global prompt to avoid misuse. Subagents can specialize internally, but only Sidekick talks to the merchant. Evaluation uses simulators and judges because vibe testing was insufficient for production confidence.

“Sidekick is the only point of contact, and the The merchant can't really speak to the subagents directly. This ensures that the tone and voice of Sidekick is consistent, and you're still delegating to a subagent to handle some of those”transcript, stated

“There was a lot of vibe testing of Sidekick, I would say. We tried it out, it looked good, so we shipped it, or it vibe tested well is a phrase that I've heard quite a bit.”transcript, stated

Models used: LLMs · LLM judges · Merchant simulator · Reinforcement learning

Scale

Shopify reported Cohen's kappa around 0.69 for Sidekick judge agreement.
“calculated Cohen's Cap and it was like 0. 69, which is actually about what it is on Sidekick with the agreement between some of our PMs. And now we're going to make a prompt to try to match what the GTX is doing.”transcript, stated
Sidekick has few defined workflows at the time of the talk.
“Sidekick today doesn't actually have that many defined workflows, and I know that many agents are built with very defined workflows in mind to keep consistency.”transcript, stated

Alternatives considered (roads not taken)

Put all tool instructions into the main system prompt.

Rejected because: Tool-specific citation instructions leaked and were misused by other tools.

“when we used to have that format and those instructions, the system prompt, Sidekick would actually misuse that citations format”transcript, stated

Let merchants talk directly to specialized subagents.

Rejected because: A single Sidekick point of contact preserves consistent tone and voice.

“agent, the main agent that the merchant is talking to is still Sidekick. Sidekick is the only point of contact, and the The merchant can't really speak to the subagents directly.”transcript, stated

Trust vibe testing for agent quality.

Rejected because: The team moved to LLM judges and simulators for higher confidence.

“We use LLMs, LLM as a judges and simulators in order to really have a lot of high trust in our systems, and it really brought Sidekick, I think, to the next level.”transcript, stated

Patterns

`tool-use-orchestration` (agent tooling)

Sidekick moves tool-specific guidance into structured tool results that the agent uses just in time.

“Just-in-time instructions removes the complexity of having all of the conditionals within your main system”transcript, stated

`multi-agent-orchestration` (subagents)

Specialized subagents run internal loops with domain tools while Sidekick remains the merchant-facing agent.

“Subagents are specialized agents”transcript, stated

`llm-as-judge` (evaluation)

LLM judges evaluate Sidekick conversations across safety, goal fulfillment, grounding, and merchant sentiment.

“We use LLMs, LLM as a judges and simulators in order to really have a lot of high trust in our systems”transcript, stated

`human-in-the-loop-gate` (merchant control)

The merchant must approve changes before they are committed.

“merchant has to actually sign off and commit the changes”transcript, stated

`safety-policy-enforcement` (agent safety)

Safety is an explicit evaluation criterion for Sidekick runs.

“a criteria, for Sidekick, it's safety, goal fulfillment, grounding, merchant sentiments”transcript, stated

Tradeoffs

Instruction placement

Chose Just-in-time instructions in tool results over A giant dynamic system prompt.

Rationale: This reduces instruction collisions and makes the static prompt more cache friendly.

“Just-in-time instructions removes the complexity of having all of the conditionals within your main system, your agent's main system prompt”transcript, stated

Subagent UX

Chose Sidekick as sole merchant-facing agent over Direct merchant access to subagents.

Rationale: A single front agent keeps tone consistent while still delegating specialized internal work.

“Sidekick is the only point of contact, and the The merchant can't really speak to the subagents directly.”transcript, stated

Optimization method

Chose Judges and simulators with caution over Unmonitored RL optimization.

Rationale: RL can exploit judge weaknesses, so evaluation needs scrutiny.

“you can still have faults in your judge, and RL will find a way to exploit them.”transcript, stated

Gotchas (what bit them)

Tool instructions can contaminate unrelated tool behavior.

Root cause: Putting tool-specific citation format instructions in the main prompt caused Sidekick to reuse them for other tools.

Resolution: Move instructions into structured tool results through just-in-time instructions.

“Sidekick would actually misuse that citations format for other tools that weren't help related at all.”transcript, stated

Evaluation judges are exploitable.

Root cause: Reinforcement learning can optimize against flaws in the judge rather than true merchant utility.

Resolution: Treat judges as imperfect and monitor for exploitation during RL.

“Even if you build a I trust Judge, RL is going to learn how to exploit you.”transcript, stated

Subagents can fragment the user experience.

Root cause: Direct subagent access could produce inconsistent tone and voice.

Resolution: Keep Sidekick as the only merchant-facing point of contact.

“This ensures that the tone and voice of Sidekick is consistent”transcript, stated

Evaluation

Shopify evaluates Sidekick with a merchant simulator and LLM judges across safety, goal fulfillment, grounding, merchant sentiment, and overall score, while measuring judge agreement.

Metrics: Safety · Goal fulfillment · Grounding · Merchant sentiment · Cohen's kappa

“it's merchant-facing, so we call it our merchant simulator.”transcript, stated