Building Production AI Agents for E-commerce and Food Delivery at Scale

prosus · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce and food-delivery AI agents

Prosus needed production agents for shopping, food ordering, and internal productivity across a large global portfolio, but pure prompt/model work was not enough: success depended on context engineering, messy real-time enterprise data, search quality, user memory, UI adoption, and evaluation design.

Constraints:

• Food and shopping agents must answer broad, ambiguous user intents such as romantic dinner, hiking trip, or furnish my house.
• Enterprise context is scattered across real-time databases covering promotions, payment methods, inventory, merchant hours, catalogs, and user history.
• Keyword search fails on semantic or ambiguous conversational requests, while semantic search alone still cannot resolve subjective intent.
• Pure chatbot interfaces underperformed familiar commerce UIs in A/B tests despite working technically.
• Buying food or products is visual and action-oriented, so text-only conversation cannot carry the whole experience.
• Evaluation has to begin before launch and remain understandable to business stakeholders.

Architecture

The production approach treats the agent as a layered commerce experience rather than a standalone chatbot. A context-engineering layer injects system prompt, user message, enterprise data, and user history into the agent. Existing app behavior provides cold-start memory. A multi-stage search pipeline performs LLM-based query understanding/personalization/expansion, hybrid keyword and semantic search, and LLM-based reranking over candidate items. The UI combines chat with generated visual components and contextual micro-interventions in the familiar product journey. Evaluation starts offline with synthetic and manual datasets, uses deterministic funnel metrics before LLM-as-judge, and is calibrated through labeling parties and custom annotation tools that show conversations alongside the items users saw.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  portfolio-agent-products["Portfolio agent products"] --> context-assembly-layer["Context assembly layer"]
  portfolio-agent-products --> generative-ui-layer["Generative UI layer"]
  portfolio-agent-products --> contextual-intervention-layer["Contextual intervention layer"]
  context-assembly-layer --> enterprise-data-pipelines["Enterprise data pipelines"]
  context-assembly-layer --> user-history-memory["User history memory"]
  context-assembly-layer --> query-understanding-layer["Query understanding layer"]
  query-understanding-layer --> hybrid-search-layer["Hybrid search layer"]
  hybrid-search-layer --> llm-reranker["LLM reranker"]
  llm-reranker --> generative-ui-layer
  generative-ui-layer --> contextual-intervention-layer
  offline-eval-dataset["Offline eval dataset"] --> deterministic-funnel-metrics["Deterministic funnel metrics"]
  offline-eval-dataset --> llm-judge["LLM judge"]
  deterministic-funnel-metrics --> llm-judge
  llm-judge --> labeling-party-workflow["Labeling party workflow"]
  labeling-party-workflow --> annotation-tool["Annotation tool"]

Components

Data flow

A user asks for food, shopping, or productivity help. The system combines the user message with product-specific context: real-time promotions, opening hours, payment options, catalog data, and user history. For commerce search, an LLM rewrites or expands broad intent into concrete queries, the retrieval layer runs keyword and semantic search, and an LLM reranker uses the original query plus user context to pick a small set of results. The experience then renders generated UI components or a contextual intervention rather than only text. Evaluation data flows from team simulations, synthetic query generation, deterministic funnel outcomes, LLM-as-judge, human labeling parties, and custom annotation tools.

Why this architecture

The architecture prioritizes context, search, UI, and evaluation because Prosus found that model choice and prompts were not the limiting factors in production commerce agents. Messy enterprise data and user history determine answer usefulness, hybrid and LLM-augmented search determine whether users can find the right items, generative/contextual UI determines adoption, and practical evaluations determine whether the product should launch and keep improving.

Models used: State-of-the-art LLMs referred to as Model A and Model B · LLM for query understanding, personalization, and expansion · Embedding model for semantic search · LLM for reranking · LLM-as-a-judge · ChatGPT · v0/Vercel for quick annotation tooling

Scale

• The speaker says the agents are used across industries by two billion customers globally.
  “these agents are kind of used by two billion customers around the world.”transcript, stated
• Prosus's internal Tokan tool is used by 15,000 internal users.
  “this tool is used by 15,000 users internally at process”transcript, stated
• Tokan had been under development for three years.
  “So we started building this tool 3 years back.”transcript, stated
• Cold-start user-history context can provide roughly a three-month runway while conversational memory accumulates.
  “that already does wonder that will give you a runtime of 3 months”transcript, stated
• The search pipeline may rerank thousands of candidates down to a handful of options.
  “these are the 10,00 responses from the first two steps this is the context about the user rerank you know what is and then you get these three or these 10 options”transcript, stated
• The UI design example assumes an agent might have 20 tools behind a blank screen.
  “behind the scene there's an agent which can do a lot of things. Maybe there are 20 tools connected to that.”transcript, stated
• The generative UI example describes designing 10 UI components the agent can choose among.
  “with the design team I build 10 UI components and then based on the user request sometimes you get similar dishes”transcript, stated
• A contextual intervention example triggers after five minutes of browsing headphones.
  “maybe you spend 5 minutes looking for things and you know looking for items it pops up at the right time contextual”transcript, stated
• The suggested offline eval dataset size is 20, 50, or 100 examples before launch.
  “you create a data set of 20 50 100 uh and then that's a data set and then you see how your agent responds”transcript, stated
• A labeling party example uses 15 people for one and a half hours to produce about 150 labels.
  “You get 15 people in the room. You order some pizza.”transcript, stated

  “you spend one and a half hours and at the end of one and a half hour you'll have I don't know every person will label 10 data points you'll have 150 data points”transcript, stated

Alternatives considered (roads not taken)

Optimize primarily around prompt engineering or model choice.

Rejected because: The speaker says the difference in production comes from context engineering more than comparing state-of-the-art models without the right context.

Launch memory with a complex long-term/short-term/episodic architecture on day one.

Rejected because: The recommended cold-start is to reuse existing app data first, then collect dynamic memory as users converse.

Use keyword search alone.

Rejected because: Keyword search misses implicit matches such as vegetarian margherita pizza and cannot handle broad conversational queries.

Use semantic search alone.

Rejected because: Semantic search helps with implicit similarity but still cannot resolve subjective ambiguous requests like romantic dinner.

Replace the commerce UI with a full chatbot.

Rejected because: A/B testing showed the chatbot experience performed terribly despite working technically, because users did not adopt the new interface.

Jump directly to complex tool-level LLM-as-judge evals.

Rejected because: The speaker recommends deterministic funnel metrics and simple business questions first, because complex evals create analysis paralysis and confuse stakeholders.

Patterns

rag-retrieval (retrieval)

The agent retrieves real-time enterprise data such as promotions, merchant hours, payment methods, and catalogs into the prompt so answers are grounded in the current commerce context.

agent-memory (context_management)

The agent uses existing product data such as previous food orders and browsed items as cold-start memory before dynamic conversational memory matures.

hybrid-search (retrieval)

The commerce agent uses keyword search when taxonomy matches are enough and semantic search when users phrase requests conversationally or implicitly.

query-rewriting (retrieval)

An LLM turns broad requests such as romantic dinner into personalized concrete subqueries before retrieval.

retrieval-ranking (retrieval)

After initial keyword/semantic retrieval, an LLM reranks candidate products or restaurants using the user query and context.

synthetic-data-generation (evaluation)

Before launch, teams generate synthetic queries from seed examples and personas to build an initial evaluation dataset instead of waiting for real user data.

eval-driven-development (evaluation)

The team treats evals as a product moat, starts with deterministic business metrics, uses LLM-as-judge after simpler signals, and calibrates the judge with human labels.

observability-tracing (evaluation)

The evaluation workflow uses tracing/observability tools such as Langfuse, then wraps them with custom annotation tooling so business users can inspect conversations and visual items.

Tradeoffs

Use existing behavioral data for memory cold start.

Chose Existing orders, browses, and app context over Complex memory architecture from day one.

Rationale: The simpler approach creates immediate personalization and lets teams focus on product-market fit while dynamic memory develops.

Layer LLM query understanding and reranking around search.

Chose Hybrid search plus LLM expansion and reranking over Keyword-only or semantic-only search.

Rationale: Keyword and semantic search solve different classes of queries; LLM layers help translate broad subjective intent into retrievable and rankable candidates.

Mix chat with generated UI components.

Chose Generative UI over pure chatbot or static UI alone over Replacing commerce flows with a text-only assistant.

Rationale: The mixed interface preserves visual shopping behavior while allowing the agent to decide which dynamic components to show.

Prefer contextual micro-interventions over universal chat for many tasks.

Chose Narrow, timely popups or buttons inside the existing UI over A chatbot that claims to do everything.

Rationale: Micro-interventions create clearer aha moments with less interface friction and may require only simple LLM+tool calls.

Use deterministic funnel signals before LLM-as-judge.

Chose Conversion, cart, and funnel-stage metrics first over Immediate LLM judgment of every conversation.

Rationale: Deterministic commerce events are more reliable and business-relevant than judge labels, which can themselves be wrong.

Calibrate judge prompts with recurring human labeling parties.

Chose Stakeholder labels and few-shot judge prompt iteration over Expecting a one-shot judge prompt to be accurate.

Rationale: Human-vs-LLM discrepancies reveal where the judge is lenient or strict, and those examples improve future judge prompts.

Gotchas (what bit them)

Context engineering is the production bottleneck, not just prompt or model selection.

Root cause: Real commerce decisions depend on live promotions, merchant hours, payment methods, catalogs, user history, and messy scattered enterprise data.

Resolution: Invest in data pipelines and context assembly so the LLM receives current enterprise context at query time.

Search failure breaks user trust at the start of the journey.

Root cause: Commerce agents rely on search as the first step, and wrong results stop users before cart, recommendation, or checkout tools matter.

Resolution: Treat search as a core agent subsystem with keyword, semantic, query-understanding, and reranking layers.

Technically capable chatbots can still fail A/B tests.

Root cause: Users are accustomed to existing commerce UIs and will not adopt a new interface unless its value is immediately obvious.

Resolution: Use user research, guided onboarding, familiar UI, generative UI, and contextual interventions instead of assuming a blank chatbot will be adopted.

A blank chatbot hides agent capabilities.

Root cause: Users do not know that an agent has many tools or what it can do, so a plain screen creates intimidation and failed discovery.

Resolution: Handhold users with onboarding, prompts, tooltips, suggested UI boxes, and guided journeys.

Pure text conversation is too limiting for commerce.

Root cause: Shopping and food decisions depend on images, scrolling, clicking, swiping, and visual comparison.

Resolution: Use generative UI that combines chat with dynamic visual components and interaction-aware recommendations.

Waiting for real user data delays evaluation until after the product is already failing.

Root cause: Teams assume evals need production traffic and postpone testing until launch.

Resolution: Create pre-launch datasets from team testing, seed queries, personas, and synthetic generation.

Business stakeholders get lost in overly technical evals and raw traces.

Root cause: Turn-level tool-call analysis and JSON traces expose too much low-level detail before basic business questions are answered.

Resolution: Start with simple business questions, deterministic funnel metrics, and custom annotation tools showing the actual items or restaurants users saw.

Evaluation

Prosus recommends an evaluation hierarchy: build pre-launch datasets through team simulation and synthetic generation; capture deterministic funnel metrics such as cart creation, add-to-cart, and conversion; use LLM-as-judge for whole-conversation business questions before tool-level analysis; calibrate the judge with labeling parties; and provide custom annotation tools that show conversations with the visual item context evaluators need.

Metrics: A/B conversion numbers · Cart generation or cart-link creation · Purchase/conversion · Funnel progression from search to browse to cart/order · Conversation satisfied user intent · Agent tried to close the order · Human-vs-LLM judge disagreement rate · LLM-as-judge mismatch reduction after few-shot calibration · Tool-call correctness only after higher-level business metrics

Outcomes

• Prosus operates internal productivity agents and commerce/food-ordering agent work across a portfolio with claimed two-billion-customer reach.
• The team found context engineering, not model selection alone, was decisive for production usefulness.
• Pure chatbot commerce launches underperformed in A/B tests, pushing the team toward generative UI and contextual micro-interventions.
• The search architecture matured into LLM query understanding, hybrid keyword/semantic retrieval, and LLM reranking.
• The evaluation approach prioritizes deterministic business metrics, synthetic pre-launch datasets, human labeling parties, and calibrated LLM judges.

The source reports two billion customers reached by the businesses, 15,000 internal Tokan users, three years of internal tool development, a three-month cold-start memory runway, 20 tools behind some agents, 10 UI components in the generative-UI example, five-minute contextual browsing triggers, 20/50/100 prelaunch eval examples, and 15-person labeling parties producing about 150 data points.