Building Production Web Agents for Food Ordering

ifood · E-commerce · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce web agents for food ordering

Prosus/iFood wanted agents that could complete food-ordering and other e-commerce tasks through existing websites because many consumer actions are not exposed as agent-friendly APIs.

Constraints:

• Consumer journeys run through websites, browsers, and apps rather than clean APIs.
• Food ordering requires user context such as dietary restrictions, location, address, time of day, and promotions.
• Existing web-agent frameworks produced unpredictable results on real production tasks.
• Web pages contain dynamic content, huge DOMs, popups, captchas, cookie banners, and hard-to-map coordinates.
• Execution reliability depends on limiting the agent's action space.
• Production scale requires deterministic components where possible, not free-reign agent behavior.

Architecture

The team built a custom modular web-agent framework after existing frameworks failed production-style tasks. The system separates planning from execution: a planner talks with the user, validates required information, and breaks the task into detailed steps, while an execution agent navigates a constrained web action space. The executor can run in traditional screenshot+DOM mode, fast mode using stored trajectories for familiar pages, or reflex mode using deterministic macros for predictable web actions. DOM cleaning, platform-specific instructions, merged tools, trajectory storage, simulations, and deterministic data pulls reduce LLM calls and improve reliability.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> main-agent[Main agent]
  main-agent --> deterministic-user-context-loader[Deterministic user-context loader]
  main-agent --> planning-agent[Planning agent]
  deterministic-user-context-loader --> planning-agent
  planning-agent --> platform-specific-prompt-loader[Platform-specific prompt loader]
  planning-agent --> execution-agent[Execution agent]
  platform-specific-prompt-loader --> execution-agent
  execution-agent --> browser-state-reader[Browser state reader]
  browser-state-reader --> dom-cleaner[DOM cleaner]
  dom-cleaner --> execution-agent
  execution-agent --> tool-selector[Tool selector]
  tool-selector --> website[Website]
  execution-agent --> trajectory-store[Trajectory store]
  trajectory-store --> execution-agent
  execution-agent --> reflex-automation[Reflex automation]
  reflex-automation --> website
  simulation-harness[Simulation harness] --> website
  simulation-harness --> trajectory-store
  browser-state-reader --> website

Components

Data flow

The user asks for a food-ordering outcome. The main agent gathers required deterministic context such as address and dietary restrictions, then the planner turns the request into detailed page-specific steps. The execution agent receives a constrained tool/action space plus cleaned DOM and screenshot/page state. If the page and task are familiar, fast mode uses stored trajectories and skips screenshots; if a workflow segment is fully predictable, reflex mode executes it as a macro; otherwise traditional mode loads screenshot and content for step-by-step action selection. Simulations repeatedly explore websites, store successful paths and page states, and feed that learned navigation back into future executions.

Why this architecture

The custom architecture exists because general web-agent frameworks were unreliable on production food-ordering tasks. The team treated the system as software engineering rather than data science: constrain tools, keep modules separate, make deterministic steps deterministic, load platform-specific instructions, and use the web's simulatability to map action spaces. Planning and execution are split because planning needs intent reasoning, while execution needs precise action-space translation.

Models used: Foundation planning models · Foundation execution models · Multimodal vision models · Web-agent frameworks evaluated: Multion, Web Voyager, Agent E · Selenium-like browser automation tooling

Scale

• Prosus describes serving about two billion consumers across its companies and products.
  “we interface with our consumers about two billion of them all over the world through our companies and their products”transcript, stated
• The team predicted that about 10% of e-commerce transactions could be performed by agents on behalf of consumers.
  “one of the predictions we've made at the marketplace that 10% of the e-commerce transactions will be done by agents on behalf of consumers”transcript, stated
• The web-agent project had been running for about six months.
  “Kiara who has been working on web agents for the last six months she spent a half year diving into it”transcript, stated
• The team evaluated repeated simulations such as success out of 20 attempts for specific tasks.
  “we would go to look at how many times out of 20 would this thing succeed on a task that we care about”transcript, stated
• The target success rate for representative e-commerce tasks was 80%.
  “did you have a certain accuracy score that you needed to get above so our Target was 80%”transcript, stated
• The production target for the company's platforms is tens to hundreds of millions of users.
  “because of the size of our platforms just to scale it to tens hundreds of millions of users”transcript, stated

Alternatives considered (roads not taken)

Use existing web-agent frameworks directly

Rejected because: Frameworks such as Multion and Web Voyager did not translate reliably to the team's real food-ordering tasks.

Expose clean APIs for all commerce actions

Rejected because: Many real consumer workflows are only available through browser interactions and are not yet API-ified for agents.

Give the executor all possible web tools

Rejected because: The team found that limiting the action space and merging always-paired tools improved reliability.

Always run full screenshot-based agent execution

Rejected because: Known pages and repeated tasks can use stored trajectories or deterministic reflex mode for speed and reliability.

Use an agent to query user data

Rejected because: Pulling structured user data deterministically and adding it to context is simpler and more controllable.

Rely on Web Arena benchmark performance

Rejected because: The team found benchmark averages did not translate to their target tasks and websites.

Patterns

tool-use-orchestration (agent-architecture)

The executor uses browser actions as tools, but the tool set is constrained and combined according to page/task context.

multi-agent-orchestration (agent-architecture)

The system separates a user-facing/main agent, a planner, and a constrained execution agent.

fallback-degradation (operations)

The agent attempts faster known-path modes first and falls back to slower screenshot/context-heavy execution when needed.

agent-memory (context-engineering)

Successful trajectories and page states are stored so the agent becomes familiar with repeated websites and tasks.

eval-driven-development (quality)

The team used representative e-commerce tasks, variations, and repeated simulations to optimize the system toward an 80% target.

structured-output-contract (output-control)

The planner gives broken-down, specific instructions to reduce executor ambiguity and keep execution within a known action space.

Tradeoffs

Build a custom framework

Chose A domain-specific web-agent framework for food ordering over General-purpose web-agent frameworks.

Rationale: Existing tools were optimized for general web browsing rather than the narrower production food-ordering task.

Separate planner and executor

Chose Reasoning-heavy planning plus constrained execution over A single agent doing all reasoning and clicking.

Rationale: Planning requires user-intent reasoning, while execution requires knowing the action space and completing specific web steps.

Use fast/reflex modes when possible

Chose Known-trajectory execution and deterministic macros over Full screenshot-and-reasoning execution for every page.

Rationale: Repeated familiar tasks can be faster and more reliable when the agent already knows the path or no reasoning is needed.

Limit agent tools

Chose Page/task-specific tools and merged actions over Large universal action space.

Rationale: Smaller action spaces reduce executor ambiguity and make web actions more predictable.

Prototype freely, then productionize deterministically

Chose Fast exploration followed by modular, controlled production design over Leaving the prototype as a broad high-abstraction agent.

Rationale: Large platforms need predictable systems that can scale to tens or hundreds of millions of users.

Gotchas (what bit them)

General web-agent benchmarks did not predict real task performance.

Root cause: Success varied by website/task, and real food-ordering workflows involved dynamic state changes absent from broad averages.

Resolution: Build representative simulations for the actual target tasks and measure repeated success.

Agents got stuck in loops or did not know what to do next.

Root cause: Planner tasks, action spaces, and coordinate-to-action mapping were unclear for complex pages.

Resolution: Separate planning/execution, reduce the tool/action space, and provide more specific instructions.

Screenshots and DOMs can both mislead web agents.

Root cause: Raw DOMs contain irrelevant information, while visual models struggle to map desired actions to exact coordinates.

Resolution: Clean DOMs, keep only useful clickable elements, and combine visual/page-state strategies selectively.

Over-agentifying deterministic work reduced control.

Root cause: The first design used an agent to retrieve user database information even though a direct data pull was sufficient.

Resolution: Use deterministic data loading for prerequisites and reserve agents for genuinely open-ended web navigation.

Food ordering cannot proceed without required user context.

Root cause: Execution needs prerequisite information such as address and desired dishes before web navigation can succeed.

Resolution: Make the planner deterministically verify required inputs before invoking the web executor.

High-level frameworks were useful for prototyping but not enough for production control.

Root cause: Production e-commerce agents need predictable behavior, modularity, and scale readiness.

Resolution: Distill the prototype into modular software with deterministic workflows and controlled agentic components.

Evaluation

Evaluation used representative e-commerce tasks, task variations, repeated simulations, and success-state definitions such as finding an item or adding it to cart. The target success rate was 80%, with results interpreted by website, task complexity, and user specificity.

Metrics: Success rate over repeated task attempts · Target success rate of 80% · Representative e-commerce task completion · Task variation robustness · Agents stuck in loops · Required-prerequisite completeness · Fast-mode versus traditional-mode success

Outcomes

• The team built its own modular web-agent framework after existing frameworks were unreliable for the target task.
• The architecture used planner/executor separation, three execution modes, DOM cleaning, constrained tools, and stored trajectories.
• The project target was an 80% success rate on representative e-commerce tasks.
• A major lesson was to treat web agents as software engineering systems with LLM steps, not as pure data-science projects.
• The source does not report a final production rollout percentage, latency, cost, or user-satisfaction metric.

The source reports Prosus-scale consumer context of about two billion users, a prediction that 10% of e-commerce transactions may be agent-run, six months of web-agent work, repeated task simulations such as success out of 20 attempts, an 80% target success rate, and eventual platform-scale requirements of tens to hundreds of millions of users.