Building a Conversational Shopping Assistant with Multi-Modal Search and Agent Architecture

olx · E-commerce · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Multimodal ecommerce shopping assistant

OLX wanted a conversational shopping experience for a large secondhand marketplace that could handle broad, specific, text, image, and voice requests while still using catalog search effectively.

Constraints:

users search across a broad classified marketplace with many product categories
requests range from broad discovery to very specific product matching
non-technical users may not know how to prompt a chat interface
agentic flows add latency and cost compared with traditional search
catalog search tools must scale to support production traffic

Architecture

OLX built Magic, an agentic shopping assistant that combines a familiar search UI with conversational refinement, multimodal search, a model-router-backed agent brain, catalog tools, hybrid keyword/semantic search, visual search, guardrails, language detection, streaming responses, and A/B testing from existing OLX search context.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  experiment-entrypoint[Experiment entrypoint] --> magic-ui[Magic UI]
  magic-ui --> magic-agent[Magic agent]
  magic-agent --> model-router[Model router]
  model-router --> llm-pool[LLM pool]
  magic-agent --> tool-layer[Tool layer]
  tool-layer --> hybrid-search[Hybrid search]
  tool-layer --> visual-search[Visual search]
  tool-layer --> web-search[Web search]
  hybrid-search --> olx-catalog[OLX catalog]
  visual-search --> olx-catalog
  magic-agent --> guardrails-and-language-detection[Guardrails and language detection]
  magic-agent --> streaming-response[Streaming response]
  streaming-response --> magic-ui

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Experiment entrypoint`	A/B-tested entry from the existing OLX search platform into Magic.	`Magic UI`
`Magic UI`	Search and conversational interface for text, image, voice, refinements, smart compare, and result browsing.	`Experiment entrypoint`, `Magic agent`, `Streaming response`
`Magic agent`	Agent brain that interprets intent, plans tool calls, asks clarifying questions, and composes responses.	`Magic UI`, `Model router`, `Tool layer`, `Guardrails and language detection`, `Streaming response`
`Model router`	Routes agent work across commercial and internally fine-tuned LLMs by use case.	`Magic agent`, `LLM pool`
`LLM pool`	Best-in-class commercial LLMs and internally fine-tuned LLMs.	`Model router`
`Tool layer`	Agent-callable tools for web search, text search, visual search, URL parsing, retrieval, and ranking.	`Magic agent`, `Hybrid search`, `Visual search`, `Web search`
`Hybrid search`	Combines keyword and semantic search over the OLX catalog.	`Tool layer`, `OLX catalog`
`Visual search`	Finds visually similar products and supports image-plus-modifier queries.	`Tool layer`, `OLX catalog`
`Web search`	Retrieves external information for broad shopping questions that are not fully represented in the catalog.	`Tool layer`
`Guardrails and language detection`	Policy and language handling layer for keeping Magic on shopping tasks.	`Magic agent`
`OLX catalog`	Marketplace listings, images, descriptions, prices, and category data.	`Hybrid search`, `Visual search`
`Streaming response`	Streams partial results back to reduce perceived latency.	`Magic agent`, `Magic UI`

Data flow

A user enters text, image, voice, or an OLX search context into Magic. The agent brain interprets intent, selects models through a router, plans calls to tools such as web search, text/hybrid search, visual search, and URL parsing, passes through guardrails and language detection, streams responses back, and surfaces results from the OLX catalog.

3 supporting quotes

“our agent is able to interact with the user uh in natural language and understand the user's intent and request then autonomously perform tasks”transcript, implied

“our brain has access to several uh models um through a model router and we are using um both uh combination of both uh best in in class commercial llms”transcript, stated

“this agent um is able to call and has access to several tools according to the generated uh plan and within these tools we can see web search as Nish showed in the demo text search and visual search”transcript, stated

Why this architecture

OLX combined agentic planning with familiar search affordances because pure chat was too unfamiliar for many users, while hybrid and visual search were needed to retrieve marketplace items accurately across text and image requests. Streaming and guardrails address the latency and misuse risks introduced by agentic flows.

3 supporting quotes

“we bring the experience closer to what users already understand and they're familiar with so again let me uh walk you through a quick demo so it starts now uh with a search bar”transcript, implied

“hybrid search uh is a combination of both um keywords of keyword search and semantic search and it really is the best of both worlds”transcript, stated

“we also do uh this in streaming way to try to alleviate the latency issue that we get from passing through all these um these steps and and calls”transcript, stated

Models used: commercial LLMs · internally fine-tuned LLMs · reranker flows · LLM-as-a-judge

Scale

A Magic search can return 25 catalog results for a product request.
“a lot of things are happening and uh I get these 25 results presented to me which I can scroll and uh some of these”transcript, stated
OLX was A/B testing Magic in production.
“slides yeah so uh the product is live so we are currently AB testing it and uh the way we AB test is uh uh so on the”transcript, stated
About one third of users used richer natural-language queries while two thirds still used keyword-style queries.
“oneir of people are starting to write uh give more context and write in natural language but 2/3 of them are already using it are still using it like uh uh like a keyword search”transcript, implied
Agent responses take 5 to 10 seconds compared with 300 to 500 milliseconds for typical search.
“there's a latency of responses 5 to 10 seconds uh which for example in a typical search you would expect something like 300 millisecond 500 millisecond”transcript, stated

Alternatives considered (roads not taken)

Pure chat interface

Rejected because: User interviews showed that chat is not always the best interface for simple shopping intent.

“conversational interface might not be suit so though it's powerful because you know it helps you uh refine your search but it might not be the best choice for everything”transcript, stated

Keyword-only search

Rejected because: Keyword search missed semantically similar catalog items that hybrid search could retrieve.

“in traditional keyword matching search I only get one result because this only this product has exactly this keywords but if I use semantic it will understand the meaning”transcript, stated

Text-only product search

Rejected because: Some user intent is better expressed through images and modifiers than through descriptions.

“a image is worth more than a thousand words so um we have our visual search um and we can um so upload our an image and and find um visually similar products”transcript, stated

Patterns

`tool-use-orchestration` (agent-design)

Magic's agent generates a plan and calls web search, text search, visual search, URL parsing, retrieval, and ranker tools.

“this agent um is able to call and has access to several tools according to the generated uh plan”transcript, stated

`model-routing` (model-ops)

The agent brain uses a model router to choose between commercial and internally fine-tuned LLMs.

“our brain has access to several uh models um through a model router”transcript, stated

`hybrid-search` (retrieval)

OLX combines keyword and semantic search so Magic can match both literal and meaning-based shopping requests.

“hybrid search uh is a combination of both um keywords of keyword search and semantic search”transcript, stated

`retrieval-ranking` (retrieval)

The tool layer includes retrieval and ranker flows for text and visual search results.

“within um these visual search and text search uh we include our retrieval and ranker flows”transcript, stated

`safety-policy-enforcement` (safety)

Magic applies guardrails and language detection, tightening guardrails after users explored non-shopping requests.

“we also highlight our steps of uh guard rails and language detection and we had to iterate a bit on this”transcript, stated

`llm-as-judge` (evaluation)

OLX used LLM-as-judge and human-in-the-loop evaluation to guide search-system development.

“we used techniques like like llm as a judge and human in the loop and they were really um important to help us”transcript, stated

Tradeoffs

Primary user interface

Chose familiar search bar with agentic helper features over pure chat-first shopping assistant.

Rationale: The search bar matched existing user habits while still enabling natural-language refinement and multimodal inputs.

“it starts now uh with a search bar uh with a search bar which is much much more powerful but it's also familiar to the users”transcript, stated

Search retrieval strategy

Chose hybrid keyword plus semantic search over keyword-only or semantic-only retrieval.

Rationale: Hybrid search combined exact matching with meaning-based recall for marketplace queries.

“with ibrid we can combine both um and so yeah this is a key enabler for um for search in in Magic”transcript, stated

Response delivery

Chose streaming responses over waiting for all agent calls to complete before showing output.

Rationale: Streaming reduces perceived latency from multi-step model and tool calls.

“we stream back uh all these responses back to the user and we also do uh this in streaming way to try to alleviate the latency issue”transcript, stated

Gotchas (what bit them)

Pure chat is not always the right shopping interface.

Root cause: Many shoppers expect immediate search results or visible controls rather than open-ended chat prompts.

Resolution: OLX moved the second version closer to familiar search with buttons, filters, smart groups, and contextual tags.

“there are it it suits in some environments but uh but not everywhere”transcript, stated

Users did not discover hidden chat capabilities such as smart compare.

Root cause: Non-technical users did not know what the chat interface could do.

Resolution: Expose features as visible product controls instead of relying on users to type the right command.

“you can actually in the in the message type it to compare but people don't know about it uh if there's a button for smart compare then you know it's in front of them”transcript, stated

Agentic search is much slower than conventional search.

Root cause: Magic performs many model and tool calls under the hood.

Resolution: Stream responses and continue work on latency, cost, and scalable tools.

“agents uh so you know there's a lot of lot which is happening uh under the hood uh many calls so it takes time”transcript, stated

Evaluation

OLX evaluated Magic with A/B testing, user interviews, feature-level conversion tracking, LLM-as-judge, and human-in-the-loop assessment of the search system.

Metrics: A/B test results · conversion by feature and category · natural-language versus keyword query share · latency of 5 to 10 seconds · typical search latency of 300 to 500 milliseconds

“we built it we put it in front of the customers uh we see how customers are using a product in fact we interviewed a lot of these people”transcript, stated

“we track all the different features and uh we see different conversion for different categories for each of these features”transcript, implied

Outcomes

Magic supports conversational refinement, visual search, smart compare, smart groups, contextual tags, and natural-language filters.
OLX was running an A/B test that launches Magic from existing search context.
User behavior showed partial adoption of natural language, with many still using keyword-style search.

The production A/B test reported 5-10 second agent response latency versus 300-500 ms traditional search expectations.

“it actually takes their entire context from Ox uh feeds it to Magic and it already pre- prompts magic uh for results”transcript, stated

“latency of responses 5 to 10 seconds uh which for example in a typical search you would expect something like 300 millisecond 500 millisecond”transcript, stated