Improving Local Search with Multimodal LLMs and Vector Search

offerup · E-commerce · 2025 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketplace search relevance with multimodal vector retrieval

OfferUp's keyword-based Elasticsearch search could surface broad inventory, but it could not understand query context, synonyms, visual similarity, or multi-concept intent well enough for local marketplace discovery. The team needed to modernize search with multimodal semantic retrieval while preserving high throughput, low latency, local relevance, and measurable business outcomes.

Constraints:

• existing search used Elasticsearch v7.10 on EC2 with BM25 keyword ranking
• keyword search could not disambiguate terms such as apple across contexts
• synonyms and variations such as car versus sedan were missed
• multi-concept queries such as red running shoes returned partial matches
• marketplace search depends on local relevance and seller engagement
• the system had to index millions of active listings and millions more added monthly
• managed services introduced AWS dependency but reduced operational burden

Architecture

OfferUp transformed its search pipeline from lexical BM25 search to multimodal vector search using Amazon Titan Multimodal Embeddings G1 and Amazon OpenSearch Service. New and updated listings still flow through S3 uploads, posting and writer microservices, DynamoDB, SNS, SQS, and a Lambda indexer; the new indexer path retrieves images, base64-encodes them, sends listing data to OpenSearch, and uses an OpenSearch ingest pipeline with a Bedrock connector to call Titan and store image/description embeddings. Query-time neural search sends text or image queries through the same Titan embedding model, then OpenSearch performs k-NN retrieval over vectors. A/B tests and production-data relevance experiments guided rollout, including choosing k=128 and initially backfilling 12 million listings in three high-density DMAs.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  OfferUp_user["OfferUp user"] --> Amazon_S3["Amazon S3"]
  OfferUp_user --> Posting_microservice["Posting microservice"]
  Posting_microservice --> Listing_writer_microservice["Listing writer microservice"]
  Listing_writer_microservice --> Amazon_DynamoDB["Amazon DynamoDB"]
  Listing_writer_microservice --> Amazon_SNS["Amazon SNS"]
  Amazon_SNS --> Amazon_SQS["Amazon SQS"]
  Amazon_SQS --> Lambda_indexer["Lambda indexer"]
  Amazon_DynamoDB --> Listing_reader_microservice["Listing reader microservice"]
  Listing_reader_microservice --> Lambda_indexer
  Amazon_S3 --> Image_microservice["Image microservice"]
  Image_microservice --> Lambda_indexer
  Lambda_indexer --> Amazon_OpenSearch_Service["Amazon OpenSearch Service"]
  Amazon_OpenSearch_Service --> OpenSearch_ingest_pipeline["OpenSearch ingest pipeline"]
  OpenSearch_ingest_pipeline --> Bedrock_connector["Bedrock connector"]
  Bedrock_connector --> Amazon_Titan_Multimodal_Embeddings["Amazon Titan Multimodal Embeddings"]
  Amazon_Titan_Multimodal_Embeddings --> Amazon_OpenSearch_Service
  OfferUp_user --> Search_microservice["Search microservice"]
  Search_microservice --> Neural_search_pipeline["Neural search pipeline"]
  Neural_search_pipeline --> Amazon_Titan_Multimodal_Embeddings
  Neural_search_pipeline --> k_NN_retrieval["k-NN retrieval"]
  k_NN_retrieval --> Amazon_OpenSearch_Service
  Search_microservice --> A_B_testing["A/B testing"]
  A_B_testing --> Business_metrics["Business metrics"]
  A_B_testing --> Technical_relevance_metrics["Technical relevance metrics"]

Components

Data flow

When users create or update listings, images go to S3 and listing metadata flows through the posting microservice, listing writer, DynamoDB, SNS, SQS, and Lambda indexer. The indexer retrieves the full listing and encoded images, then OpenSearch invokes the Bedrock connector so Titan Multimodal Embeddings creates vectors for images and descriptions. At query time, the search microservice forwards text or image searches to OpenSearch neural search, which embeds the query with the same Titan model and uses k-NN to retrieve similar listings. A/B tests compare the multimodal variant against keyword-search control groups.

Why this architecture

The architecture preserves OfferUp's proven listing-event pipeline while adding multimodal embeddings at indexing and query time. Titan gives a shared semantic space for text and images, OpenSearch supplies managed vector search and neural-search pipelines, and A/B testing keeps business and technical relevance improvements measurable before wider rollout.

Models used: Amazon Titan Multimodal Embeddings G1 · amazon.titan-embed-image-v1 · BM25 keyword ranking baseline

Scale

• OfferUp hosts millions of active listings, with millions more added monthly.
  “OfferUp hosts millions of active listings, with millions more added monthly by its users.”article, stated
• Titan Multimodal Embeddings supports 256 max text tokens and 25 MB max image input.

  "Max input text tokens – 256

• Max input image size – 25 MB"

— article, stated

• Titan outputs vectors of 1,024 dimensions by default, with 384 and 256 options.
  “Output vector size – 1,024 (default), 384, 256”article, stated
• The OpenSearch deployment used 3 Availability Zones.
  “The OpenSearch domain is deployed across 3 Availability Zones (AZs) to provide high availability.”article, stated
• The cluster used 3 cluster manager nodes and 24 data nodes.
  “The cluster comprises 3 cluster manager nodes (m6g.xlarge.search instance) dedicated to manage cluster operations. For data handling, 24 data nodes”article, stated
• The index used 12 shards and three read replicas, with about 11.6GB memory per shard.
  “The index is configured with 12 shards and three read replicas to enhance read performance. Each shard consumes around 11.6GB of memory.”article, stated
• The first backfill converted 12 million active listings.
  “In the first phase, OfferUp backfilled 12 million active listings.”article, stated
• The initial rollout targeted three high-density business-critical locations.
  “OfferUp initially identified three business-critical high-density locations where multimodal search solutions demonstrated promising results”article, stated
• A/B testing selected k=128 for k-NN retrieval.
  “After extensive A/B testing with various k values, OfferUp found that a k value of 128 delivers the best search results”article, stated
• Technical relevance recall used 6 months of production data and top k=10 results.
  “utilizing 6 months of production system data to examine relevance recall with a focus on the top k=10 most relevant results”article, stated

Alternatives considered (roads not taken)

Continue BM25 lexical keyword search

Rejected because: Keyword search missed context, synonyms, and multi-concept query intent.

Pure text retrieval

Rejected because: Marketplace listings include images, and Titan embeds both text and images into a common semantic space.

Immediate global rollout

Rejected because: OfferUp phased the launch through high-density DMAs and A/B testing.

Use arbitrary k-NN result depth

Rejected because: A/B tests showed k=128 balanced search result quality with compute resources.

Patterns

rag-retrieval (retrieval)

OfferUp retrieves marketplace listings by embedding text and image queries and searching nearest listing vectors in OpenSearch.

retrieval-ranking (retrieval)

The search stack tunes k-NN result depth and measures relevance recall to improve ranked local results.

batch-offline-processing (data)

OfferUp backfilled active listings into multimodal vectors before rollout.

eval-driven-development (quality)

Offline experiments, A/B tests, business metrics, and relevance-recall calculations determined rollout and parameter choices.

latency-optimization (operations)

The team chose managed OpenSearch vector search and tuned k=128 to balance quality with compute resources and fast responses.

Tradeoffs

Managed multimodal search stack

Chose Amazon Titan Multimodal Embeddings plus Amazon OpenSearch Service over self-managed vector infrastructure.

Rationale: Managed services simplified deployment and operation while meeting throughput and latency needs, at the cost of AWS dependency.

Multimodal semantic retrieval

Chose text and image embeddings in one semantic space over lexical search over listing text only.

Rationale: Marketplace listings rely on both image and text semantics, and embeddings improve context and similarity beyond keywords.

Phased DMA rollout

Chose start with three high-density business-critical locations over full-market rollout.

Rationale: Offline experiments identified promising markets and A/B testing controlled risk.

k-NN depth

Chose k=128 over untuned default k values.

Rationale: A/B testing found k=128 delivered the best results while managing compute resources.

Gotchas (what bit them)

Keyword search could not infer semantic context.

Root cause: BM25/keyword search treated terms lexically and could not distinguish intent across contexts.

Resolution: Embed text and images into a shared semantic vector space and retrieve by vector similarity.

Synonyms and adjacent product terms were missed or overmatched.

Root cause: Lexical search lacked semantic awareness of word relationships and product variations.

Resolution: Use multimodal embeddings and vector similarity rather than exact keyword matching alone.

Complex multi-concept queries produced partial matches.

Root cause: The foundational search treated keywords independently rather than modeling combined intent.

Resolution: Use Titan multimodal embeddings and neural search to represent query meaning.

Vector search quality depended on k and compute cost.

Root cause: Higher or lower k values change result quality and resource use.

Resolution: Run extensive A/B tests and select k=128.

Large-scale listing backfill had to precede useful search coverage.

Root cause: Active listings needed image vectors stored in OpenSearch before the new query path could retrieve them.

Resolution: Backfill active listing images into Titan embeddings and OpenSearch before rollout.

Evaluation

OfferUp evaluated multimodal search with A/B tests comparing keyword-search control groups against multimodal variants, tracked business metrics such as engagement and EWSR, and used six months of production data to measure relevance recall on top k=10 results across high- and low-density DMAs.

Metrics: relevance recall improved 27% · geographic spread reduced 54% · user engagement increased 2.2% · EWSR improved 3.8% · search depth grew 6.5% · fanout searches decreased 54.2% · ad impressions rose 0.91% · top-k relevance recall used k=10 · k-NN search selected k=128

Outcomes

• OfferUp reported 27% improved relevance recall.
• Geographic spread fell by 54%, meaning results were more local.
• User engagement rose 2.2% and EWSR improved 3.8%.
• Search depth grew 6.5%, fanout searches decreased 54.2%, and ad impressions rose 0.91%.
• The first phase backfilled 12 million active listings into vector embeddings.
• The case demonstrates a production path from lexical search to multimodal vector retrieval while keeping event-driven indexing infrastructure.

This is an AWS co-written case study, so vendor-service framing and metrics should be treated as reported by OfferUp/AWS. The architecture itself is detailed enough to extract reusable production patterns for multimodal retrieval, vector indexing, phased rollout, and A/B evaluation.