Multi-LLM Orchestration for Product Matching at Scale

mercado_libre · E-commerce · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce product matching

Mercado Libre needed to identify identical products across noisy seller listings for pricing, ads, personalization, search, and inventory without unsafe autonomous errors.

Constraints:

• Seller titles, photos, attributes, and identifiers are inconsistent.
• Wrong matches are costly in pricing and promotion decisions.
• The system must evaluate millions of items and matches cost-effectively.
• Existing ML and embeddings were sophisticated but not accurate enough for full autonomy.

Architecture

Mercado Libre evolved from a simple LLM prototype to a seven-node multi-LLM orchestration pipeline layered alongside embeddings, ANN retrieval, ML scoring, adaptive prompts, context-aware decisions, consensus, and human oversight for sensitive cases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  target-item-selector["Target Item Selector"]
  embedding-and-ann-retrieval["Embedding and ANN Retrieval"]
  ml-match-scorer["ML Match Scorer"]
  multi-llm-orchestration-pipeline["Multi-LLM Orchestration Pipeline"]
  verdi-orchestration-system["Verdi Orchestration System"]
  human-oversight["Human Oversight"]
  vector-databases["Vector Databases"]
  target-item-selector --> embedding-and-ann-retrieval
  embedding-and-ann-retrieval --> ml-match-scorer
  embedding-and-ann-retrieval --> vector-databases
  ml-match-scorer --> multi-llm-orchestration-pipeline
  multi-llm-orchestration-pipeline --> ml-match-scorer
  multi-llm-orchestration-pipeline --> human-oversight
  verdi-orchestration-system --> multi-llm-orchestration-pipeline
  human-oversight --> multi-llm-orchestration-pipeline
  vector-databases --> embedding-and-ann-retrieval

Components

Data flow

Target items are selected, encoded with embeddings, retrieved through ANN as candidate matches, scored by existing ML, then evaluated through a multi-node LLM orchestration pipeline using adaptive prompts and consensus before high-stakes decisions.

Why this architecture

The architecture combines cheap candidate generation and existing ML with LLM reasoning because high-stakes product matching needs both scale and human-level precision at very low cost.

Models used: multiple LLMs · open-source embedding models · Mercado Libre ML models

Scale

• The final system achieved 95% precision and over 50% recall.
  “they achieved human-level performance with 95% precision and over 50% recall”summary, stated
• The system operates at less than $0.001 per request.
  “at a cost-effective rate of less than $0.001 per request”summary, stated
• The LLM pipeline evolved from two nodes to seven nodes.
  “evolved from a simple 2-node architecture to a complex 7-node pipeline”summary, stated
• Initial LLM work improved from 38% precision and 13% recall to 65% precision and 79% recall after prompt refinement.
  “Their initial three-step LLM pipeline produced disappointing results with only 38% precision and 13% recall”summary, stated

Alternatives considered (roads not taken)

Pure existing ML and embeddings

Rejected because: The prior system could not reach autonomous accuracy for high-stakes scenarios like pricing.

One large model solving matching directly

Rejected because: The team found consistency, cost, and decomposition mattered more than raw model capability alone.

Patterns

multi-agent-orchestration (architecture)

A seven-node LLM workflow uses task decomposition, adaptive prompts, and consensus.

rag-retrieval (retrieval)

Embedding models and ANN retrieval generate candidate product matches before LLM reasoning.

eval-driven-development (quality)

The team iterated on small labeled samples and scrutinized failures to improve prompts and flow.

Tradeoffs

Use multi-node orchestration

Chose Structured LLM components and consensus over A single prompt or model call.

Rationale: Breaking the task into human-like steps improved reliability.

Keep cost under tight thresholds

Chose Careful orchestration with cheap per-request cost over Unbounded use of large models.

Rationale: Millions of item comparisons require sub-cent economics.

Gotchas (what bit them)

High recall alone is dangerous for pricing.

Root cause: False matches can distort competitive pricing and promotions.

Resolution: Optimize for human-level precision and keep human oversight for sensitive cases.

Automated prompting tools were less useful than expected.

Root cause: The task required understanding failure cases and product-domain nuance.

Resolution: Use systematic human-driven iteration over small labeled samples.

Evaluation

Mercado Libre used human-labeled samples, precision/recall metrics, iterative failure analysis, and cost-per-request constraints.

Metrics: precision · recall · cost per request

Outcomes

• The final system reached 95% precision and over 50% recall.
• It costs less than $0.001 per request.
• It enables scalable autonomous product matching across millions of items.

Millions of item comparisons, seven LLM nodes, and less than $0.001 per request.