Building a Scalable LLM Gateway for E-commerce Recommendations

mercado_libre · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM gateway for ecommerce recommendations

Mercado Libre needed to make generative AI available across a large ecommerce and fintech organization while handling request scale, provider limits, non-technical users, model churn, monitoring, cost visibility, security, and a real-time recommendation use case across Latin American languages.

Constraints:

• Large-scale adoption can create request volumes that exceed model limits
• More than 50,000 employees may need access, many without programming knowledge
• The model market changes quickly across providers
• The company needed metrics for quality, response times, and cost
• Information security is a priority
• Recommendation notifications needed to run in real time and support country-specific Spanish and Brazilian Portuguese

Architecture

Mercado Libre built a centralized LLM gateway on its internal Fury platform to manage, monitor, secure, and control generative AI consumption. The gateway provides one entry point across multiple LLM providers, supports routing, rate limits, fallback logic, centralized billing, security functions, SDK access, and an internal playground for non-technical employees. A production recommendation use case uses LLMs to generate push-notification copy and recommended items from user interactions with marketplace products.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  internal-user[Internal User] --> playground[Playground]
  internal-user[Internal User] --> sdk[SDK]
  playground[Playground] --> llm-gateway[LLM Gateway]
  sdk[SDK] --> llm-gateway[LLM Gateway]
  llm-gateway[LLM Gateway] --> fury-platform[Fury Platform]
  llm-gateway[LLM Gateway] --> llm-providers[LLM Providers]
  llm-gateway[LLM Gateway] --> recommendation-service[Recommendation Service]
  product-interaction-signals[Product Interaction Signals] --> recommendation-service[Recommendation Service]
  recommendation-service[Recommendation Service] --> marketplace-user[Marketplace User]

Components

Data flow

Internal users or applications access generative AI through the gateway or playground. The gateway centralizes communication with LLM providers, applies security, routing, rate limits, monitoring, and billing, and can fall back across providers. In the recommendation use case, marketplace user interactions with an item feed an LLM prompt that generates top-item recommendations and push-notification wording in the user's country language, leading users to a recommendation landing page.

Why this architecture

A gateway fit Mercado Libre's scale because it centralizes access, hides provider differences, enforces security, manages rate limits, exposes monitoring and costs, and supports fallback across providers. The internal Fury platform made it possible to scale quickly, while playground and SDK layers democratized access for both non-technical employees and coders.

Models used: LLM providers · generative artificial intelligence models

Scale

• Mercado Libre's mission addresses millions of people in Latin America.
  “services to transform the lives of millions of people in Latin America imagine the variety of possibilities and applications for using this technology infinity solutions with high potential and impact well how to take advantage of gen to do it and what”transcript, stated
• The company has more than 50,000 employees.
  “technology we are more than 50,000 employees who do not necessarily have computer knowledge how to make it available and even worse how to be aware with the large branch of models and the changing Market Day by Day additionally how to have trustability or or of the”transcript, stated
• Gateway adoption reached more than 16,000 unique users in 2023.
  “mer in 2023 we have more than 16,000 of Unix users interacting with that and for more Cod users we will a SDK tool key to facilitate and simplify the user experience in summary having a gway we can offers benefits such as centralized”transcript, stated
• The gateway exposed four major LLM providers.
  “currently we have available four of the most important llm providers that facilitates and unify the user experience in addition we are in capacity to have a fullback system with a logic to guarantee the continued”transcript, stated
• The recommendation use case asks for top 10 item recommendations.
  “answer the question about what are the top 10 items that you will be interesting in buying Bas it in your preference and the idea is that the user at the end Finish buying them some of the challenge that we face in this case was we have a real time applications then we need to give the”transcript, stated
• The gateway had more than 150 use cases.
  “today we have more than 150 user case using our Gateway we are in continuous Improvement and our our Gateway allow us to make these Technologies available in our company and generate the transformation and impact that we are”transcript, stated

Alternatives considered (roads not taken)

Let teams integrate directly with providers

Rejected because: Direct integration would duplicate provider handling, monitoring, security, rate limits, billing, and model access across the company.

Rely on one provider without fallback

Rejected because: Mercado Libre wanted four providers and fallback logic to guarantee continued service.

Expose generative AI only to engineers

Rejected because: The company needed non-technical employees to use generative AI through a playground.

Patterns

model-routing (platform)

The gateway centralizes access to multiple LLM providers and can route/fallback across them.

guardrails-io-filtering (security)

The gateway acts as a security barrier with encryption, authentication, and authorization.

cost-aware-routing (cost)

The gateway centralizes consumption and associated cost across different providers and models.

latency-optimization (performance)

The gateway applies intelligent routing and rate limits to reduce latency and improve response times.

prompt-versioning (prompt-operations)

The recommendation use case uses multiple dynamic prompts with prompt versions for experimentation.

Tradeoffs

Centralize generative AI access through a gateway

Chose A managed gateway architecture over Direct point-to-point provider integrations.

Rationale: The gateway simplified architecture, centralized control, and reduced direct dependencies between systems.

Support multiple providers with fallback

Chose Multi-provider gateway with fallback logic over Single-provider consumption.

Rationale: Multiple providers unified user experience and improved continued service when one provider path is constrained.

Use LLMs for personalized recommendation pushes

Chose LLM-generated recommendation and notification text over Only existing recommendation algorithms.

Rationale: The team wanted to improve notification algorithms, support dynamic benefits, and increase marketplace pushes.

Gotchas (what bit them)

Provider model limits became a production-scale bottleneck.

Root cause: Large request volumes per minute could exceed model limits.

Resolution: Mercado Libre centralized consumption through a gateway with rate limits, routing, and provider management.

Non-technical users could not consume provider APIs directly.

Root cause: More than 50,000 employees did not necessarily have programming knowledge.

Resolution: Mercado Libre built an internal playground and SDK toolkit.

Recommendation copy needed regional language variation.

Root cause: Latin American markets use different Spanish variants and Brazilian Portuguese.

Resolution: Prompts were made dynamic and versioned for experimentation across languages and benefits.

Evaluation

Mercado Libre monitors quality, response times, cost, recommendation quality, and NPS expectations through the gateway and use-case metrics, though the transcript does not describe a detailed offline evaluation harness.

Metrics: quality response times and cost · recommendation quality · NPS · unique gateway users · number of use cases

Outcomes

• The gateway had more than 16,000 unique users in 2023
• More than 150 use cases used the gateway
• The recommendation use case showed improvements and expected positive NPS impact

The transcript reports more than 50,000 employees, more than 16,000 gateway users in 2023, four LLM providers, and more than 150 use cases.