Practical Lessons from Deploying LLMs in Production at Scale

mercado_libre · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM applications for developer documentation, data documentation, and structured extraction

Mercado Libre needed to move LLM applications from impressive prototypes into production use across internal documentation, data-catalog documentation, and natural-language input interpretation, while avoiding hallucinations, poor documentation quality, vague prompts, and overuse of raw LLM calls where structured outputs or simpler preprocessing were better.

Constraints:

• Company-specific questions require proprietary context that base LLMs do not possess.
• RAG systems hallucinated when users asked about undocumented internal tools or workflows.
• Existing table documentation was too cursory for either humans or the RAG system.
• Roughly half of 4,000 production tables lacked adequate documentation.
• Generated documentation had to satisfy table owners and handle acronyms, structure, and cases where existing documentation was already good.
• Natural-language extraction had to interpret ambiguous phrases such as dates and product quantities into structured fields.
• The company wanted to reserve higher-cost models for cases where simpler or cheaper approaches were insufficient.

Architecture

Mercado Libre explored a portfolio of LLM production applications. For developer documentation, it used Llama Index to build a RAG pipeline that constructs and stores knowledge indexes, retrieves relevant context, and generates answers with source links. For data documentation, it generated table descriptions from table names, field names, field types, and existing technical documentation, then iterated prompts and QA based on table-owner feedback. For natural-language input interpretation, it used function calling so LLMs could extract dates, product quantities, and booking intents into predefined structured outputs instead of prose.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Internal_documentation_corpus["Internal documentation corpus"] --> Llama_Index_RAG_pipeline["Llama Index RAG pipeline"]
  Llama_Index_RAG_pipeline --> Knowledge_index["Knowledge index"]
  Knowledge_index --> Context_retriever["Context retriever"]
  Context_retriever --> LLM_answer_generator["LLM answer generator"]
  LLM_answer_generator --> Test_query_suite["Test query suite"]
  Test_query_suite --> Documentation_improvement_loop["Documentation improvement loop"]
  Documentation_improvement_loop --> Internal_documentation_corpus
  Data_catalog_metadata["Data catalog metadata"] --> Documentation_generation_prompt["Documentation generation prompt"]
  Documentation_generation_prompt --> Table_owner_QA_workflow["Table-owner QA workflow"]
  Table_owner_QA_workflow --> Prompt_iteration_process["Prompt iteration process"]
  Prompt_iteration_process --> Documentation_generation_prompt
  Natural_language_input["Natural language input"] --> Function_calling_extractor["Function-calling extractor"]
  Function_calling_extractor --> Booking_and_product_workflows["Booking and product workflows"]

Components

Data flow

Internal documentation and tool knowledge are indexed by Llama Index, retrieved in response to developer questions, and passed to the LLM to generate sourced answers. Table metadata and technical documentation feed a documentation-generation prompt that produces draft descriptions for owners to review. Raw user text for products or bookings is passed through function-calling schemas so the model returns structured fields such as item counts, requested topics, and date values. Across these flows, test queries, stakeholder feedback, prompt iteration, and QA loops refine the data and prompts.

Why this architecture

The architecture separates use cases because each LLM failure had a different root. Developer Q&A needed retrieval over proprietary context; table documentation needed better source data and prompt QA; and raw natural language extraction needed structured function-calling outputs. The article repeatedly emphasizes preparing data, narrowing the task, using clear schemas, and iterating prompts and QA rather than treating raw LLMs as a universal solution.

Models used: GPT-3.5-turbo · GPT-4V · LLaMA 2 · LLMs

Scale

• The RAG system had to answer questions about widely used tools and proprietary Mercado Libre tools.
  “This included widely used tools like BigQuery, Tableau, Looker, as well as proprietary tools such as Fury Data Applications, Data Flow, and others.”article, stated
• The data-documentation problem covered 4,000 production tables.
  “For instance, out of 4,000 productive tables, half lacked adequate documentation..”article, stated
• About half of those 4,000 production tables lacked adequate documentation.
  “out of 4,000 productive tables, half lacked adequate documentation..”article, stated
• The first documentation-generation prompt was accepted with minor changes by about 90% of stakeholders.
  “we were able to generate documentation that was well-received by 90% of our stakeholders.”article, stated
• The remaining 10% of table owners rejected the generated documentation for three main reasons.
  “we found that the 10% who didn’t like our documentation had three main concerns:”article, stated
• The product-quantity example includes a label with 25 sheets and 2000 total labels.
  “An image of a listing of a single product with 25 pages full of etiquetes, for a total of 2000.”article, stated

Alternatives considered (roads not taken)

Expect the base LLM to know Mercado Libre internal tools

Rejected because: The article states that company-specific internal knowledge is not available to general LLMs without retrieval context.

Let the RAG system answer even when documentation is missing

Rejected because: User adoption exposed hallucinated or inaccurate responses when the documentation lacked the needed information.

Manually document thousands of tables

Rejected because: The scope made manual documentation infeasible and not cost-effective, so the team used LLMs to enrich documentation.

Use a static generic documentation prompt

Rejected because: The 10% rejection group showed the need for clearer structure, acronym handling, and conditional behavior based on available inputs.

Use normal prose generation for extraction tasks

Rejected because: The Data Doctors and product examples needed structured values such as dates, quantities, topics, and agendas, so function calling was better suited.

Patterns

rag-retrieval (architecture)

Mercado Libre used Llama Index to retrieve relevant internal documentation and generate sourced answers for developers.

eval-driven-development (quality)

The team built test queries for answers the system should provide and questions it should avoid, using failures to reveal documentation gaps.

data-quality-loop (operations)

RAG failures led the team to improve the underlying documentation rather than only tuning prompts or models.

prompt-versioning (operations)

Documentation generation improved through iterative prompt engineering and QA of generated outputs.

structured-output-contract (quality)

Function calling was used to force natural-language extraction into predefined structured fields rather than free-form explanations.

context-engineering (architecture)

The team repeatedly narrows and improves model context through retrieval, better documentation, dynamic prompts, and structured inputs.

cost-aware-routing (operations)

The final guidance explicitly recommends testing whether simpler and less expensive models can satisfy the task before escalating.

Tradeoffs

Use RAG for company-specific developer questions

Chose retrieval over internal documentation over expecting a general LLM to know proprietary tools.

Rationale: RAG bridges the gap between general language ability and Mercado Libre-specific information.

Improve documentation after RAG failures

Chose documentation enhancement over only prompt or model tuning.

Rationale: The failures revealed missing or problem-misaligned source data, so better documentation was the real fix.

Generate table documentation with LLMs

Chose automated documentation drafts over manual documentation for thousands of tables.

Rationale: The scale was too large for cost-effective manual work, and 90% stakeholder acceptance made automation useful.

Use function calling for extraction

Chose structured function outputs over free-form text explanations.

Rationale: Operational workflows needed concrete values such as dates and counts, not long natural-language reasoning about them.

Simplify tasks before using expensive models

Chose preprocessing and clear task design over raw high-cost LLM calls for every problem.

Rationale: The final guidance is to prepare data and test simpler options first, escalating only when needed.

Gotchas (what bit them)

A good RAG prototype hid missing documentation until broader adoption.

Root cause: Early demo queries had enough source context, but users asked about topics not covered in the knowledge base.

Resolution: Build explicit test sets for answerable and non-answerable questions and improve documentation when gaps appear.

The model hallucinated when the knowledge base lacked the answer.

Root cause: Without adequate retrieved context, the LLM fell back to general pretraining knowledge for Mercado Libre-specific questions.

Resolution: Constrain answers to contextual knowledge and improve the underlying documentation.

Procedure-oriented documentation did not match problem-oriented user queries.

Root cause: Some documents described processes without explaining why a user would need them.

Resolution: Rewrite documentation so it connects procedures to user problems and contexts.

Table descriptions could not be added to RAG effectively while catalog metadata was shallow.

Root cause: Existing data-catalog documentation lacked depth, relationships, and use cases.

Resolution: Generate richer table documentation from metadata and technical docs, then review it with table owners.

A generic documentation prompt was not enough for the last 10% of stakeholders.

Root cause: Rejected outputs lacked structure, missed internal acronyms, or worsened already-good documentation.

Resolution: Use guided, flexible prompts with a clear schema and QA questions about behavior and information use.

Raw text contains values that are semantically obvious to people but ambiguous to simpler parsers.

Root cause: Product labels and booking requests contain dates, quantities, abbreviations, and units that vary in form.

Resolution: Use LLM reasoning with function calling to extract structured values from the text.

Evaluation

Mercado Libre used practical QA loops rather than a single offline benchmark: RAG was tested against specific queries it should answer and queries it should not answer; generated documentation was reviewed by table owners and then improved based on rejection reasons; and natural-language extraction was framed around whether function-calling outputs matched the structured fields needed by downstream workflows.

Metrics: Answer accuracy on required RAG queries · Correct refusal or non-response on unsupported RAG queries · Documentation coverage and depth · Stakeholder/table-owner acceptance rate · Prompt QA questions about intended behavior and information use · Correct structured extraction of quantities, dates, topics, and agenda values

Outcomes

• The Llama Index RAG prototype worked well as a developer documentation search engine but exposed documentation gaps after broader use.
• Testing revealed undocumented internal tools and procedure-focused documentation that did not match user problems.
• LLM-generated table documentation was accepted by about 90% of stakeholders with minor adjustments.
• The remaining 10% of stakeholders drove improvements around structure, acronyms, and avoiding unnecessary rewrites.
• Function calling gave Mercado Libre a structured way to interpret raw product and booking text.
• The company concluded that raw LLMs are not a universal solution and that data preparation, task simplification, QA, and model-cost choices matter.

The article reports production-scale context including tools used across Mercado Libre, 4,000 production tables with half inadequately documented, 90% initial table-owner acceptance for generated documentation, and natural-language extraction tasks involving product quantities and Data Doctors booking requests.