Improving Multilingual Search with Few-Shot LLM Translations

delivery_hero · E-commerce · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Multilingual marketplace search translation and hybrid retrieval

Delivery Hero needed a globally served search system to understand food and grocery queries across languages, dialects, transliterations, typos, and markets. Traditional machine translation and keyword search struggled with intent and contextual nuance, so the company needed more accurate LLM translations that could feed both Elastic Search and vector search without excessive cost, hallucination, or latency.

Constraints:

• search is served centrally from Berlin to 68 countries
• markets use many languages including English, Spanish, Arabic, Turkish, Greek, Chinese, and Thai
• single countries can contain multiple languages and dialectal variants
• transliteration and spelling errors are common in non-Latin-script markets
• ChatGPT-Turbo was more accurate but more expensive than Gemini
• too many few-shot examples caused hallucination and increased token count
• LLM translations had to improve hybrid search rather than replace existing search infrastructure

Architecture

Delivery Hero evaluates translation options, selects Gemini for the balance of accuracy and cost, and uses few-shot prompting to handle dialects, transliteration, spelling corrections, and context-aware translation. The prompt is tuned with deterministic parameters and a small curated set of edge cases; ChatGPT is used to generate initial Arabic examples when labeled data is scarce, and the model itself helps clarify prompt instructions. At runtime, each query is translated three times and the majority response is selected to reduce outliers. Translations are used in two retrieval paths: Elastic Search considers the original query plus translations, while Vector Search enriches training data with LLM-generated translations so regional variations and synonyms retrieve similar products.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  User_search_query["User search query"] --> Few_shot_Gemini_translation_prompt["Few-shot Gemini translation prompt"]
  Prompt_example_dataset["Prompt example dataset"] --> Few_shot_Gemini_translation_prompt
  ChatGPT_generated_samples["ChatGPT-generated samples"] --> Prompt_example_dataset
  Few_shot_Gemini_translation_prompt --> Gemini["Gemini"]
  Gemini --> Majority_voting["Majority voting"]
  Majority_voting --> LLM_generated_translation["LLM-generated translation"]
  User_search_query --> Elastic_Search["Elastic Search"]
  LLM_generated_translation --> Elastic_Search
  LLM_generated_translation --> Vector_Search_training_data["Vector Search training data"]
  Vector_Search_training_data --> Vector_Search["Vector Search"]
  Elastic_Search --> Hybrid_result_mixer["Hybrid result mixer"]
  Vector_Search --> Hybrid_result_mixer
  Hybrid_result_mixer --> Search_results["Search results"]
  Local_user_evaluators["Local user evaluators"] --> Offline_translation_evaluation["Offline translation evaluation"]
  Offline_translation_evaluation --> Model_selection["Model selection"]
  Model_selection --> Few_shot_Gemini_translation_prompt
  Search_results --> A_B_testing["A/B testing"]
  A_B_testing --> Talabat_and_Hungerstation_rollout["Talabat and Hungerstation rollout"]

Components

Data flow

A user search query enters the multilingual search system. The few-shot Gemini translation prompt handles dialect, typo, transliteration, and regional context; the system runs the query three times and selects the most frequent translation. Elastic Search then retrieves using the original query and available translations, while Vector Search uses LLM-generated translations in its enriched training data to retrieve semantically similar products. Evaluation happens both offline, with local users validating translations for the top 1,000 queries from nine Arabic-speaking countries, and online through A/B tests at the search-query level.

Why this architecture

The architecture keeps mature search systems in place and uses LLM translation as an augmentation layer. Gemini was chosen over ChatGPT-Turbo because it was cheaper with acceptable accuracy, few-shot prompting avoided expensive fine-tuning, majority voting reduced unreliable outputs, and the translations improved both Elastic Search recall and vector-search semantic equivalence.

Models used: Gemini · ChatGPT-Turbo · Google Translate · DeepL · ChatGPT for sample query generation

Scale

• Delivery Hero operates in over 68 countries across four continents.
  “Delivery Hero operates in over 68 countries across four continents, focusing on food and groceries.”article, stated
• Search is served centrally from Berlin to all 68 countries.
  “The search solution is served from Berlin to all 68 countries, overcoming the challenges of multilingual data”article, stated
• The model comparison used 100 popular grocery-related Arabic search queries.
  “we compared Google Translate, DeepL, ChatGPT-Turbo, and Gemini (our preferred translation model) on 100 popular grocery-related search queries.”article, stated
• ChatGPT-Turbo scored 86% accuracy in the comparison.
  “While ChatGPT-Turbo provides the most accurate translations (86% accuracy), it is also more expensive.”article, stated
• Gemini delivered about 82% accuracy.
  “On the other hand, Gemini delivered about 82% accuracy, which is a solid performance with a better balance of cost and efficiency”article, stated
• Commercial translation tools generally performed under 80% accuracy.
  “Commercial translation tools, such as Google Translate and DeepL, generally perform under 80% accuracy”article, stated
• Prompt parameters used temperature 0, top-k 1, top-p 0, and max output tokens 100.

  "Temperature: Set to 0 to produce deterministic, non-random responses.

• Top-k and Top-p values: Set to 1 and 0"

— article, stated

• The team translated the top 1,000 queries from nine Arabic-speaking countries.
  “We translated the top 1,000 most popular queries from nine Arabic-speaking countries using a few-shot learning approach”article, stated
• Restaurant-related translations exceeded 90% accuracy.
  “The results showed over 90% accuracy for restaurant-related translations.”article, stated
• Majority voting runs each query through the model three times.
  “we ran each query through the model three times and selected the most frequent response as the final translation.”article, stated

Alternatives considered (roads not taken)

Traditional machine translation

Rejected because: It frequently missed intent, typos, dialects, and context-specific meaning.

ChatGPT-Turbo for production translation

Rejected because: It was the most accurate in the test but too expensive relative to Gemini.

Fine-tune a model

Rejected because: Fine-tuning was not feasible with a large/diverse dataset and limited labeled data.

Add many few-shot examples

Rejected because: Too many examples caused hallucination and unnecessary token cost.

Replace search with LLM translation alone

Rejected because: The production system uses LLM translations to enhance existing Elastic Search and Vector Search pathways.

Patterns

rag-retrieval (retrieval)

LLM-generated translations expand search recall by feeding both Elastic Search and vector-search retrieval.

retrieval-ranking (retrieval)

The hybrid system mixes Elastic Search and vector-search results so equivalent queries retrieve similar products.

prompt-chaining (workflow-orchestration)

Prompt examples, deterministic parameters, LLM-assisted prompt refinement, and majority voting form a multi-step translation workflow before search.

eval-driven-development (quality)

Delivery Hero compared translation tools offline, used local evaluators for top queries, and ran query-level A/B tests before rollout.

data-quality-loop (data)

The team generated and curated prompt/evaluation examples for Arabic dialects, misspellings, transliterations, and local food terms.

latency-optimization (operations)

The prompt limits output tokens and example count to control token costs and response overhead.

Tradeoffs

Model choice

Chose Gemini over ChatGPT-Turbo.

Rationale: ChatGPT-Turbo was more accurate in the sample, but Gemini had a better cost/performance balance for production search.

Few-shot prompting

Chose small example set over fine-tuning on a large diverse dataset.

Rationale: Few-shot learning avoided infeasible fine-tuning while still guiding dialect, transliteration, and typo handling.

Reliability mechanism

Chose majority voting over three model calls over single translation call.

Rationale: The extra calls increase cost but filter out outlier translations and improve confidence.

Search integration

Chose hybrid Elastic Search plus Vector Search over LLM-only search replacement.

Rationale: Translations improve both the existing lexical retrieval path and the vector semantic path.

Gotchas (what bit them)

The same written query can mean different foods in different dialects.

Root cause: Regional Arabic variants map identical strings to different concepts such as bread versus rice.

Resolution: Use context-aware LLM translations that incorporate regional dialect information.

Transliteration and abbreviations break literal search.

Root cause: Users write non-Latin names phonetically or use local shortcuts.

Resolution: Include transliteration edge cases in the few-shot prompt and LLM translation workflow.

Few-shot prompts can become too large and less reliable.

Root cause: Too many examples caused hallucination and increased token count.

Resolution: Use only a small number of representative edge cases.

LLM translations can be creative but unreliable.

Root cause: Generative models sometimes produce outlier translations despite deterministic prompt tuning.

Resolution: Run each query three times and choose the most frequent translation.

Translation quality must be judged by local meaning, not just dictionary equivalence.

Root cause: User intent depends on dialect, market, food domain, and local spelling conventions.

Resolution: Have local users evaluate translations from top queries in Arabic-speaking countries.

Evaluation

Delivery Hero compared Google Translate, DeepL, ChatGPT-Turbo, and Gemini on 100 popular grocery-related queries with correct translations from local Arabic speakers. It then translated the top 1,000 most popular queries from nine Arabic-speaking countries, had local users evaluate accuracy, and ran A/B tests at the search-query level before rolling out to Talabat and Hungerstation.

Metrics: 100 popular grocery-related search queries for initial comparison · ChatGPT-Turbo: 86% accuracy · Gemini: about 82% accuracy · commercial translation tools: generally under 80% accuracy · top 1,000 queries from nine Arabic-speaking countries · over 90% restaurant-translation accuracy · positive A/B test movement in user engagement

Outcomes

• Gemini was selected for production because it balanced cost and accuracy better than ChatGPT-Turbo.
• Few-shot LLM translations reached over 90% accuracy for restaurant-related translations.
• Search-query-level A/B tests showed positive user-engagement improvements.
• The system was rolled out for Talabat and Hungerstation.
• Translations improved both Elastic Search recall and vector-search synonym/dialect handling.

The article gives strong architectural detail but limited quantitative A/B business metrics. The reported success should be treated as company-reported, with the key reusable pattern being LLM translation as an augmentation layer for hybrid search rather than a replacement for search infrastructure.