Scaling Search Query Understanding with LLMs: From POC to Production

yelp · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: local search query understanding

Yelp needed to use LLMs for search-query understanding tasks such as segmentation, spell correction, canonicalization, and review-highlight generation without paying GPT-4 latency and cost on every search.

Constraints:

• The work had to improve user intent understanding for production search, not just demonstrate an offline prompt.
• The architecture had to exploit Yelp's power-law query distribution so expensive LLM calls could be cached for common queries.
• Long-tail or never-before-seen queries still needed a lower-latency path.
• Offline quality was subjective enough that human review and online A/B testing were both required.

Architecture

Yelp prototyped query-understanding prompts with high-end LLMs, enriched prompts with business names or categories when useful, generated golden datasets from representative queries, fine-tuned smaller models such as GPT-4o-mini for offline precomputation, stored cached query-understanding signals in datastores, and used BERT/T5-style models for realtime long-tail query handling.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["User search query"] --> B["LLM prototyping workflow"]
  B --> C["RAG enrichment layer"]
  C --> D["Golden dataset builder"]
  B --> D
  D --> E["Fine-tuned GPT-4o-mini batch model"]
  E --> F["Query understanding datastore"]
  F --> G["Search ranking and review highlighting systems"]
  H["BERT/T5 realtime tail model"] --> G
  I["Offline and A/B evaluation"] --> B
  I --> D
  G --> I

Components

Data flow

A user query enters the query-understanding workflow; the team prototypes the task with a powerful GPT-4/o1-class prompt, optionally enriches the input with viewed business names or category predictions, uses high-end model outputs to create a golden dataset, fine-tunes a smaller model for batch precomputation on head queries, uploads quality-checked outputs into query-understanding datastores, and feeds downstream search ranking, segmentation, spell correction, and review-highlight systems.

Why this architecture

Yelp chose a staged architecture because powerful LLMs were useful for task discovery and label generation, while production search needed cached or smaller-model paths to control cost and latency across millions of queries.

Models used: GPT-4 · o1 · o1-mini · o1-preview · GPT-4o-mini · BERT · T5

Scale

• The production rollout precomputed signals for 95% of traffic.
  “we scaled to 95% of traffic by pre-computing snippet expansions for those queries using OpenAI’s batch calls.”article, stated
• Fine-tuned query-understanding models were used offline at tens-of-millions scale.
  “Fine tune a smaller model (GPT4o-mini) that we can run offline at the scale of tens of millions”article, stated
• The smaller-model path delivered up to 100x cost savings versus direct GPT-4 prompting.
  “we have seen up to a 100x savings in cost, compared to using a complex GPT-4 prompt directly.”article, stated
• Six segmentation classes were used after iteration.
  “After several iterations, we settled on six classes for query segmentation: topic, name, location, time, question, and none.”article, stated

Alternatives considered (roads not taken)

Run complex GPT-4 prompts directly in production search

Rejected because: Yelp moved to fine-tuned smaller models because direct complex prompting was too costly for production scale.

Apply LLMs to every query before proving value

Rejected because: The team first scoped whether an LLM fit the problem, then used proof-of-concept and online experiments before scaling.

Use only offline subjective judgment for review highlights

Rejected because: Subjective offline evaluation was insufficient, so Yelp cross-checked humans, quantitative methods, and online A/B tests.

Treat spell correction and segmentation as fully separate prompts

Rejected because: Yelp found a sufficiently powerful model could handle them together and added a meta tag for spell-corrected sections.

Patterns

rag-retrieval (retrieval)

Yelp enriched raw queries with business names or categories so the model could disambiguate names, topics, misspellings, and review-highlight phrases.

semantic-caching (serving)

The production strategy cached precomputed LLM outputs for frequent head queries and stored them in query-understanding datastores.

fine-tuning-adaptation (model-adaptation)

High-end model outputs became a golden dataset for fine-tuning smaller models that could run cheaply at Yelp's traffic scale.

eval-driven-development (evaluation)

Yelp used human-labeled offline datasets, specialized review by strong annotators, online A/B tests, and query-level metrics to decide what to teach or scale.

batch-offline-processing (serving)

Yelp shifted the expensive LLM work into offline batch precomputation for common queries and kept realtime serving for the tail.

Tradeoffs

Use head-query precomputation before broader realtime rollout.

Chose cached high-end LLM responses for frequent queries over directly serving high-end LLMs for all queries.

Rationale: Power-law traffic made head-query caching a low-latency, lower-cost way to prove value over a large traffic share.

Fine-tune smaller models for production scale.

Chose fine-tuned GPT-4o-mini and optional BERT/T5 tail models over keeping the complex GPT-4 prompt as the production path.

Rationale: The production tasks used short inputs/outputs, making smaller models cheaper and fast enough for search serving.

Combine spell correction and segmentation in one prompt.

Chose a joint prompt with a meta tag for corrected sections over separate spell-correction and segmentation passes.

Rationale: The tasks were conceptually related and a sufficiently powerful model could handle both.

Gotchas (what bit them)

Subjective review-highlight quality was hard to validate offline.

Root cause: Generated phrases needed product and qualitative judgment, not just lexical or automatic checks.

Resolution: Yelp cross-checked human annotators, used quantitative phrase-commonness checks, and then ran online A/B experiments.

Scaling LLM query understanding to the long tail faced cost and infrastructure pressure.

Root cause: A power-law query distribution means marginal traffic coverage can require disproportionately high investment.

Resolution: Yelp precomputed head-query signals, fine-tuned smaller models, and used faster realtime models for long-tail coverage.

Fine-tuning data quality could be worse than the expensive model's raw output.

Root cause: Representative GPT-4 samples can still contain mislabeled subsets that would propagate into the smaller production model.

Resolution: Yelp isolated likely mislabeled inputs and targeted them for human relabeling or removal before fine-tuning.

Evaluation

Yelp evaluated query-understanding outputs with human-labeled datasets, downstream search metrics, specialized human review, query-level metrics, and online A/B experiments.

Metrics: accuracy against human-labeled name-match and location-intent datasets · Session / Search CTR · query-level metric regressions versus status quo · traffic coverage for precomputed snippets

Outcomes

• Yelp scaled precomputed snippet expansion to 95% of traffic.
• Fine-tuned query-understanding models achieved up to 100x cost savings versus complex GPT-4 prompts.
• Review-highlight changes increased Session / Search CTR across platforms.

The deployment exploited the power-law query distribution: head-query batch computation covered most traffic, while realtime smaller models handled the remaining long tail.