Fine-tuning and Scaling LLMs for Search Relevance Prediction

faire · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: wholesale marketplace search relevance

Faire needed a scalable way to measure semantic relevance for marketplace search results because manual monthly labeling was too slow and expensive, while the existing GPT-based relevance labeling path was constrained by provider API cost as search personalization increased query-product variation.

Constraints:

semantic relevance had to follow a definition that human labelers could consistently agree on
manual relevance snapshots were expensive and delayed by about a month
the deployed GPT labeling path was limited by external API costs as labeling volume grew
the new model had to score tens of millions of query-product pairs daily for offline measurement
modeling had to handle vague retailer queries, sparse behavioral ground truth, personalization effects, and marketplace-specific relevance definitions

Architecture

Faire defined search relevance with the ESCI framework, used human-labeled query-product pairs as ground truth, first deployed a fine-tuned GPT text-completion classifier for daily relevance snapshots, then moved to self-hosted fine-tuned Llama models. The production-oriented Llama3-8B path uses LoRA fine-tuning on labeled datasets up to 250k examples, DeepSpeed training, 8-bit quantized batch inference on A100 GPUs, and horizontal GPU scaling to produce tens of millions of relevance predictions per day for search measurement and downstream analysis.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  esci-relevance-taxonomy[ESCI relevance taxonomy]
  human-labeling-process[Human labeling process]
  training-and-evaluation-datasets[Training and evaluation datasets]
  fine-tuning-pipeline[Fine-tuning pipeline]
  model-evaluation[Model evaluation]
  fine-tuned-llama-relevance-model[Fine-tuned Llama relevance model]
  gpu-batch-inference-cluster[GPU batch inference cluster]
  search-relevance-analytics[Search relevance analytics]
  esci-relevance-taxonomy --> human-labeling-process
  human-labeling-process --> training-and-evaluation-datasets
  training-and-evaluation-datasets --> fine-tuning-pipeline
  training-and-evaluation-datasets --> model-evaluation
  fine-tuning-pipeline --> fine-tuned-llama-relevance-model
  fine-tuned-llama-relevance-model --> model-evaluation
  fine-tuned-llama-relevance-model --> gpu-batch-inference-cluster
  gpu-batch-inference-cluster --> search-relevance-analytics

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`ESCI relevance taxonomy`	Defines Exact, Substitute, Complement, and Irrelevant labels so search relevance can be judged consistently.	`Human labeling process`, `Fine-tuned Llama relevance model`
`Human labeling process`	Produces ground-truth relevance labels and evolves guidelines for ambiguous marketplace search cases.	`ESCI relevance taxonomy`, `Training and evaluation datasets`
`Training and evaluation datasets`	Holds labeled query-product examples in small, medium, large, and hold-out splits for model training and comparison.	`Fine-tuning pipeline`, `Model evaluation`
`Fine-tuning pipeline`	Uses LoRA, DeepSpeed, multi-GPU training, batching, and checkpointing choices to train open-source Llama models.	`Training and evaluation datasets`, `Fine-tuned Llama relevance model`
`Model evaluation`	Compares candidate models using Krippendorff's Alpha, F1, and hold-out relevance labels.	`Training and evaluation datasets`, `Fine-tuned Llama relevance model`
`Fine-tuned Llama relevance model`	Classifies query-product relevance with Faire's domain definition at lower serving cost than external LLM APIs.	`GPU batch inference cluster`, `Model evaluation`
`GPU batch inference cluster`	Runs quantized batch inference with DeepSpeed and horizontal GPU scaling for daily relevance prediction volume.	`Fine-tuned Llama relevance model`, `Search relevance analytics`
`Search relevance analytics`	Consumes predicted relevance labels for offline retrieval analysis, personalization measurement, experiments, and ranker tradeoff exploration.	`GPU batch inference cluster`

Data flow

Human labelers create ESCI labels for query-product pairs, including query text, product name and description, brand, and categorical product type. Faire trains and evaluates fine-tuned Llama relevance classifiers against a hold-out set, chooses the best Llama3-8B model, hosts it on a GPU cluster, and runs batch predictions for new search sessions so relevance labels can feed offline retrieval analysis, personalization measurement, experiment contribution analysis, and ranker tradeoff exploration.

“In the prompt, we concatenated the search query text with product information (including the name and description of the product), the name of the brand offering the product, and the categorical type of product.”article, stated

“The selected Llama3–8b model is hosted on our GPU cluster to create batch predictions of relevance for new search sessions.”article, stated

Why this architecture

Faire chose a self-hosted fine-tuned open-source model because the relevance task was domain-specific, prompt engineering underperformed, external GPT labeling was cost-limited, and search needed daily high-throughput measurements that could scale to tens of millions of predictions.

“Prompt engineering alone cannot fully capture semantic search relevance on Faire. LLM fine-tuning is needed for models to understand our definition of relevance.”article, stated

“We needed a way to increase the throughput of relevance labeling at a reduced cost and at equal or better performance.”article, stated

Models used: Llama3-8B · Llama2-7B · Llama2-13B · fine-tuned GPT model

Scale

Faire is a global wholesale marketplace connecting hundreds of thousands of brands and retailers.
“As a global wholesale marketplace, Faire connects hundreds of thousands of independent brands and retailers all over the world.”article, stated
The initial deployed GPT relevance model labeled about 300k query-product pairs per hour.
“We reached 0.56 Krippendorff’s Alpha and could label ~300k query product pairs per hour at the time we deployed this to production.”article, stated
The Llama training comparisons used datasets of 11k, 50k, and 250k examples with a hold-out set of about 5k records.
“We also tested three datasets with different sizes: Small (11k), Medium (50k) and Large (250k).”article, stated
The largest Llama2-13B training run took about five hours on the large dataset.
“The largest model Llama2–13b took about five hours to complete training on the Large dataset.”article, stated
The self-hosted inference setup reached 70 million predictions per day using 16 GPUs for backfill.
“By doing so, we were able to reach 70 million predictions per day using 16 GPUs when we backfilled our labels at scale.”article, stated

Alternatives considered (roads not taken)

monthly human relevance labeling

Rejected because: it was essential for bootstrapping and guideline iteration, but too expensive and delayed for actionable measurement.

“Although human labeling was essential at the start, it was expensive and not immediately actionable due to the one month delay between our measurement and available labels.”article, stated

zero-shot prompt engineering

Rejected because: basic prompting did not capture Faire's relevance definition well enough.

“We briefly explored prompt engineering and found that basic prompt engineering with zero-shot prediction was not performant in predicting our definition of semantic search relevance.”article, stated

continue using the external fine-tuned GPT solution

Rejected because: it made relevance actionable, but API costs limited the volume of labels Faire could obtain.

“Our fine-tuned GPT solution made our relevance problem more actionable, but the amount of labels we could obtain was still primarily limited by the costs of using the API offered by the LLM provider.”article, stated

train much larger open-source models by default

Rejected because: Faire hypothesized the task was specific enough to avoid hundreds-of-billions-parameter models and found Llama3-8B competitive with Llama2-13B after more data.

“A key hypothesis we had was that despite the nuances of search relevance understanding, it is a specific language understanding problem that may not require a huge model with hundreds of billions of parameters.”article, stated

Patterns

`fine-tuning-adaptation` (model-operations)

Faire fine-tuned smaller open-source Llama models with LoRA so the classifier learned Faire's domain-specific semantic relevance definition.

“We froze the weights from the underlying model and employed Parameter Efficient Fine-Tuning using the LoRA adapter.”article, stated

`eval-driven-development` (evaluation)

Candidate GPT and Llama relevance models were compared on hold-out labels with Krippendorff's Alpha and F1 to decide whether the model was usable for measurement.

“We measured performance with Krippendorff’s Alpha, which intuitively is the agreement between our fine-tuned model and the labels for all four relevance categories.”article, stated

`latency-optimization` (serving)

The self-hosted inference system used quantization, batching, DeepSpeed, and horizontal GPU scaling to make high-volume relevance prediction practical.

"we set up inference to maximize throughput in the following way:

Quantize the model to 8 bit
Run batches on a single A100 GPU
Improve inference speed using DeepSpeed"

— article, stated

`human-in-the-loop-gate` (human-oversight)

Human labelers and quality audits established the relevance definition and ground-truth labels before LLMs were used to scale measurement.

“We built out several iterations of decision trees to reach > 90% agreement amongst labelers and our quality audits.”article, stated

Tradeoffs

Use an explicit ESCI taxonomy and human labels before model scaling.

Chose structured human-guideline bootstrapping over inferring relevance directly from behavioral engagement.

Rationale: Behavioral signals were sparse and confounded, while a shared relevance definition could be audited and taught to models.

“We need a clear definition for relevance that can be objectively agreed upon amongst different people before we even begin trying to model the problem.”article, stated

Move from external GPT labeling to self-hosted fine-tuned Llama models.

Chose open-source fine-tuned Llama hosted on Faire's GPU cluster over continuing to rely primarily on a provider-hosted GPT labeling API.

Rationale: Self-hosting lowered marginal inference cost and allowed higher-volume daily relevance measurement.

“This approach considerably lowers training and inference cost compared to external LLM providers.”article, stated

Use LoRA on smaller Llama models.

Chose parameter-efficient fine-tuning of 7B, 8B, and 13B models over full fine-tuning or defaulting to very large models.

Rationale: The models fit on A100s, were faster to prototype, and required optimizing only about 4% of base parameters.

“One benefit of working with these models is that they fit into the memory of an A100 GPU card and are fast to prototype and iterate on.”article, stated

Keep the current relevance model mostly offline rather than immediately serving it in real time.

Chose offline batch measurement and analytics over real-time search-result relevance changes.

Rationale: The current use was measurement-heavy; real-time impact would require an additional low-cost, low-latency serving path.

“The current use of relevance is mainly offline. Deploying the model in a real-time setting or distilling information with smaller models will provide opportunities to change the relevance of search results shown to users.”article, stated

Gotchas (what bit them)

Semantic relevance could not be treated as an objective click-derived label.

Root cause: Queries were ambiguous, user engagement was sparse, and relevance, desirability, and personalization were entangled in behavior data.

Resolution: Faire defined an explicit ESCI relevance taxonomy and created human labeling guidelines before scaling model-based relevance prediction.

“Historical engagements such as clicks, carts, and orders are sparse and it’s difficult to infer what a lack of engagement means since many things are never seen or never engaged with.”article, stated

Prompt engineering failed to capture Faire's domain-specific relevance definition.

Root cause: Search relevance depended on marketplace-specific labels and edge-case decisions rather than generic language understanding alone.

Resolution: The team fine-tuned models on labeled query-product examples and found fine-tuning necessary for their definition of relevance.

“The fine-tuned GPT model has almost 2x accuracy compared to prompt engineering.”article, stated

The original daily GPT labeling path was still bottlenecked by external API economics.

Root cause: Personalized retrieval increased the number and variation of query-product pairs that needed relevance measurement.

Resolution: Faire shifted to self-hosted open-source Llama models and optimized GPU inference throughput.

“personalized retrieval sources, which increased the variation of query-product pairs being shown to different retailers and made our existing measurement solutions blind to the aggregate effect”article, stated

Model performance was governed more by labeled data composition than by model family alone.

Root cause: The relevance definition had enough nuance that larger or newer base models did not dominate without sufficient ground-truth data.

Resolution: Faire trained across small, medium, and large datasets and concluded that dataset size, quality, and composition had the strongest effect.

“Increasing the size and composition of the labeled dataset is the most important factor in improving the performance.”article, stated

Evaluation

Faire evaluated relevance classifiers against held-out human relevance labels, primarily using Krippendorff's Alpha for agreement across all four ESCI categories and F1 for exact versus non-exact relevance checks.

Metrics: Krippendorff's Alpha · F1 score for exact vs not exact · 0.56 Krippendorff's Alpha for the deployed fine-tuned GPT model · 28% relative improvement over the existing fine-tuned GPT model for Llama3-8B on the large dataset · labeling throughput of about 300k pairs per hour for the GPT path · 70 million predictions per day on 16 GPUs for the Llama backfill path

“We also referenced other metrics like F1 Score for exact vs not exact to gauge whether our models were applicable for measurement.”article, stated

“our best performing model, Llama3–8b trained on the Large dataset, is able to achieve 28% improvement on Krippendorff’s Alpha from our existing production model.”article, stated

Outcomes

The fine-tuned GPT path reached 0.56 Krippendorff's Alpha and about 300k query-product pair labels per hour.
The best Llama3-8B model improved Krippendorff's Alpha by 28% over the existing fine-tuned GPT model.
Llama3-8B trained on the large dataset matched Llama2-13B closely and beat Llama2-7B by about 1.4% in that comparison.
The self-hosted inference setup reached 70 million predictions per day on 16 GPUs for label backfill.
Scaled relevance predictions unlocked offline retrieval analysis, personalization measurement, experiment contribution analysis, and ranker Pareto-frontier exploration.

Faire moved from monthly human relevance snapshots to daily and then high-throughput batch relevance prediction, with a self-hosted Llama3-8B model supporting tens of millions of predictions per day across a global marketplace.

“This allowed us to start measuring relevance on a daily basis from limited samples.”article, stated

“These scaled out predictions unlocked many downstream use cases including offline retrieval analysis, measurement of personalization, measurable contribution of experiments towards relevance”article, stated