Automated Search Quality Evaluation Using LLMs for Typeahead Suggestions

linkedin · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Search quality evaluation

LinkedIn needed to evaluate typeahead suggestion quality for a personalized, multi-vertical search surface at over-billion-member scale without waiting days or weeks for human evaluation cycles.

Constraints:

• typeahead suggestions blend people, companies, jobs, courses, posts, and plain text
• the same query can have different relevance for different members
• manual evaluations with multiple humans are slow
• the evaluator needs a representative golden test set from platform traffic
• ranked suggestions require top-position-sensitive quality metrics

Architecture

LinkedIn built a GenAI Typeahead Quality Evaluator around OpenAI GPT served through Azure. The team first created measurement guidelines for each suggestion type, simplified judgments to binary high/low quality, and built a golden test set of query/member-id pairs sampled from clicked, bypass, abandoned, and random sessions across intent categories and member-life-cycle segments. Prompt templates include identity, task guidelines, optional few-shot examples, input, and output sections tailored by suggestion type. The batch evaluation pipeline runs experiment configs against the golden set, collects typeahead responses, generates prompts for GPT-3.5 Turbo, batch-calls the API for per-suggestion quality scores and reasoning, and computes TyahQuality@1/3/5/10 metrics.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  golden-test-set[Golden test set] --> typeahead-backend[Typeahead backend]
  typeahead-backend --> prompt-generation-layer[Prompt generation layer]
  prompt-generation-layer --> azure-gpt-evaluator[Azure GPT evaluator]
  azure-gpt-evaluator --> quality-score-postprocessor[Quality score postprocessor]
  quality-score-postprocessor --> experiment-reporting[Experiment reporting]

Components

Data flow

Experiment configurations generate typeahead requests on the golden test set. LinkedIn collects responses, builds result-type-specific prompts, sends them to GPT through Azure, post-processes binary scores and reasoning, and aggregates ranked quality scores across sessions to measure experiment impact.

Why this architecture

LinkedIn needed automated evaluation, not another serving model, because typeahead quality was already bottlenecked by human review. Binary scoring reduces ambiguity, while tailored prompt templates reflect the fact that people, jobs, entities, and plain text require different notions of quality. Query/member-id golden sets preserve personalization context. Ranked metrics emphasize the top suggestions members see first.

Models used: OpenAI GPT model served through Azure · GPT 3.5 Turbo

Scale

• LinkedIn serves more than 1 billion members.
  “For more than 1 billion members on LinkedIn, the global search bar is a tool for looking up jobs, company pages, member profiles and posts, LinkedIn Learning courses, and much more.”article, stated
• The golden set samples 200 queries from each major search intent category.
  “We sample 200 queries from each of the search intent categories covering People, Company, Product, Job, Skill Queries”article, stated
• The golden set also samples 200 queries from bypass and random sessions.
  “We sampled 200 queries from both bypass and random sessions. 3. Identifies frequent members by member life cycles (MLC): When sampling member ids for the golden test set, we leverage the Member Life Cycle (MLC) data that represents the frequency and consistency of a member's visits.”article, stated
• Manual evaluation took days or weeks, while automated evaluation completes in a few hours.
  “Traditional manual search quality evaluations, often involving multiple human evaluators, can take days or even weeks”article, stated

  “In contrast, the GenAI automated search quality evaluator accomplishes the task in just a few hours. This rapid turnaround enables a more immediate assessment of a project's impact, allowing the project to maintain momentum and progress quickly.”article, stated

Alternatives considered (roads not taken)

Human-only search quality evaluation

Rejected because: Manual review could not sustain the platform's growth and iteration needs.

One universal prompt for all suggestion types

Rejected because: Different result types require different identity, guidelines, input fields, and examples.

Evaluate only clicked typeahead sessions

Rejected because: Bypass and abandoned sessions are crucial for understanding typeahead quality problems.

Patterns

llm-as-judge (evaluation)

GPT scores typeahead suggestions as high or low quality and explains the score.

eval-driven-development (evaluation)

New typeahead relevance experiments are run through the automated evaluator to measure quality-score deltas quickly.

prompt-versioning (prompt-operations)

The team iterates prompt templates after comparing GPT judgments with human evaluations.

structured-output-contract (output-control)

Prompts require reasoning plus a binary score so downstream metrics can be computed consistently.

Tradeoffs

Use binary high/low scoring

Chose 1 or 0 suggestion quality over multi-point subjective scales.

Rationale: Binary labels reduce gray areas in a subjective personalized evaluation task.

Sample traffic into a golden test set

Chose pre-defined query/member-id pairs from platform traffic over ad hoc evaluator prompts.

Rationale: A fixed representative set allows repeatable experiment comparisons across the typeahead experience.

Use ranked quality metrics

Chose TyahQuality@1/3/5/10 over unweighted average over all suggestions only.

Rationale: Top-ranked suggestions matter more because they are more visible to members.

Gotchas (what bit them)

Typeahead evaluation criteria differ by vertical.

Root cause: People, companies, jobs, skills, and plain text suggestions represent different user intents and data structures.

Resolution: Create specialized prompt templates and examples for each result type.

Personalization makes universal quality labels subjective.

Root cause: The same query can yield different correct suggestions for different members.

Resolution: Build the golden set from query/member-id pairs and include member lifecycle sampling.

Automated evaluator accuracy required repeated calibration.

Root cause: GPT outputs did not initially align perfectly with human judgments across ambiguous suggestion types.

Resolution: Run several cycles of cross-evaluation and prompt iteration using human-GPT discrepancies.

Evaluation

LinkedIn evaluates typeahead experiments by running GPT-scored suggestions over a golden test set and calculating ranked TyahQuality metrics before and after changes.

Metrics: TyahQuality1 · TyahQuality3 · TyahQuality5 · TyahQuality10 · low-quality suggestion reduction · evaluation turnaround time

Outcomes

• One representative experiment improved TyahQuality10 from 66.70% to 73.50%.
• The improvement corresponds to a 6.8% absolute lift and 20% reduction in low-quality suggestions.
• Automated evaluation reduced review time from days or weeks to a few hours.
• The evaluator increased visibility into ongoing typeahead quality.

The system evaluates typeahead search for LinkedIn's more than 1 billion members using representative golden-set sampling.

"TyahQuality10

66.70%

73.50%"