Scaling LLM-Based Ranking Systems with Prefill-Only Optimization

linkedin · Tech · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM ranking inference

LinkedIn needed to serve LLM-based ranking for AI Job Search and AI People Search, scoring many candidates per query under strict P99 latency constraints.

Constraints:

Ranking workloads require prefill-only scoring rather than autoregressive generation.
P99 latency had to stay within a few hundred milliseconds under load.
CPU-side serialization, decode-loop overhead, prefix recomputation, and Python runtime behavior all limited throughput.

Architecture

LinkedIn adapted SGLang for prefill-only ranking through staged optimization: batch tokenization, preserving batch boundaries, a scoring-only fast path, reduced CPU/GPU synchronization, in-batch prefix caching, multi-process gRPC serving, and multi-process scheduling.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Ranking request"] --> B["Batch tokenizer"]
  B --> C["Batch send transport"]
  C --> D["SGLang scheduler"]
  D --> E["Scoring-only fast path"]
  E --> F["In-batch prefix cache"]
  G["Python gRPC server"] --> H["Multi-process gRPC servicers"]
  H --> I["Multi-process scheduler workers"]
  I --> D

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Ranking request`	Query/member context plus many candidate-item prompts to score.	`Batch tokenizer`
`Batch tokenizer`	Tokenizes many prompts concurrently instead of serially.	`Ranking request`, `Batch send transport`
`Batch send transport`	Preserves tokenized batch structure over ZMQ to the scheduler.	`Batch tokenizer`, `SGLang scheduler`
`SGLang scheduler`	Builds prefill batches and dispatches inference work.	`Batch send transport`, `Scoring-only fast path`, `Multi-process scheduler workers`
`Scoring-only fast path`	Runs prefill and final-logit scoring without decode, sampling, or unnecessary logprob extraction.	`SGLang scheduler`, `In-batch prefix cache`
`In-batch prefix cache`	Reuses shared query-prefix KV within a single forward pass.	`Scoring-only fast path`
`Python gRPC server`	Handles request parsing, access validation, metrics, logging, and tracing.	`Multi-process gRPC servicers`
`Multi-process gRPC servicers`	Parallelizes request preprocessing and network I/O beyond one Python process.	`Python gRPC server`, `Multi-process scheduler workers`
`Multi-process scheduler workers`	Parallel scheduler processes per GPU to remove CPU scheduling bottlenecks.	`Multi-process gRPC servicers`, `SGLang scheduler`

Data flow

A ranking request contains a query/member context prefix plus many item-specific suffixes. Prompts are tokenized in batch, transmitted as one batch to the scheduler, scored through a prefill-only path that returns final-token logits without decode/sampling, reuses shared prefix KV within the batch, and passes through multi-process gRPC and SGLang schedulers to keep CPU and GPU work pipelined.

3 supporting quotes

“All prompts for a given query share a common prefix: system instructions, query text, and member context. Only the item-specific suffix differs.”article, stated

“This allowed the scheduler to see the full batch and execute it in a single GPU prefill pass.”article, stated

“This execution path extracts only the final token’s logits required for relevance scoring”article, stated

Why this architecture

LinkedIn optimized for the real critical path of ranking rather than generic text generation, preserving semantic equivalence while removing decode, CPU serialization, redundant prefix work, and Python runtime bottlenecks.

3 supporting quotes

“ranking models simply score many options against the same query and return the best matches.”article, stated

“why run a full decode/sampling loop when ranking only needs one forward pass through the prompt”article, stated

“Optimization is only as good as how well you profile and understand the workload characteristics”article, stated

Models used: Embedding-0.6B · 0.6B decoder-only ranker · 0.4B pruned ranking model · 375M decoder-only ranker · SGLang

Scale

End-to-end P99 latency needed to stay within a few hundred milliseconds.
“End-to-end P99 latency must stay within a few hundred milliseconds under load.”article, stated
Batch tokenization reduced P99 latency from 4583 ms to 464 ms for an Embedding-0.6B workload.
“P99 latency dropped from 4583 ms to 464 ms — roughly 10× faster.”article, stated
Batch send reduced average latency from 70.39 ms to 41.12 ms per request.
“this change reduced average latency from 70.39 ms to 41.12 ms per request—a 41.5% reduction”article, stated
The scoring path reduced P99 latency from 6220 ms to 454 ms and increased throughput by about 25%.
“P99 latency dropped from 6220 ms to 454 ms (13.7×), while throughput increased by ~25%”article, stated
The final results were measured on H100 GPUs under P99 latency of at most 500 ms.
“All results are measured on H100 GPUs under a p99 latency ≤ 500 ms”article, stated

Alternatives considered (roads not taken)

Generic generation path for ranking

Rejected because: Decode and sampling work are unnecessary for prefill-only relevance scoring.

“the default execution path still entered the full decode and sampling loop, adding unnecessary CPU work and memory traffic.”article, stated

Rely on client-side batching alone

Rejected because: The tokenizer manager fragmented batch structure before the scheduler.

“although prompts were tokenized as a batch, the tokenizer manager transmitted them individually over the ZMQ socket.”article, stated

Multi-Item Scoring with item-aware masking

Rejected because: In-batch prefix caching preserved standard batched inputs and avoided specialized kernel-level masking.

“in-batch prefix caching operates at a higher level in the execution stack. It preserves standard batched inputs”article, stated

Patterns

`latency-optimization` (production-operations)

LinkedIn removed serial CPU work, decode overhead, synchronization gaps, redundant prefix computation, and Python bottlenecks to meet P99 latency targets.

“This was our first “aha” moment: CPU-side serialization can erase GPU gains before inference even starts.”article, stated

`semantic-caching` (inference)

In-batch prefix caching reuses shared query-prefix KV for all items in a ranking batch.

“In-batch prefix caching removes that overhead by reusing query-prefix KV within a single forward pass.”article, stated

`observability-tracing` (production-operations)

Profiling and gRPC observability exposed whether bottlenecks were in tokenization, GPU execution, scheduling, or Python runtime behavior.

“Profiling showed the next bottlenecks were no longer in CUDA kernels—but in Python runtime behavior around the engine.”article, stated

`structured-output-contract` (model-interface)

The specialized scoring API treats ranking as a first-class path that returns relevance logits instead of generated tokens.

“we also recognized that scoring deserved a first-class execution path rather than being treated as a special case of generation.”article, stated

Tradeoffs

Ranking execution path

Chose specialized prefill-only scoring path over standard generative decode path.

Rationale: Ranking only needs final-token logits, so skipping decode and sampling removes unnecessary overhead while preserving scoring semantics.

“runs identical prefill computation to generation, preserving accuracy, while skipping all token sampling, decode loops, and KV cache updates.”article, stated

Prefix amortization technique

Chose in-batch prefix caching over specialized Multi-Item Scoring kernels for this focus.

Rationale: In-batch prefix caching kept standard batch structure while reusing shared prefix KV.

“It preserves standard batched inputs, avoids concatenation, and does not require specialized kernel-level masking.”article, stated

Python serving architecture

Chose multi-process servicers and scheduler workers over single Python process handling preprocessing and scheduling.

Rationale: Multiprocessing scaled CPU-side work with available cores while keeping GPU execution fed.

“allowed CPU-side work to scale with available cores while keeping the GPU fully utilized through pipelined batch preparation and execution”article, stated

Gotchas (what bit them)

Tokenization serialization consumed hundreds of milliseconds before GPU work started.

Root cause: Prompts were tokenized one at a time on the CPU.

Resolution: Enable batch tokenization and dynamic async batch tokenization.

“tokenization was performed sequentially one prompt at a time. For large inputs—say, ~2k tokens across 100 items—this alone could consume hundreds of milliseconds”article, stated

Tokenized batches fragmented before reaching the scheduler.

Root cause: The tokenizer manager sent prompts individually over ZMQ.

Resolution: Introduce batch send so the scheduler executes the full batch in one GPU prefill pass.

“the original batch structure was lost, leading to fragmented execution (e.g., 12 + 38) and unnecessary extra forward passes”article, stated

Python runtime pauses became a P99 latency problem.

Root cause: Garbage collection and GIL contention paused request handling and preprocessing.

Resolution: Use multi-process gRPC serving and scheduler workers.

“Python’s generational garbage collector occasionally scanned large numbers of long-lived objects, causing 100–300 ms pauses every few seconds.”article, stated

Evaluation

LinkedIn evaluates with profiling, PyTorch traces, throughput/latency benchmarks, H100 P99 latency constraints, semantic equivalence to generation, and production relevance for AI Job Search and AI People Search.

Metrics: P99 latency · throughput · items per second per GPU · average latency · CPU/GPU synchronization gaps · GPU utilization

“Latency and throughput comparison before and after scoring-path optimizations on a 0.6B model with 300-token inputs at 100 QPS.”article, stated

“All results are measured on H100 GPUs under a p99 latency ≤ 500 ms, and reflect the cumulative effect of batching, scoring-only execution, shared-prefix amortization, and runtime/multiprocessing optimizations.”article, stated

Outcomes

Batch tokenization reduced P99 latency roughly 10x for an embedding workload.
Batch send reduced average latency by 41.5% for a 0.6B ranking workload.
The scoring path reduced P99 latency from 6220 ms to 454 ms and increased throughput by about 25%.
Multi-process serving delivered an additional roughly 40% throughput increase.
The optimizations power AI Job Search and AI People Search for millions of LinkedIn members.

The article is unusually specific about optimization stages and benchmark measurements, making this a strong latency-optimization case.

“In production benchmarks, this delivered an additional ~40% throughput increase beyond what a single Python process could sustain.”article, stated

“these advancements power AI Job Search and AI People Search to deliver state-of-the-art LLM ranking to millions of members.”article, stated