Scaling GenAI Applications with vLLM for High-Throughput LLM Serving

linkedin · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM inference serving at LinkedIn scale

LinkedIn needed high-throughput, low-latency LLM serving for dozens of real-time and offline GenAI applications.

Constraints:

• Use cases include both online interactive products and large-scale batch processing.
• Some workloads require low p95 latency under thousands of QPS.
• Long prompts and high output-token counts create GPU memory and throughput pressure.

Architecture

LinkedIn adopted vLLM as the inference engine for more than 50 GenAI use cases. The serving stack evolved from offline low-QPS mode to tuned v0.7/v1 engines and then a modular OpenAI-compatible architecture that decouples LinkedIn's custom gRPC server from the vLLM engine client.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["LinkedIn GenAI applications"] --> B["Custom gRPC server"]
  B --> C["OpenAI-compatible API adapter"]
  C --> D["vLLM engine client"]
  D --> E["vLLM engine"]
  E --> F["GPU serving hosts"]

Components

Data flow

LinkedIn applications send requests to a custom gRPC serving layer, which maps requests into OpenAI-compatible API calls and forwards them to the vLLM engine. vLLM uses efficient memory management and prefix caching for online products such as Hiring Assistant and AI Job Search and for offline embedding/document-processing workloads.

Why this architecture

vLLM fits LinkedIn's need for efficient GPU memory use, prefix reuse, real-time and offline support, and modular serving extensibility without replicating vLLM server logic.

Models used: LLMs served by vLLM · embedding generation models · classification/explanation LLMs

Scale

• vLLM supports more than 50 GenAI use cases at LinkedIn.
  “Today, vLLM supports more than 50 different GenAI use cases at LinkedIn”article, stated
• LinkedIn runs vLLM on thousands of hosts.
  “It also runs on thousands of hosts across our platform”article, stated
• Tuning num-scheduler-steps yielded about 10% TPS improvement at 1.5 QPS.
  “This yielded a ~10% TPS improvement at 1.5 QPS—without any custom kernels or engine modifications”article, stated
• The v1/v0.7 tuned engines reached about 1245 tokens/sec and saved over 60 GPUs for one workload.
  “Both reached ~1245 tokens/sec under saturation (~10% higher than v0.6.1.post2), saving over 60 GPUs for this workload.”article, stated
• AI Job Search had to meet sub-600ms p95 latency under thousands of QPS.
  “meeting stringent low-latency demands (<600ms p95 under thousands of QPS) for a global user base”article, stated

Alternatives considered (roads not taken)

Offline mode only

Rejected because: Offline vLLM validated performance and accuracy but was suited mainly for low-QPS workloads.

Tightly coupled server and engine

Rejected because: It limited independent component evolution and increased maintenance burden.

Traditional NER models for AI Job Search query interpretation

Rejected because: NER models were brittle and costly to maintain as language and job taxonomies changed.

Patterns

latency-optimization (serving)

LinkedIn tuned vLLM scheduling and CUDA graph coverage to improve TPS, TPOT, and low-latency serving.

semantic-caching (serving)

vLLM prefix caching reuses computation for shared prompt prefixes in Hiring Assistant workloads.

structured-output-contract (application)

AI Job Search turns free-form queries into structured interpretations and facet suggestions.

cost-aware-routing (serving)

GPU savings and efficient serving choices reduce the compute footprint of high-volume workloads.

Tradeoffs

Future serving engine

Chose vLLM v1 engine over staying on v0.7.0 with tuning.

Rationale: Both performed similarly under saturation, but v1 offered future readiness, simpler config, and better high-QPS scheduling efficiency.

Serving API shape

Chose modular OpenAI-compatible architecture over custom tightly coupled serving logic.

Rationale: The adapter reduces maintenance and lets LinkedIn support advanced features without duplicating vLLM server behavior.

Gotchas (what bit them)

Offline inference mode did not support higher-concurrency serving.

Root cause: The initial LLM-class/engine.step deployment had limited concurrency.

Resolution: Evolve through vLLM engine versions, scheduler tuning, and modular serving architecture.

Tightly coupled serving layers slow feature evolution.

Root cause: Server and engine changes could not evolve independently.

Resolution: Decouple the gRPC server from the vLLM engine through an OpenAI-compatible API layer.

Hiring Assistant prompts are token-heavy.

Root cause: Classification plus explanations require nearly 1000 output tokens per candidate, with hundreds to thousands of candidates per hirer.

Resolution: Use prefix caching and vLLM serving optimizations to reduce prefill latency and GPU load.

Evaluation

LinkedIn evaluates vLLM through TPS, tokens/sec under saturation, GPU savings, TPOT, p95 latency, QPS, and production use-case support.

Metrics: TPS improvement · tokens per second · GPU savings · TPOT · p95 latency · QPS · use cases supported

Outcomes

• vLLM supports more than 50 LinkedIn GenAI use cases.
• The tuned engine reached around 1245 tokens/sec and saved more than 60 GPUs for one workload.
• Scheduler tuning produced about 10% TPS improvement at 1.5 QPS.
• CUDA graph work improved TPOT by about 7%.

The system runs across thousands of hosts and serves both online and offline applications.