JUDE: Large-Scale LLM-Based Embedding Generation for Job Recommendations

linkedin · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale job recommendation embeddings

LinkedIn needed higher-quality semantic representations for matching jobs, member profiles, and resumes at massive scale while keeping latency, cost, privacy, and operational complexity manageable. Standardized features, smaller ML models, rigid upstream pipelines, and taxonomies were not enough for the job recommendation and search ranking stack.

Constraints:

• LinkedIn serves over 1 billion registered members and tens of millions of active job postings
• job posters expect postings to be indexed and recommended immediately
• members expect profile or resume updates to affect recommendations and search results instantly
• LLM production deployment introduces high computational cost, complex deployment pipelines, and continuous domain adaptation needs
• embedding generation must cover jobs, member profiles, and member resumes
• 7B+ parameter models require memory, training, and serving optimizations
• features must reflect the same underlying state as database changes
• embeddings must support online ranking and offline model training with time-aware joins

Architecture

LinkedIn built JUDE, an end-to-end platform for generating and serving LLM-derived embeddings for job recommendations. The model side uses a shared base LLM with specialized prompt templates for job descriptions and member profiles, fine-tuned on relevance and engagement signals with LoRA, Flash Attention 2, Liger kernels, bfloat16, gradient accumulation/checkpointing, DeepSpeed ZeRO, and engineered losses. The serving side uses Kappa nearline inference: Kafka/Brooklin changelog streams feed Samza pipelines, which extract features, apply prompts, skip unchanged text, call GPU Model Serving over GRPC, and write embeddings to Venice for online ranking plus Kafka/HDFS for offline training. Batch Flyte/Spark/K8S/Ray pipelines bootstrap and backfill embeddings, and hashing/deduplication reduce inference volume.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Job_document_index["Job document index"] --> Embedding_based_retrieval["Embedding-based retrieval"]
  Embedding_based_retrieval --> Ranking_cascade["Ranking cascade"]
  Relevance_supervision_labels["Relevance supervision labels"] --> Shared_base_LLM["Shared base LLM"]
  Engagement_supervision_labels["Engagement supervision labels"] --> Shared_base_LLM
  Specialized_prompt_templates["Specialized prompt templates"] --> Shared_base_LLM
  Shared_base_LLM --> Fine_tuning_optimization_stack["Fine-tuning optimization stack"]
  Shared_base_LLM --> Loss_engineering_stack["Loss engineering stack"]
  Fine_tuning_optimization_stack --> Model_Cloud_serving["Model Cloud serving"]
  Model_Cloud_serving --> GRPC_model_serving_endpoints["GRPC model serving endpoints"]
  Kafka_Brooklin_changelog_streams["Kafka Brooklin changelog streams"] --> Samza_nearline_pipelines["Samza nearline pipelines"]
  Specialized_prompt_templates --> Samza_nearline_pipelines
  Samza_nearline_pipelines --> Change_detection_deduplication["Change detection deduplication"]
  Change_detection_deduplication --> GRPC_model_serving_endpoints
  GRPC_model_serving_endpoints --> Venice_online_embedding_store["Venice online embedding store"]
  GRPC_model_serving_endpoints --> Kafka_HDFS_offline_sinks["Kafka HDFS offline sinks"]
  Offline_batch_bootstrap["Offline batch bootstrap"] --> Venice_online_embedding_store
  Offline_batch_bootstrap --> Kafka_HDFS_offline_sinks
  Venice_online_embedding_store --> Embedding_based_retrieval
  Loss_engineering_stack --> JUDE_embeddings["JUDE embeddings"]
  JUDE_embeddings --> Ranking_cascade

Components

Data flow

Job postings, member profiles, and resumes emit separate Kafka/Brooklin changelog streams. Dedicated Samza real-time pipelines extract relevant text, apply the appropriate prompt, skip inference when text has not changed meaningfully, call the LLM Model Serving cluster over GRPC, and write generated embeddings to online Venice storage and offline Kafka/HDFS sinks. Before nearline inference starts, Flyte/Spark/K8S/Ray batch inference bootstraps new embedding versions and performs targeted backfills. The embeddings then serve as features in job recommendation and search L2 ranking models.

Why this architecture

JUDE uses fine-tuned LLM embeddings because job matching needs semantic understanding across jobs, profiles, and resumes, but production scale requires a nearline-first Kappa architecture, shared LLM/two-tower efficiency, prompt templates per entity type, aggressive training/inference optimizations, and separate online/offline sinks. This lets LinkedIn improve ranking quality while keeping embedding latency, cost, feature freshness, and maintenance manageable.

Models used: fine-tuned LLMs · 7B+ decoder-only models · encoder-decoder architectures · Mistral family models · cross-encoders · two-tower architecture · LoRA adapters · DeepSpeed ZeRO · Flash Attention 2

Scale

• LinkedIn has over 1 billion registered members.
  “The scale of LinkedIn's platform — with over 1 billion registered members and tens of millions of active job postings”article, stated
• The platform has tens of millions of active job postings.
  “The scale of LinkedIn's platform — with over 1 billion registered members and tens of millions of active job postings”article, stated
• JUDE handles embeddings for three entity types.
  “We generate embeddings for three key types of entities: job postings, member profiles, and member resumes.”article, stated
• Modern 7B LLM embedding inference remains under 300ms at p95.
  “For a modern 7B LLM, latency remains under 300ms at p95 quantile, which is quite acceptable for our application”article, stated
• Change detection reduces embedding inference cost up to 3x.
  “This simple optimization reduces the embedding inference cost up to 3x.”article, stated
• Hashing and deduplication reduce inference volume by about 6x compared with naive database change tracking.
  “This intelligent change detection reduces inference volume by approximately 6x compared to naive database change tracking.”article, stated
• JUDE was trained with multi-node, multi-GPU distributed training on Nvidia H100 cards.
  “Due to the scale we employ multi-node multi-GPU distributed training on Nvidia H100 graphical cards”article, stated

Alternatives considered (roads not taken)

Keep standardized features, smaller ML models, taxonomies, and rigid pipelines

Rejected because: JUDE replaced standardized features with LLM-derived representations and eliminated dependencies on imprecise smaller models, hard-to-maintain taxonomies, and rigid upstream pipelines.

Use Lambda architecture with scheduled inference jobs

Rejected because: The Kappa/nearline architecture resolves time-travel issues, reduces GPU cost, and removes recovery of failed scheduled inference jobs.

Use cross-attention models directly for online inference

Rejected because: Cross-encoders performed better but had much higher computational requirements, so LinkedIn distilled them into efficient 7B decoder-only models.

Run inference on every database change

Rejected because: Meaningful-change detection and deduplication cut cost and volume by skipping unchanged text.

Patterns

fine-tuning-adaptation (model)

LinkedIn fine-tunes LLMs with proprietary relevance and engagement signals so embeddings understand job-marketplace terminology, role matching, and real hiring behavior.

rag-retrieval (retrieval)

JUDE embeddings power semantic retrieval and similarity computation in LinkedIn's job recommendation stack, replacing weaker standardized features.

cost-aware-routing (operations)

Kappa nearline inference, change detection, hashing, deduplication, two-tower efficiency, and cross-encoder distillation reduce LLM inference and training costs.

batch-offline-processing (operations)

Offline Flyte/Spark/K8S/Ray pipelines bootstrap new embedding versions and backfill historical embeddings before nearline updates become the source of truth.

eval-driven-development (evaluation)

LinkedIn evaluates architecture and model choices through relevance/engagement supervision, loss engineering, cross-encoder comparison, distillation gap closure, and production A/B metrics.

Tradeoffs

Use LLM-derived embeddings instead of standardized features and smaller ML models.

Chose fine-tuned LLM representations over standardized features, imprecise smaller models, taxonomies, and rigid upstream pipelines.

Rationale: LLM embeddings provide semantic compression, performance, transfer learning, and interoperability improvements for recommendations.

1. Compression: Reduces computational complexity"

— article, stated

Use Kappa nearline inference.

Chose streaming changelog-driven embedding generation over Lambda architecture with scheduled inference jobs.

Rationale: Kappa resolves time-travel issues, lowers GPU inference cost, and avoids failed scheduled-job recovery.

Use a shared two-tower LLM with prompt templates.

Chose shared base LLM plus specialized task descriptors over separate heavy models for each input type.

Rationale: This keeps parameter efficiency through weight sharing while guiding attention to job or member profile fields.

Distill cross-encoder performance into efficient decoder-only models.

Chose offline cross-encoder distillation over serving cross-attention models directly.

Rationale: The approach recovers 50% of the performance gap while preserving two-tower efficiency.

Gotchas (what bit them)

LLM embeddings improve quality only if the serving architecture can keep features fresh.

Root cause: Job posters and members expect new jobs, profile changes, and resume updates to affect recommendations immediately.

Resolution: LinkedIn uses Kappa nearline inference from changelog streams so embeddings are created, updated, or deleted within seconds.

Naive change tracking can make LLM inference too expensive at LinkedIn scale.

Root cause: Every database change could trigger embedding inference even when relevant text did not change meaningfully.

Resolution: Samza pipelines skip unchanged text, and hashing/deduplication reduce inference volume about 6x.

Training 7B+ embedding models needs memory and throughput engineering.

Root cause: Long text and multi-node model training exceed simple fine-tuning setups.

Resolution: LinkedIn combines LoRA, Flash Attention 2, Liger kernels, bfloat16, gradient accumulation, checkpointing, H100 distributed training, and DeepSpeed ZeRO.

Better cross-encoder quality was too expensive for direct serving.

Root cause: Full Transformer attention between text inputs has higher computational requirements than the two-tower path.

Resolution: LinkedIn uses offline cross-encoder distillation to recover half the performance gap while keeping two-tower efficiency.

Training labels pull in different directions.

Root cause: Relevance labels are high quality but scarce, while engagement labels are larger-scale but noisy.

Resolution: JUDE uses relevance labels to establish semantic quality and engagement labels to align embeddings with business outcomes.

Evaluation

LinkedIn evaluates JUDE through supervision quality, architecture experiments, loss-function engineering, two-tower versus cross-attention comparisons, latency and cost metrics, feature freshness, inference-volume reduction, and online business impact in job recommendation/search ranking models.

Metrics: +2.07% Qualified Applications · -5.13% Dismiss to Apply · +1.91% Total Job Applications · under 300ms p95 embedding inference latency · up to 3x embedding inference cost reduction from change detection · approximately 6x inference-volume reduction from hashing/deduplication · 50% of cross-encoder performance gap bridged · relevance label quality · engagement label alignment · feature freshness within seconds

Outcomes

• JUDE embeddings were ramped online in job recommendation and search L2 ranking models.
• JUDE replaced overlapping standardized features with LLM-derived representations.
• LinkedIn reports +2.07% qualified applications, -5.13% dismiss-to-apply ratio, and +1.91% total job applications.
• The team describes this as its highest metric improvement from a single model change during that half year.
• Nearline inference keeps jobs, profiles, and resumes reflected almost in real time.
• Future work will add members' job-seeking activity data and long-context LLM capabilities.

JUDE operates at LinkedIn scale: over 1 billion registered members, tens of millions of active job postings, three embedding entity types, 7B+ LLMs, multi-node H100 training, under-300ms p95 nearline embedding inference, up to 3x cost reduction from change detection, and about 6x inference-volume reduction from hashing/deduplication.