AI-Powered Skills Extraction and Mapping for the LinkedIn Skills Graph

linkedin · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Skills graph extraction and normalization

LinkedIn needed to extract implicit and explicit skills from profiles, resumes, job postings, courses, and feed content, then map them to a consistent Skills Graph so skills-first hiring, learning, search, and recommendations could stay accurate as work changes.

Constraints:

• skills can be mentioned explicitly or indirectly in many content types
• where a skill appears in a document changes its relevance
• the skills taxonomy contained more than 41,000 skills
• production serving had to meet strict nearline latency for hundreds of profile edits per second
• different product surfaces needed feedback loops for ongoing model improvement

Architecture

LinkedIn built a multi-stage skills extraction and mapping stack. It first segments raw content into structured sections, then uses both a trie-based token tagger and a semantic two-tower Multilingual BERT model to identify candidate skills. The Skills Graph expands candidates via related parent, child, sibling, and group skills. A multitask cross-domain scorer combines contextual text encoding, contextual entity embeddings, manual co-occurrence features, and vertical-specific towers for jobs, profiles, feeds, and other sources. For production serving, LinkedIn distills larger models into smaller CPU-serving models, uses offline/Spark scoring for full reprocessing, and collects recruiter, job-seeker, and skill-assessment feedback to improve extraction.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  content-segmentation-parser["content-segmentation-parser"] --> token-skill-tagger["token-skill-tagger"]
  content-segmentation-parser --> semantic-skill-tagger["semantic-skill-tagger"]
  token-skill-tagger --> skills-graph-expander["skills-graph-expander"]
  semantic-skill-tagger --> skills-graph-expander
  skills-graph-expander --> multitask-skill-scorer["multitask-skill-scorer"]
  contextual-text-encoder["contextual-text-encoder"] --> multitask-skill-scorer
  contextual-entity-encoder["contextual-entity-encoder"] --> multitask-skill-scorer
  multitask-skill-scorer --> model-serving-layer["model-serving-layer"]
  model-serving-layer --> product-feedback-loops["product-feedback-loops"]
  product-feedback-loops --> model-serving-layer
  model-serving-layer --> downstream-skills-products["downstream-skills-products"]

Components

Data flow

Raw text from job postings, profiles, resumes, courses, or feed posts is segmented into meaningful parts such as qualifications, responsibilities, skills, or experience. Token and semantic taggers extract direct and inferred skill mentions, then normalize them to canonical taxonomy skills. The Skills Graph expands candidates with structurally related skills. A shared text/entity encoder and domain-specific towers score content-skill pairs for mention, required, core, importance, and relevance relationships. Online and nearline serving publish extracted skills to downstream products, while recruiter edits, seeker feedback, and assessments feed model iterations.

Why this architecture

LinkedIn combined symbolic and semantic approaches because exact skill taxonomy matching is fast but misses real-world phrasing, while semantic BERT models capture implicit mentions. Segmenting documents preserves section-level importance. Skill Graph expansion broadens candidates beyond direct mentions, and multitask domain towers let each product vertical model nuanced relationships without duplicating the shared contextual encoders. Distillation and hybrid offline/nearline serving were required to meet strict profile-edit latency on CPU infrastructure.

Models used: Multilingual BERT · Transformer · two-tower model · knowledge distillation · Samza-BEAM · Spark offline scoring · multitask learning · uncertainty weighting

Scale

• LinkedIn's skills taxonomy contains more than 41,000 skills.
  “the more than 41,000 skills across our platform.”article, stated
• The standardized member profile skills feature handles about 200 global profile edits per second.
  “With approximately 200 global profile edits per second”article, stated
• Each profile edit message must be processed in under 100 milliseconds.
  “each message must be processed in under 100 milliseconds.”article, stated
• Knowledge distillation reduced online model size by 80%.
  “Knowledge Distillation reduces the model size by 80% without compromising performance”article, stated
• Skill Assessments award verified badges to members scoring in the 70th percentile or higher.
  “Upon successfully passing a SA with a 70th percentile or higher score, members are awarded a “verified skill” badge”article, stated
• A/B-tested improvements included +0.4606% predicted confirmed hires for job recommendations and +0.7577% PPC revenue for job search.
  “Predicted Confirmed Hires: +0.4606%”article, stated

  “PPC revenue: +0.7577%”article, stated

Alternatives considered (roads not taken)

only extracting explicitly listed skills

Rejected because: Skills appear in summaries, experiences, resumes, job descriptions, courses, and transcripts rather than only dedicated fields.

trie/token matching alone

Rejected because: Taxonomy lookup is fast but cannot capture every real-world expression of a skill.

serving the full 12-layer BERT model online

Rejected because: Profile-edit throughput and under-100ms latency made full BERT serving too expensive/slow on the existing CPU serving stack.

one model per relationship from scratch

Rejected because: Separate models for mention, required, core, and future relationships would be inefficient and not scale for better resolution.

Patterns

hybrid-search (retrieval)

LinkedIn combines trie/token skill matching with semantic two-tower BERT matching to cover both exact and implicit skills.

model-distillation (model-operations)

A larger teacher network trains a smaller student network for lower-latency online serving.

data-quality-loop (quality)

Recruiter edits, seeker feedback, and Skill Assessment signals provide production feedback for model iterations.

fine-tuning-adaptation (model-operations)

Models are fine-tuned for different content types because skills appear differently in resumes, profiles, and job descriptions.

batch-offline-processing (operations)

LinkedIn uses Spark offline scoring for full data reprocessing while nearline processors handle low-latency profile edits.

Tradeoffs

Use trie matching plus semantic matching.

Chose hybrid token/semantic skill tagging over only fast taxonomy lookup or only semantic inference.

Rationale: Trie matching is extremely fast at scale, while semantic BERT covers implicit wording the taxonomy may not enumerate.

Use shared encoders with domain-specific towers.

Chose shared text/entity context plus vertical-owned towers over one universal scorer or fully separate models per vertical.

Rationale: Common context transfers across domains while vertical towers preserve flexibility for jobs, profiles, feeds, and other surfaces.

Use offline resources for reprocessing and nearline processors for nearline traffic.

Chose hybrid offline/nearline serving over one serving path for every workload.

Rationale: Full reprocessing and profile-edit traffic have different latency/cost requirements.

Gotchas (what bit them)

Skill mention location changes relevance.

Root cause: A skill in a qualifications section is usually more important than one buried in company description text.

Resolution: Segment raw text into meaningful sections before skill extraction and scoring.

A skills graph becomes stale without continuous extraction.

Root cause: Work requirements and skill language change rapidly, especially during the generative AI boom.

Resolution: Continuously extract and map skills from platform content and product feedback.

Model size conflicted with nearline latency requirements.

Root cause: A full 12-layer BERT model is computationally demanding under 200 edits/second and 100ms message constraints.

Resolution: Use knowledge distillation to shrink the model and a hybrid offline/nearline serving approach.

Evaluation

LinkedIn evaluated the skills system with product feedback loops, offline evaluation, and A/B tests across job recommendations, job search, and job-member skills matching.

Metrics: +0.1391% member job applicants and offsite apply clickers · +0.4606% predicted confirmed hires for job recommendations · +0.1468% job sessions · +0.7577% PPC revenue · +0.2271% engagements · +0.87% qualified applications

Outcomes

• the Skills Graph is continuously enriched from unstructured platform content
• extracted skills power jobs, courses, feed ranking, recommendations, Job Search, and Recruiter Search
• distillation enabled nearline serving under strict latency constraints
• A/B tests showed measurable improvements in job recommendation, search, and skill matching metrics

The source reports 41,000+ skills, about 200 profile edits/second under 100ms, 80% model-size reduction, and product lifts up to +0.7577% PPC revenue.