Building and Scaling a Production Generative AI Assistant for Professional Networking

linkedin · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Professional-networking assistant

LinkedIn needed a production generative assistant that could help members explore feed posts, assess job fit, and get career advice while grounding answers in LinkedIn data and maintaining consistent product quality across many specialized agent experiences.

Constraints:

• queries needed to route to the right specialized agent and stay in scope
• answers required LinkedIn-specific people, company, job, skill, course, and content data not present in base LLM training
• multiple vertical teams needed autonomy without fragmenting identity, UX, instrumentation, and evaluation
• production quality demanded high factuality, low hallucination, Responsible AI handling, and empathetic responses
• large-model capacity, TTFT, TBT, and token cost had to be managed while still streaming full answers to members

Architecture

LinkedIn built a fixed three-step RAG assistant pipeline. Routing uses smaller models and development sets to choose whether a query is in scope and which specialized agent should handle it. The chosen agent performs retrieval by deciding which internal skills, LinkedIn APIs, or Bing calls to invoke, then a larger generation model filters the retrieved dossier into a response with attachments. A horizontal platform pod owns hosting, shared prompts, evaluation, instrumentation, UX templates, server-driven UI, and common skills, while vertical pods own individual agents such as job assessment, company understanding, post takeaways, and interview tips.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  member-experience[Member experience] --> assistant-hosting-service[Assistant hosting service]
  assistant-hosting-service --> router[Router]
  router --> specialized-agents[Specialized agents]
  specialized-agents --> skill-layer[Skill layer]
  skill-layer --> linkedin-internal-apis[LinkedIn internal APIs]
  skill-layer --> bing-api[Bing API]
  skill-layer --> defensive-yaml-parser[Defensive YAML parser]
  specialized-agents --> generation-model[Generation model]
  shared-prompt-templates[Shared prompt templates] --> specialized-agents
  generation-model --> realtime-messaging-infrastructure[Realtime messaging infrastructure]
  realtime-messaging-infrastructure --> member-experience
  assistant-hosting-service --> evaluation-pipeline[Evaluation pipeline]

Components

Data flow

A member clicks a starter question beside a feed item or job post. The service routes the request to a specialized agent, such as a general-knowledge, job, or career agent. That agent gathers grounding data through internal APIs, LinkedIn skills, and external search such as Bing, using structured YAML outputs to pick skills and call parameters. The generation step filters and synthesizes the dossier, decorates the answer with attachments, streams the final response through realtime messaging, and sends conversations into evaluation and annotation pipelines for quality tracking.

Why this architecture

LinkedIn kept the product architecture deliberately decomposed: routing and retrieval are classification-like and could be handled with smaller models and per-step evals, while generation needed larger models and more creative tuning. A common horizontal platform prevented independent agent teams from fracturing the experience, and skill wrappers let LLMs use LinkedIn-specific APIs without retraining on private data. Streaming, async execution, defensive parsing, and model tiering are production concessions to latency, capacity, and structured-output reliability.

Models used: smaller routing/retrieval models · larger generation models · in-house models · model-based evaluators

Scale

• The initial product was developed over six months.
  “Over the last six months, our team here at LinkedIn has been working hard to develop a new AI-powered experience.”article, stated
• LinkedIn had dozens of specialized agents that could handle user queries.
  “KSA stands for “Knowledge Share Agent”, one of the dozens of agents that can handle user queries”article, stated
• The linguist annotation process evaluated up to 500 conversations per day.
  “Our internal linguist team built tooling and processes by which we could evaluate up to 500 daily conversations and get metrics around: overall quality score, hallucination rate, Responsible AI violation, coherence, style, etc.”article, stated
• Annotator feedback had roughly a one-day turnaround, while some member-feedback metrics took more than three days per change.
  “Annotators give more granular feedback but have a ~1 day turnaround. Members are the final judges and give us scale, but some metrics can take 3+ days for a single change”article, stated
• Malformed YAML parameters occurred in about 10% of LLM responses before the defensive parser and prompt hints reduced them to about 0.01%.
  “~10% of the time the LLM would make mistakes and often output data that was invalid as per the schema supplied, or worse not even valid YAML.”article, stated

  “We were ultimately able to reduce occurrences of these errors to ~0.01%.”article, stated
• A 200-token hidden reasoning step can add about two seconds of latency for every 10ms increase in TBT.
  “for a 200-token reasoning step, even a 10ms TBT increase means an extra 2s of latency”article, stated

Alternatives considered (roads not taken)

One general assistant team owning every capability

Rejected because: LinkedIn split work across vertical agent teams to move quickly, then added a horizontal platform pod to reduce fragmentation.

A fragmented set of fully independent agent products

Rejected because: Fragmentation made it hard to maintain a uniform user experience across models, prompts, and tools, so common prompts, UX, evaluation, and hosting were centralized.

Re-prompting the LLM whenever structured YAML output failed

Rejected because: Re-prompting would add latency and burn scarce GPU capacity, so the team repaired common YAML mistakes locally.

Optimizing only time to first token

Rejected because: Chain-of-thought reasoning made time between tokens a major hidden-latency driver, so capacity and latency management had to account for TBT too.

Patterns

rag-retrieval (knowledge-access)

The assistant grounds responses by routing to an agent that calls LinkedIn internal APIs and Bing, then uses retrieved context to synthesize an answer.

multi-agent-orchestration (orchestration)

Member queries are routed to specialized agents for job assessment, company understanding, post takeaways, career advice, and general knowledge.

tool-use-orchestration (tooling)

LinkedIn wraps internal RPC APIs as LLM-friendly skills so agents can select APIs, emit call parameters, and invoke LinkedIn-specific data services.

structured-output-contract (interface)

Skill invocation requires structured YAML parameters matching LLM-friendly schemas, with parser repair for common schema and syntax mistakes.

eval-driven-development (quality)

LinkedIn made evaluation the main signpost for prompt iteration and launch readiness, combining engineer checks, annotators, member feedback, and model-based evaluators.

latency-optimization (operations)

The system streams end to end, progressively parses tool-call parameters, fires APIs as soon as parameters are ready, and uses async non-blocking execution to reduce perceived latency and wasted threads.

Tradeoffs

Agent organization

Chose vertical specialized agents plus a horizontal platform pod over one monolithic assistant team or fully independent agent experiences.

Rationale: Specialized teams could move quickly, while shared prompts, evaluation, UX, and hosting reduced product fragmentation.

Model allocation

Chose small models for routing/retrieval and larger models for generation over using the largest model for every step.

Rationale: Routing and retrieval behaved more like classification tasks, while generation required higher-capability models for precision and synthesis.

Structured output format

Chose YAML skill-call parameters with a defensive parser over JSON or retrying failed structured-output calls with another LLM request.

Rationale: YAML consumed fewer tokens than JSON, and parser repair avoided extra latency and GPU capacity for re-prompts.

Latency strategy

Chose end-to-end streaming and async non-blocking execution over waiting for full LLM/tool completion before responding.

Rationale: Full answers could take minutes, so streaming and non-blocking processing reduced perceived latency and avoided tying up resources on I/O.

Gotchas (what bit them)

Independent agent teams can fragment the user experience.

Root cause: Different vertical teams may ship different models, prompts, tools, and interaction patterns for subsequent turns of the same assistant.

Resolution: Create a horizontal pod for hosting, evaluation, shared prompts, shared UX components, instrumentation, and server-driven UI.

Human evaluation guidelines were hard to make consistent for nuanced career advice.

Root cause: Some answers had to be factual and empathetic at the same time, especially when users were considering career changes.

Resolution: Develop consistent annotation guidelines and use linguist-built tooling and processes for scaled scoring.

Structured tool-call parameters failed often enough to break production parsing.

Root cause: About 10% of LLM outputs had invalid schema formats or were not valid YAML at all.

Resolution: Analyze common mistakes, patch them before parsing, and add prompt hints to reduce bad structured outputs to about 0.01%.

Quality improvement plateaued after early progress.

Root cause: The easy 80% experience arrived quickly, but hallucination detection and every additional quality percentage point required far more effort.

Resolution: Invest in more sophisticated evaluation, automated evaluators, and fine-tuning to make the pipeline more data-driven.

Bounding only TTFT hid latency created by chain-of-thought tokens.

Root cause: Hidden reasoning tokens make time-between-tokens accumulate into visible delay even before member-visible output appears.

Resolution: Increase capacity quickly during the ramp and work on moving simpler tasks to in-house fine-tuned models, predictable deployments, and token reduction.

Evaluation

LinkedIn used a layered evaluation process: engineers ran fast directional checks, internal linguists annotated up to 500 conversations per day with quality and safety metrics, member feedback supplied scale, and model-based evaluators were being built for faster experimentation.

Metrics: overall quality score · hallucination rate · Responsible AI violation · coherence · style · up to 500 daily conversations · 1+ day delay for annotation metrics · 3+ days for some member-feedback metrics

Outcomes

• the team launched the product experience and planned continued rollout to more users
• the basic RAG framework was working within a few days
• the team reached about 80% of the target experience within the first month but spent four more months pushing past 95%
• defensive YAML parsing reduced structured-output errors from about 10% to about 0.01%

The case reports a six-month build, dozens of agents, annotation of up to 500 conversations daily, and production ramps that exposed capacity and latency limits.