Building and Evolving a Production GenAI Application Stack

linkedin · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise GenAI application platform

LinkedIn needed a reusable GenAI application stack that could move from early prompt-based features to multi-turn assistants and agents while balancing fast product delivery with long-term platform leverage.

Constraints:

LinkedIn's online serving stack was historically Java-based.
AI engineers preferred Python for LLM workflows, prompt engineering, and evaluation.
Prompt, skill, memory, and model-inference infrastructure needed standardization across products.
Migration had to preserve operational stability and minimize product-roadmap disruption.

Architecture

LinkedIn evolved from Java midtiers and simple prompt-in/string-out systems to a Python GenAI application framework atop LangChain. The stack includes prompt source of truth, Jinja prompt resolution, skills registered as dynamic LangChain tools, messaging-based conversational memory, centralized GenAI proxy, model abstractions across Azure OpenAI and fine-tuned Llama models, and incremental migration practices.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  genai-application-framework[GenAI Application Framework] --> prompt-source-of-truth[Prompt Source Of Truth]
  genai-application-framework --> dynamic-langchain-tool[Dynamic LangChain Tool]
  skill-registry[Skill Registry] --> dynamic-langchain-tool[Dynamic LangChain Tool]
  dynamic-langchain-tool --> genai-application-framework[GenAI Application Framework]
  genai-application-framework --> conversational-memory[Conversational Memory]
  genai-application-framework --> model-inference-layer[Model Inference Layer]
  model-inference-layer --> azure-openai[Azure OpenAI]
  model-inference-layer --> fine-tuned-llama[Fine Tuned Llama]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`GenAI Application Framework`	Versioned internal library built atop LangChain for new GenAI applications.	`Prompt Source Of Truth`, `Skill Registry`, `Conversational Memory`, `Model Inference Layer`
`Prompt Source Of Truth`	Standardizes prompt modularization, versioning, and Jinja resolution.	`GenAI Application Framework`
`Skill Registry`	Registers endpoint implementations as LLM-invocable skills.	`Dynamic LangChain Tool`
`Dynamic LangChain Tool`	Retrieves skill definitions and invokes actual skills at runtime.	`Skill Registry`, `GenAI Application Framework`
`Conversational Memory`	Messaging-based memory integrated through LangChain conversational memory.	`GenAI Application Framework`
`Model Inference Layer`	Abstracts Azure OpenAI, GenAI proxy, and LinkedIn AI platform serving.	`Azure OpenAI`, `Fine Tuned Llama`, `GenAI Application Framework`
`Azure OpenAI`	Proprietary foundational model provider for initial GenAI apps.	`Model Inference Layer`
`Fine Tuned Llama`	Open-source model option fine-tuned for LinkedIn tasks.	`Model Inference Layer`

Data flow

A GenAI application uses LinkedIn's Python LangChain framework, resolves versioned prompts from the prompt source of truth, retrieves skills dynamically from the skill registry, attaches conversational memory from LinkedIn messaging, and routes model calls through the GenAI proxy or AI platform.

“Our GenAI application framework is now a thin wrapper atop LangChain, bridging it with LinkedIn infrastructure for logging, instrumentation, storage access and more. It is vended as a versioned internal library that is now mandated for use by all new GenAI applications at LinkedIn.”article, stated

“dynamic LangChain tool that retrieves skill definitions from the skill registry and invokes the actual skill with the supplied arguments, **eliminating developer specified skills in prompts and giving significantly larger agency to”article, stated

Why this architecture

LinkedIn accepted short-term fragmentation to preserve momentum, then moved Python into online serving because the GenAI ecosystem and AI engineers were Python-first. The stack uses abstractions so products can share prompts, tools, memory, and model routing while migrating incrementally off bespoke Java solutions.

“we made a conscious call to start with fragmented online and offline stacks with basic tooling to bridge them.”article, stated

“landscape and associated open source libraries continued to evolve primarily in Python**, it became clear that staying on Java for serving was a suboptimal long-term choice. We decided to invest in Python as a first-class language for both offline iteration and online serving at”article, stated

Models used: Azure OpenAI models · Llama · fine-tuned Llama · OpenAI Chat Completions API

Scale

LinkedIn started rolling out GenAI features in early 2023.
“In early 2023, we started rolling out a completely reimagined product portfolio, including for the first time features that leveraged Generative AI (GenAI).”article, stated
The stack supported products including collaborative articles, AI-assisted Recruiter, AI-powered insights, and Hiring Assistant.
“The products enabled by this stack include collaborative articles, AI-assisted Recruiter, AI-powered insights, and most recently, their first AI agent called Hiring Assistant.”summary, stated
The framework is mandated for all new LinkedIn GenAI applications.
“Our GenAI application framework is now a thin wrapper atop LangChain, bridging it with LinkedIn infrastructure for logging, instrumentation, storage access and more. It is vended as a versioned internal library that is now mandated for use by all new GenAI applications at LinkedIn.”article, stated

Alternatives considered (roads not taken)

Continue with shared Java midtiers.

Rejected because: They became development and operational bottlenecks as use cases grew.

“As the number of use cases grew, this midtier became a development and operational bottleneck, prompting us to split things up into multiple different use-case specific Java midtier services**.**”article, stated

Rebuild Python LLM tooling in Java.

Rejected because: LinkedIn chose fragmented stacks initially rather than be blocked by rebuilding the tooling.

“Rather than be blocked by the onerous task of rebuilding this tooling in Java or getting our online serving stack to support Python, we made a conscious call to start with fragmented online and offline stacks with basic tooling to bridge them.”article, stated

Developer-specified skills in prompts.

Rejected because: A dynamic skill registry gave LLMs larger agency and removed manual prompt-specified skill lists.

“LangChain tool that retrieves skill definitions from the skill registry and invokes the actual skill with the supplied arguments, **eliminating developer specified skills in prompts and giving significantly larger agency to LLMs”article, stated

Use only proprietary foundational models.

Rejected because: Fine-tuned Llama models could match or beat commercial models for LinkedIn tasks at lower cost and latency.

“often achieve** comparable or better quality as state of the art commercial foundational models on these tasks, but at **much lower costs and latencies.”article, stated

Patterns

`prompt-versioning` (prompt management)

LinkedIn introduced a prompt source of truth with Jinja templates and a prompt resolution library.

“To provide more structure around modularization and versioning, we introduced a Prompt Source of Truth component.”article, stated

`tool-use-orchestration` (skills)

Skills are registered and invoked dynamically through a LangChain tool.

“automatically register them in the skill registry service with the necessary validations around schema structure and documentation”article, stated

`agent-memory` (conversation state)

LinkedIn integrated messaging-based conversational memory into the GenAI application framework.

“LinkedIn messaging based Conversational Memory infrastructure** into our **GenAI application framework”article, stated

`model-routing` (model abstraction)

The stack abstracts proprietary Azure OpenAI and LinkedIn fine-tuned open-source models through shared infrastructure.

“All requests were routed via a centralized GenAI proxy which offered functionalities like Trust and Responsible AI checks”article, stated

`fine-tuning-adaptation` (model customization)

LinkedIn fine-tunes Llama for LinkedIn-specific tasks when quality, cost, and latency support it.

“when fine-tuned for LinkedIn specific tasks, often achieve **comparable or better quality”article, stated

Tradeoffs

Language and framework

Chose Python LangChain framework over Long-term Java-only serving.

Rationale: Python matched AI engineer workflows and the fast-moving GenAI ecosystem.

“prompt engineering and evaluations preferred Python** given the plethora of **open source Python solutions”article, stated

Migration strategy

Chose Incremental component migration over Big-bang rewrite.

Rationale: Incremental migration preserved stability and minimized product-roadmap impact.

“ensured high levels of operational stability, and minimized the impact to the product roadmap”article, stated

Model portfolio

Chose Abstract across proprietary and fine-tuned open-source models over Binding all applications to Azure OpenAI.

Rationale: Different tasks have tradeoffs across quality, cost, latency, and member data controls.

“various nuanced trade-offs around quality, cost, latency and more to navigate.”article, stated

Gotchas (what bit them)

A shared Java midtier became a bottleneck.

Root cause: Growing GenAI use cases made one reusable midtier hard to develop and operate.

Resolution: Invest in Python as a first-class online-serving language and build a LangChain-based framework.

“As the number of use cases grew, this midtier became a development and operational bottleneck”article, stated

Fragmented Java/Python stacks were costly to maintain.

Root cause: Logic had to be mirrored across offline Python workflows and online Java services.

Resolution: Converge the framework and infrastructure support around Python for both iteration and serving.

“required substantial effort to prevent divergence across the various Java midtier services and keep logic mirrored across the offline Python and online Java stacks.”article, stated

Big-bang platform migration would threaten product stability.

Root cause: GenAI products were actively shipping while framework, memory, and serving stacks changed.

Resolution: Migrate components incrementally, starting with smaller apps and ramping with A/B tests.

“Rather than do a big bang migration of all components, we decided to migrate individual components one by one.”article, stated

Evaluation

LinkedIn's stack includes prompt engineering and evaluations in Python workflows, logging/instrumentation in the shared framework, A/B tests for migrations, and model-quality comparisons across proprietary and fine-tuned open-source options.

Metrics: A/B test ramp stability · Model quality · Cost · Latency · Operational leverage

“Each migration followed a depth first approach with a small team first prototyping, identifying gaps and fixing them”article, stated