Cognitive Memory Agent: Building Stateful AI Agents with Multi-Layer Memory Architecture

linkedin · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Stateful memory infrastructure for recruiter-facing AI agents

LinkedIn needed Hiring Assistant and future agents to remember recruiter context, infer preferences, and improve over time without leaking tenant/user data or exceeding latency and context limits.

Constraints:

• Base LLMs are stateless and cannot deliver personalized longitudinal behavior without memory infrastructure.
• Persisted memory introduces consistency, latency, storage, isolation, privacy, compaction, observability, and lifecycle-management requirements.
• Memory retrieval spans heterogeneous conversational, episodic, semantic, and procedural stores rather than one static corpus.
• Incorrect or stale memories can increase user friction, cost, latency, and hallucination risk.

Architecture

LinkedIn built Cognitive Memory Agent as a horizontal memory platform for stateful agents. CMA exposes conversational, episodic, semantic, and procedural memory layers as tools; ingests user activity through streaming and batch pipelines; stores raw and aggregated preferences with ownership metadata; and uses an LLM-powered retrieval orchestrator to plan across memory providers, synthesize context, and personalize Hiring Assistant responses.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  application-agent["application agent"] --> cma-ingestion-api["CMA ingestion API"]
  application-agent --> cma-retrieval-api["CMA retrieval API"]
  application-agent --> business-context-and-tool-descriptions["business context and tool descriptions"]
  cma-ingestion-api --> streaming-ingestion-pipeline["streaming ingestion pipeline"]
  cma-ingestion-api --> batch-ingestion-pipeline["batch ingestion pipeline"]
  streaming-ingestion-pipeline --> conversational-memory["conversational memory"]
  streaming-ingestion-pipeline --> episodic-memory["episodic memory"]
  batch-ingestion-pipeline --> semantic-memory["semantic memory"]
  batch-ingestion-pipeline --> hierarchical-semantic-index["hierarchical semantic index"]
  hierarchical-semantic-index --> semantic-memory
  cma-retrieval-api --> access-control-layer["access-control layer"]
  cma-retrieval-api --> retrieval-orchestrator["retrieval orchestrator"]
  business-context-and-tool-descriptions --> retrieval-orchestrator
  retrieval-orchestrator --> conversational-memory
  retrieval-orchestrator --> episodic-memory
  retrieval-orchestrator --> semantic-memory
  retrieval-orchestrator --> procedural-memory["procedural memory"]
  access-control-layer --> conversational-memory
  access-control-layer --> episodic-memory
  access-control-layer --> semantic-memory
  evaluation-framework["evaluation framework"] --> retrieval-orchestrator
  evaluation-framework --> semantic-memory
  evaluation-framework --> application-agent

Components

Data flow

Application agents ingest user activity and conversation data into CMA. Streaming paths summarize conversations and extract episodic activity for low-latency freshness, while batch paths aggregate semantic memory with hierarchical summaries, Q&A pairs, and facets. At serving time an application sends a natural-language query and memory metadata to the retrieval API; the orchestrator selects conversational, episodic, semantic, or procedural tools, refines intermediate queries when needed, and synthesizes a response constrained by per-application, per-user memory access controls.

Why this architecture

LinkedIn treats memory as infrastructure rather than incidental prompt context because stateful personalization requires durable layered stores, asynchronous ingestion, access controls, and a planner that can decide where to look and how to reconcile memory signals.

Models used: open-source LLM hosted on LinkedIn internal infrastructure · LLM calls for memory extraction and summarization · embeddings in vector store · SFT and RL planned for orchestrator improvement

Scale

• Hiring Assistant became globally available in English in 2025.
  “In 2025, we announced the global availability of Hiring Assistant in English”article, stated
• CMA is positioned as a horizontal platform for stateful agents at scale.
  “Cognitive Memory Agent (CMA), a horizontal memory platform to build stateful and context-aware AI agents at scale.”article, stated
• Conversational memory uses LinkedIn's highly scalable messaging platform.
  “Messages are exchanged over the highly-scalable and fault tolerant LinkedIn messaging platform.”article, stated
• Tier-1 and Tier-2 evaluation metrics use continuous 0-to-1 scales.
  “Tier-1 evaluation metrics (continuous 0 to 1 scale).”article, stated

  “Tier-2 evaluation metrics (continuous 0 to 1 scale).”article, stated
• Tier-3 evaluation measures percentage of goals achieved.
  “Tier-3 evaluation metrics (% of goals successfully achieved).”article, stated

Alternatives considered (roads not taken)

Traditional user-specified memory

Rejected because: It requires users to explicitly say what to remember, while CMA stores and retrieves contextual information automatically.

Experiential memory as a bare hierarchical key-value store

Rejected because: It lacked intelligence, leaving each application to build extraction, indexing, storage, and retrieval logic manually.

Simple embed-query-retrieve-synthesize RAG

Rejected because: Static retrieval breaks down when memory is layered, heterogeneous, and continuously evolving.

GraphRAG or flattened RAG for semantic memory aggregation

Rejected because: LinkedIn chose a hierarchy to reduce edge-determination LLM calls and avoid feeding all leaf nodes into context at once.

One memory store for all applications or users

Rejected because: The access-control design isolates each application and enforces no data sharing across tenants or users.

Patterns

agent-memory (memory)

CMA provides conversational, episodic, semantic, and procedural memory layers so agents can remember interactions, preferences, events, and workflows across sessions.

tool-use-orchestration (orchestration)

The retrieval orchestrator treats memory providers as tools and plans which providers to call for each query.

context-compaction (context-management)

Conversational memory periodically summarizes dialogue so future turns can retrieve relevant history without exceeding context windows.

batch-offline-processing (data-processing)

CMA separates latency-sensitive streaming memory updates from batch semantic-memory aggregation and incremental updates.

privacy-preserving-processing (security)

CMA isolates application stores, tags ownership metadata, restricts access to logged-in sessions, and prevents tenant/user data sharing.

eval-driven-development (evaluation)

LinkedIn evaluates CMA through a three-tier framework for memory correctness, retrieval/answer alignment, and task success.

llm-as-judge (evaluation)

Tier-2 evaluation uses an LLM judge to assess orchestrator response quality and grounding.

retrieval-ranking (retrieval)

The orchestrator ranks and combines relevant memory layers through a retrieval plan instead of retrieving from one static index.

Tradeoffs

Use a layered cognitive memory architecture.

Chose Separate conversational, episodic, semantic, and procedural memory stores over One generic memory store.

Rationale: Different memory types have different semantics, latency needs, and abstraction levels, so one store cannot represent all useful agent memory equally well.

Split ingestion between streaming and batch.

Chose Streaming for conversational/episodic freshness and batch for semantic aggregation over One synchronous indexing path.

Rationale: The split keeps online interactions responsive while allowing heavier semantic extraction and aggregation to run asynchronously.

Use hierarchical semantic memory indexing.

Chose Tree-like aggregation of Q&A pairs, summaries, and facets over GraphRAG or flattened RAG.

Rationale: The hierarchy reduces LLM edge construction, avoids feeding all leaf nodes to the LLM, and supports parallel batch updates.

Couple planning, memory tool calls, and answer generation in one orchestrator loop.

Chose An internal open-source LLM orchestrator with unified reasoning over Separate retrieval planner and answer synthesizer services.

Rationale: LinkedIn says tight integration simplifies the system and keeps retrieval reasoning connected to answer synthesis.

Keep application-specific context outside the generic CMA core.

Chose Generic CMA tools customized by business context, tool descriptions, hooks, embeddings, or fine-tuned models over Hard-coding each application into the memory platform.

Rationale: This keeps CMA horizontal while still allowing each application to improve domain-specific retrieval behavior.

Gotchas (what bit them)

Persistent memory turns prompt context into production state.

Root cause: Once agent state persists across steps, memory must handle lifecycle, read/write semantics, consistency, latency, isolation, privacy, compaction, and observability.

Resolution: Treat memory as a first-class platform primitive with explicit layers, access controls, and lifecycle management.

Bare key-value memory does not create reusable agent intelligence.

Root cause: The original experiential memory layer provided storage APIs but made each application implement extraction, indexing, storage, and retrieval manually.

Resolution: Add dynamic extraction and synthesis so memory prompts can mine implicit preferences into structured records.

Episodic memory extraction is easy to over- or under-segment.

Root cause: Events overlap, interleave, become stale, or conflict, and the memory must capture enough detail without overfitting.

Resolution: Model episodic memory as timestamped contextual records and explicitly account for boundaries, staleness, conflict resolution, and context association.

Static RAG breaks down over layered evolving memory.

Root cause: A single search operation cannot determine which memory layer is relevant or coordinate signals across heterogeneous stores.

Resolution: Use a lightweight agentic orchestrator that plans memory-provider calls and synthesizes across layers.

Memory can make an agent worse if persisted facts are wrong or stale.

Root cause: Incorrect memory can increase friction, cost, latency, and hallucinated or stale responses.

Resolution: Evaluate representation correctness, retrieval alignment, task success, latency, planning steps, token usage, and ingestion LLM calls.

Memory systems create data-leakage risk across products, users, and tenants.

Root cause: Shared memory without ownership metadata and session restrictions could expose one user's or tenant's data to another.

Resolution: Isolate memory stores, tag ownership metadata, restrict access to logged-in users, and enforce no cross-tenant/user sharing.

Evaluation

LinkedIn evaluates CMA with a three-tier framework covering memory representation correctness and coverage, query-context-answer alignment via LLM-as-judge, and end-to-end product task success, while also tracking latency, reasoning tokens, planning steps, ingestion token usage, and LLM calls.

Metrics: memory representation correctness · memory coverage · retrieved information relevance · response grounding · answer completeness · task success rate · user satisfaction studies · online A/B tests · response latency · reasoning tokens · planning steps · ingestion token usage · LLM calls during ingestion

Outcomes

• CMA powers personalization in LinkedIn Hiring Assistant.
• Hiring Assistant can suggest roles based on past recruiter projects and preferences.
• CMA helps auto-populate hiring requirements for new projects.
• Recruiters can query past hiring projects and activities through Hiring Assistant.

The article positions CMA as a horizontal memory platform at LinkedIn scale, initially applied to Hiring Assistant after its global English availability in 2025; it does not provide production adoption counts beyond global availability and product use-case descriptions.