Knowledge Graph-Enhanced RAG for Customer Service Question Answering

linkedin · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer service question answering

LinkedIn customer service representatives needed to retrieve and use relevant historical issue tickets quickly, but conventional RAG treated tickets as flat text and lost ticket structure plus relationships across issues. That hurt retrieval accuracy, answer quality, and issue-resolution time.

Constraints:

• historical issue tickets contain meaningful intra-issue sections such as summaries, descriptions, reproduction steps, and fix solutions
• related tickets can be linked explicitly in issue metadata or implicitly through semantic similarity
• customer-service users need relevant past issues near the top of the result list
• answers must synthesize retrieved subgraphs into usable support guidance
• the system was deployed in production for approximately six months
• fallback behavior was needed when graph-query generation or execution failed

Architecture

LinkedIn built a knowledge-graph-enhanced RAG system for customer-service QA. Offline, Jira tickets are parsed into graph trees that preserve issue sections, explicit ticket relationships, and implicit semantic links from E5 title embeddings. Text-rich nodes are embedded into a vector store. Online, GPT-4 extracts query entities and intent using the same structural template, semantic retrieval selects top relevant tickets, GPT-4 translates a reformulated query into Cypher for subgraph extraction, and GPT-4 generates the answer from the original question plus retrieved graph context. If graph execution fails, the system falls back to traditional text retrieval.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Historical_issue_tickets["Historical issue tickets"] --> Offline_graph_construction["Offline graph construction"]
  Offline_graph_construction --> GPT4_ticket_parser["GPT-4 ticket parser"]
  GPT4_ticket_parser --> Issue_knowledge_graph["Issue knowledge graph"]
  Issue_knowledge_graph --> Node_embedding_index["Node embedding index"]
  Customer_service_query["Customer service query"] --> GPT4_query_parser["GPT-4 query parser"]
  GPT4_query_parser --> Semantic_retriever["Semantic retriever"]
  Node_embedding_index --> Semantic_retriever
  Semantic_retriever --> Cypher_query_generator["Cypher query generator"]
  GPT4_query_parser --> Cypher_query_generator
  Cypher_query_generator --> Issue_knowledge_graph
  Issue_knowledge_graph --> Retrieved_subgraph["Retrieved subgraph"]
  Cypher_query_generator --> Fallback_text_retriever["Fallback text retriever"]
  Retrieved_subgraph --> GPT4_answer_generator["GPT-4 answer generator"]
  Fallback_text_retriever --> GPT4_answer_generator

Components

Data flow

Historical tickets are transformed into a knowledge graph that retains each ticket's internal tree structure and relationships to other issues. Text-rich graph nodes receive BERT/E5-style embeddings and are stored for semantic retrieval. At query time, GPT-4 parses the user request into graph-aligned entities and intent, the retriever finds related tickets and subgraphs, GPT-4 translates the request into Cypher over those ticket IDs, and the answer generator synthesizes the retrieved subgraph into customer-service guidance. Graph failures fall back to text-based retrieval.

Why this architecture

The architecture preserves ticket structure before retrieval so answers can use sections and relationships that ordinary chunked RAG discards. Graph queries recover subgraphs spanning ticket sections and related issues, while embeddings still provide semantic recall. The fallback path keeps the system usable when generated graph queries fail.

Models used: GPT-4 · E5 embeddings · BERT embeddings · Qdrant vector database · Neo4j Cypher

Scale

• The system ran in LinkedIn customer service production for approximately six months.
  “Our method has been deployed within LinkedIn's customer service team for approximately six months”article, stated
• Production impact was measured as a 28.6% reduction in median per-issue resolution time.
  “has reduced the median per-issue resolution time by 28.6%.”article, stated
• The retrieval benchmark improved MRR by 77.6%.
  “outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU.”article, stated
• Generation quality improved by 0.32 BLEU versus the baseline.
  “outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU.”article, stated

Alternatives considered (roads not taken)

Conventional flat-text RAG over issue tickets

Rejected because: Flat retrieval ignores ticket structure and inter-ticket relationships, limiting performance.

Document-level chunking before retrieval

Rejected because: The summary says fixed text chunks can fragment logically connected ticket content across boundaries.

Graph-only retrieval without semantic embeddings

Rejected because: The implemented design still uses semantic retrieval to find relevant candidate tickets before graph subquery extraction.

Fail closed when Cypher query generation fails

Rejected because: The production design falls back to traditional text retrieval for service continuity.

Patterns

rag-retrieval (retrieval)

LinkedIn combines semantic retrieval with graph subgraph retrieval so customer-service answers preserve issue structure and relationships.

tool-use-orchestration (architecture)

GPT-4 converts the reformulated natural-language request into Cypher so the system can use graph database queries as a retrieval tool.

query-rewriting (retrieval)

The system parses user queries into graph-aligned entities and intent, then reformulates them around candidate ticket IDs before generating graph queries.

retrieval-ranking (retrieval)

Node-level embedding similarities are aggregated to rank historical tickets before graph retrieval.

eval-driven-development (evaluation)

The paper evaluates both retrieval and answer generation with MRR, Recall@K, NDCG@K, BLEU, ROUGE, METEOR, and production resolution-time impact.

Tradeoffs

Preserve issue-ticket structure in a knowledge graph.

Chose KG-enhanced RAG over plain-text RAG.

Rationale: The graph retains intra-ticket and inter-ticket structure, improving retrieval accuracy and answer quality.

Use GPT-4 for parsing and graph-query translation.

Chose LLM-mediated entity, intent, and Cypher generation over manual query construction or pure embedding search.

Rationale: The LLM maps natural-language support questions into graph-aligned entities and executable graph queries.

Combine semantic retrieval with graph queries.

Chose embedding top-K retrieval followed by subgraph extraction over graph-only lookup or embedding-only retrieval.

Rationale: Semantic retrieval finds relevant candidate tickets while graph queries recover structured sections and relationships for the answer.

Keep a traditional retrieval fallback.

Chose fallback text-based retrieval over depending exclusively on generated graph queries.

Rationale: Fallback preserves service continuity when graph query generation or execution fails.

Gotchas (what bit them)

Flat RAG loses ticket structure that customer-service answers need.

Root cause: Treating tickets as plain text ignores intra-issue fields and inter-issue relations.

Resolution: Construct a knowledge graph from historical tickets and retrieve related subgraphs.

Chunking can split logically connected ticket content.

Root cause: Fixed-length text segments do not respect ticket sections such as descriptions, reproduction steps, and fixes.

Resolution: Embed cohesive graph nodes rather than arbitrary chunks.

Graph-query generation can fail in production.

Root cause: The generated Cypher query or graph execution path can encounter edge cases.

Resolution: Fallback to traditional text-based retrieval when the graph path fails.

Static graph templates may become stale as support tickets evolve.

Root cause: The current approach depends on a YAML template and offline construction phase.

Resolution: The authors identify automated graph-template extraction and dynamic graph updates as future directions.

Evaluation

The system was evaluated on benchmark datasets with retrieval metrics (MRR, Recall@K, NDCG@K), generation metrics (BLEU, ROUGE, METEOR), and real production impact over approximately six months in LinkedIn customer service.

Metrics: 77.6% MRR improvement · 0.32 BLEU improvement · 28.6% median per-issue resolution-time reduction · Recall@K · NDCG@K · ROUGE · METEOR

Outcomes

• The KG-enhanced RAG method outperformed the baseline by 77.6% in MRR.
• The method improved BLEU by 0.32 versus the baseline.
• The deployed system reduced median per-issue resolution time by 28.6%.
• The system ran inside LinkedIn's customer service team for approximately six months.

The fetched source is the arXiv abstract page, but it states that the method was deployed within LinkedIn customer service for approximately six months and reduced median issue-resolution time by 28.6%. The ZenML summary adds that the system served multiple product lines.