Automated Knowledge Base Enhancement Using LLMs and Clustering for Customer Support

doordash · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support knowledge management

DoorDash needed to keep a support chatbot's knowledge base current as marketplace policies, product changes, and long-tail Dasher/customer issues created gaps faster than manual article maintenance could handle.

Constraints:

• support conversations occur at very high marketplace volume
• only knowledge gaps that actually cause escalation should be prioritized
• similar customer issues can be phrased many different ways
• policy, privacy, and edge-case nuances require human review before publishing
• the generation pipeline and production retrieval pipeline must stay aligned

Architecture

DoorDash filters anonymized support conversations to escalated chats, embeds their summaries, and clusters them by cosine similarity to find distinct high-impact knowledge gaps. High-ROI clusters flow to an LLM that classifies the issue type and drafts knowledge-base articles from issue summaries plus exemplary agent resolutions. Content specialists and operations partners review drafts for policy, tone, privacy, and conditional logic before approved user-generated KB articles are embedded into a vector database. In production, the chatbot embeds the live user issue, retrieves the matching UGC KB article through RAG, and answers with the retrieved content plus conversation context. Offline LLM-judge experiments and online A/B tests measure whether the new content lowers escalation rates.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  escalated-chat-transcript-pool[Escalated chat transcript pool] --> embedding-clustering-pipeline[Embedding clustering pipeline]
  embedding-clustering-pipeline --> cluster-review-loop[Cluster review loop]
  cluster-review-loop --> llm-classifier-and-drafter[LLM classifier and drafter]
  embedding-clustering-pipeline --> llm-classifier-and-drafter
  llm-classifier-and-drafter --> human-review-queue[Human review queue]
  human-review-queue --> ugc-kb-vector-store[UGC KB vector store]
  ugc-kb-vector-store --> production-chatbot-rag-layer[Production chatbot RAG layer]
  production-chatbot-rag-layer --> evaluation-loop[Evaluation loop]

Components

Data flow

Escalated support transcripts are anonymized and summarized, then embedded with an open-source semantic model. A clustering routine compares each vector with existing centroids and either updates a matching cluster or creates a new one. The highest-ROI clusters are classified by an LLM; informational clusters receive draft KB articles generated from the issue summary and exemplary agent resolutions. Human reviewers refine and approve the articles, which are then embedded by user-issue text and stored in a vector database for the production chatbot to retrieve.

Why this architecture

DoorDash chose escalation-driven clustering because escalations expose the specific gaps where the chatbot already failed, making the content backlog data-driven instead of speculative. It used LLMs for the two tasks that need language judgment: classifying whether a cluster needs workflow/policy handling and drafting article text from agent resolutions. Human review stayed in the path because policy variants, order state, and privacy details can change the correct answer inside one cluster. The production RAG design embeds the user-issue portion of each article, rather than the full KB entry, to make live issue matching more precise.

Models used: open-source embedding model · large language models · LLM judge

Scale

• The clustering input includes thousands of anonymized chat transcripts.
  “We begin by feeding thousands of anonymized chat transcripts into a semantic clustering pipeline, selecting only those conversations that were escalated to a live agent so that we can zero in on the cases where our chatbot fell short.”article, stated
• Similarity thresholds were tuned in the 0.70 to 0.90 range.
  “If the best match exceeds a configurable threshold — in practice, 0.70 ≤ τ ≤ 0.90 — we assign the chat to that cluster”article, stated
• High-traffic clusters dropped from 78% escalation in control to 43% in treatment.
  “high-traffic escalation message clusters saw escalation rates drop from 78% in the control group to 43% in the treatment group”article, stated
• Roughly 75% of KB retrieval events in treatment used only UGC KB content.
  “For example, high-traffic escalation message clusters saw escalation rates drop from 78% in the control group to 43% in the treatment group, and roughly 75% of KB retrieval events in treatment now contain only UGC KB content.”article, stated

Alternatives considered (roads not taken)

Manual knowledge-base maintenance

Rejected because: DoorDash found manual maintenance too resource-intensive and too slow for marketplace policy changes and edge cases.

Embed entire KB entries for retrieval

Rejected because: The team wanted more precise matching between live issues and solutions, so it embedded only the user-issue portion.

Fully automated publishing of LLM-written articles

Rejected because: Policy accuracy, tone, temporary policy overrides, branching behavior, and privacy redactions required human review.

Patterns

rag-retrieval (retrieval)

Approved UGC KB articles are embedded and served through a RAG layer so the chatbot can retrieve the article matching the live user's issue.

human-in-the-loop-gate (governance)

Content specialists and operations partners review every generated draft before it becomes production chatbot content.

eval-driven-development (evaluation)

DoorDash measures generated KB value with offline LLM-judge benchmarks and online A/B tests before treating content as effective.

prompt-versioning (prompt-operations)

The team refines prompts after review findings and logs corrections into future iterations.

Tradeoffs

Prioritize escalated transcripts as clustering input

Chose escalation-filtered support conversations over all support conversations.

Rationale: Escalated chats identify places where the chatbot already failed, making gap discovery more directly tied to business impact.

Embed issue text instead of full article text

Chose user-issue embeddings over entire KB-entry embeddings.

Rationale: Retrieval precision mattered more than exposing every article token to vector search.

Keep human review before publishing generated KB content

Chose machine drafting with content-specialist review over fully automated drafting and deployment.

Rationale: Policy and privacy edge cases make unreviewed support content risky even when LLM drafts are fast.

Gotchas (what bit them)

Cluster thresholds can merge genuinely different issues or split duplicate phrasings.

Root cause: Semantic similarity alone does not guarantee that a cluster corresponds to a distinct support topic.

Resolution: Iterate thresholds, inspect top clusters manually, and merge clusters that only rephrase the same question.

Generated articles can miss conditional policy paths and privacy handling.

Root cause: A single topic cluster can contain multiple valid resolutions depending on order state, policy overrides, or personal details.

Resolution: Expand the sample set, add explicit instructions for conditions/privacy, and route drafts through content and operations review.

The generation pipeline and production retrieval path can drift apart.

Root cause: Different summarization prompts, models, or embedding models can produce mismatched issue representations.

Resolution: Keep production issue summarization and UGC generation aligned, and use the same embedding model for both vectorization paths.

Evaluation

DoorDash evaluates article improvements offline with an LLM judge and online through selected-audience A/B tests measuring escalation reduction and UGC retrieval usage.

Metrics: offline LLM-judge relevance improvement · online A/B escalation rate · share of retrieval events containing only UGC KB content

Outcomes

• High-traffic escalation clusters dropped from 78% to 43% escalation in treatment.
• About 75% of treatment retrieval events contained only UGC KB content.
• Specialists were freed from manual transcript review to focus on novel edge cases.

The system targets DoorDash's high-volume Dasher and customer support traffic by turning thousands of escalated transcripts into a ranked KB backlog.