Improving AI Documentation Assistant Through Data Pipeline Reconstruction and LLM-Based Feedback Analysis

mintlify · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production monitoring and improvement of an AI documentation assistant

Mintlify's AI documentation assistant was in production with citations and code examples, but it was not performing the way the team wanted. The team needed to understand why users gave thumbs-down feedback, but feedback events were disconnected from full conversation threads and thread storage in PSQL was not queryable enough for analysis.

Constraints:

• feedback events were stored in ClickHouse without a mapping to the original conversation thread
• conversation threads were stored in PSQL in a way that made direct querying impossible
• negative feedback needed full conversation context, not isolated events
• manual review alone would not scale to larger samples
• threads could belong to more than one failure category
• a model upgrade alone had not changed feedback patterns materially

Architecture

Mintlify rebuilt its assistant feedback pipeline so negative reactions could be analyzed with full conversation context. When a feedback event is received, the server now pushes the full conversation thread to ClickHouse, and a backfill copied all historical messages with feedback from PSQL into ClickHouse. The team manually read about 100 negative conversation threads, created eight feedback categories, then used an LLM to classify a random sample of 1,000 conversations into those buckets with multi-label assignment. The analysis identified search across documentation as the assistant's biggest weakness. Mintlify then surfaced LLM-categorized conversations in the assistant insights dashboard for documentation owners and shipped UX fixes such as previous conversation threads, anchored links, mobile chat behavior, and streaming spacing.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  End_user["End user"] --> Mintlify_assistant["Mintlify assistant"]
  Mintlify_assistant --> Feedback_control["Feedback control"]
  Feedback_control --> Server_feedback_handler["Server feedback handler"]
  Feedback_control --> ClickHouse_feedback_store["ClickHouse feedback store"]
  Mintlify_assistant --> Conversation_thread_store["Conversation thread store"]
  Conversation_thread_store --> Server_feedback_handler
  Conversation_thread_store --> Backfill_script["Backfill script"]
  Server_feedback_handler --> ClickHouse_conversation_analytics["ClickHouse conversation analytics"]
  ClickHouse_feedback_store --> ClickHouse_conversation_analytics
  Backfill_script --> ClickHouse_conversation_analytics
  ClickHouse_conversation_analytics --> Manual_review["Manual review"]
  Manual_review --> Feedback_taxonomy["Feedback taxonomy"]
  Feedback_taxonomy --> LLM_feedback_classifier["LLM feedback classifier"]
  LLM_feedback_classifier --> Assistant_insights_dashboard["Assistant insights dashboard"]
  Assistant_insights_dashboard --> Documentation_owners["Documentation owners"]
  Mintlify_assistant --> Assistant_UI_improvements["Assistant UI improvements"]
  Assistant_UI_improvements --> End_user

Components

Data flow

A user interacts with the AI documentation assistant and can leave thumbs-down feedback on assistant messages. The server receives the feedback event, pushes the full conversation thread into ClickHouse, and historical feedback conversations are backfilled from PSQL. The team queries negative-feedback conversations, manually reads 100 threads to create eight categories, runs an LLM classifier over 1,000 sampled conversations, and writes categorized insights into the dashboard so documentation owners can inspect what customers are confused about.

Why this architecture

The central fix was not a model change; it was rebuilding evaluation data so feedback could be interpreted in context. Manual review produced meaningful categories, while an LLM scaled classification to 1,000 conversations. Surfacing categorized conversations to documentation owners closed the loop from user feedback to product and docs improvement.

"With that in place, we could finally query conversations with negative feedback.

[Exploring the dataset]"

Models used: LLM feedback classifier · Claude Sonnet 4.5 · AI-powered documentation assistant

Scale

• The investigation was scoped to one week.
  “it wasn’t performing the way we wanted, so we carved out a week to figure out where it fell short and how to improve it.”article, stated
• The human taxonomy came from about 100 negative conversation threads.
  “We read through about 100 negative conversation threads and created eight categories for the different types of feedback.”article, stated
• The LLM classification sample contained 1,000 conversations.
  “We then took a random sample of 1,000 conversations and used an LLM to classify each one into those eight buckets.”article, stated
• The taxonomy had eight feedback categories.
  “We read through about 100 negative conversation threads and created eight categories for the different types of feedback.”article, stated
• The model upgrade happened in mid October and did not materially change feedback patterns.
  “our model upgrade in mid October (to Sonnet 4.5) did not have a major impact and that feedback is fairly consistent across customers.”article, stated

Alternatives considered (roads not taken)

Analyze thumbs-down events without full conversation context

Rejected because: Feedback events alone did not map back to original conversations, so the team could not understand what went wrong.

[Exploring the dataset]"

— article, stated

Query conversation threads directly from PSQL

Rejected because: The PSQL setup stored threads in a way that made direct querying impossible.

"Our PSQL setup also stored threads in a way that made direct querying impossible.

To fix this, we updated the server"

Manual-only analysis of negative feedback

Rejected because: Manual reading established the taxonomy, but LLM classification was needed to scale the analysis to 1,000 conversations.

Assume a model upgrade would fix assistant quality

Rejected because: Feedback over time suggested the Sonnet 4.5 upgrade had no major impact, pointing to search and system issues instead.

Patterns

observability-tracing (operations)

Mintlify rebuilt feedback observability so thumbs-down events are connected to complete conversation threads in ClickHouse.

eval-driven-development (quality)

The team used negative feedback conversations, human taxonomy creation, and LLM classification to diagnose assistant weaknesses before shipping improvements.

llm-as-judge (quality)

An LLM classified 1,000 feedback conversations into human-defined failure buckets.

data-quality-loop (quality)

Categorized conversations are surfaced to documentation owners so user confusion can inform docs and assistant improvements.

rag-retrieval (knowledge-architecture)

The assistant's main weakness was search across documentation, making retrieval quality the diagnosed bottleneck.

[Takeaways and improvements]"

— article, stated

batch-offline-processing (operations)

A backfill script migrated historical feedback messages from PSQL to ClickHouse so analysis could cover past conversations.

Tradeoffs

Evaluation data architecture

Chose push full conversation threads to ClickHouse over store isolated feedback events only.

Rationale: Understanding failures required seeing complete conversation context alongside feedback.

[Exploring the dataset]"

— article, stated

Hybrid taxonomy workflow

Chose manual category design plus LLM classification over manual-only or LLM-only analysis.

Rationale: Humans created nuanced failure categories from 100 threads, while the LLM scaled classification to 1,000 conversations.

Multi-label classification

Chose allow threads to fall into more than one bucket over force exactly one failure category per conversation.

Rationale: Feedback conversations can contain multiple failure modes, so multi-label classification better preserves diagnostic detail.

Search-first diagnosis

Chose focus on documentation search quality over attribute poor feedback to model quality or response generation.

Rationale: The analysis showed search was the main weakness while non-search response quality looked strong.

[Takeaways and improvements]"

— article, stated

UX improvements alongside LLM diagnostics

Chose conversation history, anchored links, mobile behavior, and streaming spacing fixes over only changing retrieval or model logic.

Rationale: Assistant quality includes the surrounding interaction experience, not just answer text.

Gotchas (what bit them)

Feedback events without conversation context were not actionable.

Root cause: ClickHouse stored feedback events, but they were not mapped to the original conversation thread.

Resolution: Push full conversation threads to ClickHouse when feedback arrives.

Operational storage was not analysis-ready.

Root cause: Conversation threads lived in PSQL in a structure that could not be queried directly for feedback analysis.

Resolution: Backfill feedback messages into ClickHouse and use ClickHouse as the analytics layer.

"Our PSQL setup also stored threads in a way that made direct querying impossible.

To fix this, we updated the server"

Search quality, not answer generation, was the primary weakness.

Root cause: The assistant could generate good responses when search found relevant docs, but search failed to retrieve what users needed.

Resolution: Prioritize documentation search quality rather than assuming the generation model was the bottleneck.

[Takeaways and improvements]"

— article, stated

Model upgrades did not explain feedback trends.

Root cause: Assistant feedback stayed fairly consistent across customers and over time after the Sonnet 4.5 upgrade.

Resolution: Use feedback taxonomy and search diagnosis instead of treating model upgrades as the default fix.

Some questions were impossible for documentation search to answer.

Root cause: Users sometimes asked for actions or private context that could never be answered from documentation.

Resolution: Separate couldNotFindResult from assistantNeededContext in the taxonomy so retrieval failures are not mixed with impossible requests.

Evaluation

Mintlify evaluated the assistant by reconstructing feedback data, manually reading 100 negative threads to create an eight-bucket taxonomy, and using an LLM to classify a random sample of 1,000 conversations. They compared feedback over time and by subdomain, found no meaningful pattern from those cuts, and identified search across documentation as the main weakness.

Metrics: 100 negative conversation threads manually reviewed · eight feedback categories · 1,000 conversations LLM-classified · multi-label category assignment · feedback over time · assistant usage by subdomain · search across docs identified as biggest weakness

Outcomes

• Mintlify rebuilt the pipeline so feedback events map to full conversation threads.
• The team created an eight-category negative-feedback taxonomy from about 100 manual reviews.
• An LLM classified 1,000 sampled conversations into the taxonomy.
• The analysis identified documentation search as the assistant's biggest weakness.
• Mintlify expanded dashboard insights with LLM-categorized conversations and shipped multiple UX improvements.

This case is a strong example of LLMOps plumbing: the important work was not a clever prompt, but making production feedback analyzable and turning it into a dashboard loop for documentation owners. It is less useful as evidence about the assistant's retrieval architecture, which the source does not describe in detail.

"With that in place, we could finally query conversations with negative feedback.

[Exploring the dataset]"