Building a RAG-Based Documentation Chatbot: Lessons from Fiddler's LLMOps Journey

fiddler · Tech · 2023 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: RAG documentation chatbot with LLMOps monitoring

Fiddler needed a documentation chatbot that could answer product questions from docs using GPT-3.5 while dealing with natural-language query ambiguity, context-window limits, retrieval quality, hallucinations, feedback collection, conversational memory, and trust-building UI.

Constraints:

• Users ask the same topic in diverse structures, synonyms, and colloquial expressions
• Pronouns and references require conversational context
• LLMs can only process limited context, so long documents must be chunked
• A single retrieval may miss relevant documents
• GPT-3.5 can hallucinate technical answers
• User feedback can be skewed toward negative experiences

Architecture

Fiddler built a GPT-3.5 RAG documentation chatbot using LangChain as the orchestration framework and Fiddler LLM Observability to evaluate and monitor prompts, responses, and knowledge-base improvements. The system processes and rewrites queries, chunks documents with metadata/continuity statements, performs multiple retrievals, ranks document relevance, iterates prompts, collects thumbs/comment feedback, stores embeddings for queries/responses/documents, adds caveats to reduce hallucinations, streams responses, and maintains conversational memory.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> fiddler-chatbot[Fiddler Chatbot]
  fiddler-chatbot --> langchain-pipeline[LangChain Pipeline]
  langchain-pipeline --> question-processor[Question Processor]
  question-processor --> retriever[Retriever]
  documentation-knowledge-base[Documentation Knowledge Base] --> document-chunker[Document Chunker]
  document-chunker --> embedding-store[Embedding Store]
  embedding-store --> retriever
  retriever --> gpt-model[GPT Model]
  gpt-model --> fiddler-chatbot
  fiddler-chatbot --> streaming-ui[Streaming UI]
  fiddler-chatbot --> feedback-collector[Feedback Collector]
  langchain-pipeline --> conversational-memory[Conversational Memory]
  embedding-store --> llm-observability[LLM Observability]
  feedback-collector --> llm-observability
  llm-observability --> fiddler-chatbot

Components

Data flow

User questions enter LangChain-based preprocessing for tokenization, normalization, context tracking, referential resolution, and optional LLM question generation. Original and processed queries can retrieve multiple candidate document sets. Documents are chunked with metadata so related chunks stay close in embedding space. Ranked retrieved context is passed to GPT-3.5, responses stream to the UI, feedback and embeddings are stored, and observability monitors prompts, responses, hallucinations, and knowledge-base gaps.

Why this architecture

Fiddler used RAG because documentation answers needed external, current knowledge rather than model memory alone. LangChain provided reusable pipeline pieces for question preprocessing, retrieval, and memory, while observability closed the loop on hallucinations, prompt behavior, feedback, and knowledge-base gaps. Multiple retrievals and chunking addressed the two biggest retrieval risks: ambiguous queries and long documents.

Models used: OpenAI GPT-3.5 · LLMs · machine learning models for intent

Scale

• The Fiddler guide presents ten lessons from building the RAG chatbot.
  “fetched_at: "2026-07-04T05:27:09.844Z" handler: "article" --- # 10 Lessons from Developing an AI Chatbot Using Retrieval-Augmented Generation | Fiddler AI ## Introduction”article, stated
• The chatbot was built around GPT-3.5 and Fiddler documentation.
  “to help Fiddler users easily find answers from our documentation. We used OpenAI’s GPT-3.5 and augmented it with RAG to build the Fiddler Chatbot. With the help of Fiddler LLM Observability solutions, we have continuously evaluated and monitored the chatbot’s responses,”article, stated

Alternatives considered (roads not taken)

Use only the processed/rephrased query for retrieval

Rejected because: Sometimes the original query retrieved more relevant documents.

Pass entire documents to the LLM

Rejected because: Context windows limit how much text can be processed and long documents need coherent chunks.

Static block-format answers

Rejected because: Static answers felt disjointed, while streaming increased perceived conversational trust.

Patterns

rag-retrieval (documentation-qa)

Fiddler retrieves documentation chunks and feeds them to GPT-3.5 for grounded answers.

query-rewriting (retrieval-preprocessing)

LangChain question generation and preprocessing resolve ambiguity before retrieval, while original queries are also considered.

retrieval-ranking (multi-retrieval)

The chatbot runs multiple retrievals and ranks top documents/chunks across results.

context-compaction (document-management)

Long documents are chunked into context-window-sized pieces with continuity metadata.

observability-tracing (llmops)

Fiddler monitors prompts, responses, embeddings, feedback, and hallucination patterns through LLM Observability.

Tradeoffs

Use LangChain plumbing

Chose LangChain for RAG orchestration over custom-building every retrieval, memory, and preprocessing component.

Rationale: LangChain let developers focus on chatbot quality rather than infrastructure.

Use multiple retrievals

Chose original and processed query retrieval paths over single best query rewriting path.

Rationale: Different query forms can retrieve different relevant evidence, especially for ambiguous language.

Stream responses

Chose real-time typing-like response display over static block responses.

Rationale: Streaming made the bot feel more conversational and increased user trust.

Gotchas (what bit them)

Query rewriting can hurt retrieval.

Root cause: A processed version of a question may lose terms or framing that matched relevant docs.

Resolution: Run multiple retrievals over original and processed query forms.

Chunking can break document coherence.

Root cause: Chunks are smaller than full documents, and related content may be separated.

Resolution: Add metadata or continuity statements linking chunks.

Hallucinations exposed documentation gaps.

Root cause: The bot interpreted LLM as local linear model because the knowledge base lacked clarifying LLM/LLMOps docs.

Resolution: Fiddler added caveats and richer docs to the knowledge repository.

User feedback can be negatively skewed.

Root cause: Satisfied users are less likely to leave detailed feedback than dissatisfied users.

Resolution: Offer multiple lightweight and detailed feedback mechanisms.

Evaluation

Fiddler evaluated and improved the chatbot through LLM Observability, prompt/response monitoring, user feedback, embeddings analysis, hallucination inspection, caveat additions, prompt iteration, and knowledge-base refinement.

Metrics: user prompts · chatbot responses · feedback thumbs/comments · hallucination instances · retrieval relevance · embedding alignment · knowledge-base gaps · user trust/engagement signals

Outcomes

• Fiddler deployed a GPT-3.5 RAG documentation chatbot for users.
• LangChain accelerated integration of knowledge sources, preprocessing, memory, and retrieval experimentation.
• The team identified and addressed query ambiguity, chunking, multi-retrieval, prompt iteration, feedback, hallucination, UI, and memory lessons.
• Fiddler used its own observability tooling to monitor and improve chatbot behavior.
• Hallucination review led to knowledge-base caveats and new LLM/LLMOps documentation.

The article is a lessons-learned guide rather than a numeric deployment report; the concrete scope is a GPT-3.5 RAG chatbot over Fiddler documentation with ten documented LLMOps lessons.