Building Robust LLM Evaluation Frameworks: W&B's Evaluation-Driven Development Approach

weights_&_biases · Tech · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Evaluation-driven RAG documentation assistant development

Weights & Biases needed to improve Wandbot, its LLM-powered documentation assistant, by making automated evaluations align with human judgment while also fixing retrieval, ingestion, query understanding, and knowledge-base coverage problems.

Constraints:

• Default automated evaluation prompts did not correlate well with human assessment.
• Manual annotation was more accurate but too slow for iterative development.
• The RAG pipeline had parsing problems with Docusaurus-specific MarkdownX content.
• Some queries required knowledge outside the original documentation index.
• The assistant had to answer technical documentation and debugging questions, including code correctness.
• The reported evaluation dataset was relatively small, so broad production generalization remained a limitation.

Architecture

Wandbot v1.1 combines evaluation-driven iteration with a richer RAG pipeline. W&B curated manual annotations in Argilla, used GPT-4 as an evaluator with a custom W&B-support-expert prompt, and aligned auto-evaluation with human labels through few-shot examples. The production pipeline improved data ingestion by cleaning Docusaurus-specific Markdown artifacts before parsing, expanded the knowledge base, added query enhancement with mention stripping, Cohere language detection, Cohere intent classification, Instructor-based keyword/sub-query generation, and hybrid retrieval across FAISS semantic search, BM25 keyword search, and you.com web snippets. LlamaIndex and RAGAS metrics then compare model/pipeline variants.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-query[User query] --> query-enhancement-stage[Query enhancement stage]
  query-enhancement-stage --> cohere-language-detector[Cohere language detector]
  query-enhancement-stage --> cohere-intent-classifier[Cohere intent classifier]
  query-enhancement-stage --> instructor-query-enhancer[Instructor query enhancer]
  markdown-ingestion-cleaner[Markdown ingestion cleaner] --> knowledge-base-index[Knowledge base index]
  knowledge-base-index --> faiss-vector-retriever[FAISS vector retriever]
  knowledge-base-index --> bm25-retriever[BM25 retriever]
  faiss-vector-retriever --> hybrid-retriever[Hybrid retriever]
  bm25-retriever --> hybrid-retriever
  you-com-web-retriever[You.com web retriever] --> hybrid-retriever
  instructor-query-enhancer --> hybrid-retriever
  hybrid-retriever --> metadata-filtering-postprocessor[Metadata filtering postprocessor]
  metadata-filtering-postprocessor --> response-synthesis-prompt[Response synthesis prompt]
  instructor-query-enhancer --> response-synthesis-prompt
  response-synthesis-prompt --> auto-evaluation-framework[Auto evaluation framework]
  argilla-annotation-dataset[Argilla annotation dataset] --> auto-evaluation-framework

Components

Data flow

A user question enters Wandbot and is cleaned by removing bot/user mentions. The query enhancement stage detects language with Cohere, classifies intent with a fine-tuned Cohere model, and uses Instructor to add keywords and sub-queries. Those signals feed a hybrid retriever that combines FAISS semantic search, BM25 keyword search, and you.com web snippets over an expanded and cleaned knowledge base, then applies metadata filtering. The response synthesis prompt uses the retrieved contexts and query hints to answer. Separately, responses are evaluated against 98 curated QA pairs using GPT-4, customized LlamaIndex evaluators, and RAGAS metrics aligned with Argilla human annotations.

Why this architecture

W&B built both an evaluation stack and retrieval stack because evaluation quality and RAG quality were mutually limiting. Human labels were trustworthy but too slow, so GPT-4 evaluation had to be calibrated with Argilla-curated examples. Retrieval failures came from parser quality, missing corpus coverage, weak query understanding, and single-strategy retrieval, so the v1.1 pipeline added ingestion cleanup, knowledge-base expansion, query enhancement, and hybrid retrieval. The architecture is modular so each stage can be improved and measured independently.

Models used: GPT-4 · gpt-4-1106-preview · gpt-4-0613 · gpt-3.5-turbo-16k-0613 · Cohere language detection API · Fine-tuned Cohere classification model · Embedding model for FAISS/vector retrieval

Scale

• The auto-evaluation dataset contained 98 question-answer pairs.
  “The resulting evaluation dataset contained 98 Question-Answer pairs that served as reference answers for the auto-evaluation system.”summary, stated
• The evaluation compared four model configurations.
  “The team conducted comparative evaluations across four model configurations: gpt-3.5-turbo-16k-0613, gpt-4-0613, gpt-4-1106-preview, and gpt-4-1106-preview-v1.1”summary, stated
• Answer scoring used a three-point ordinal scale.
  “The scoring system used an ordinal scale where 1 indicated incorrect/unfaithful/irrelevant, 2 indicated ambiguous, and 3 indicated correct/faithful/relevant.”summary, stated
• The hybrid retriever combined three retrieval strategies.
  “The final hybrid retriever combined three retrieval strategies: FAISS Vectorstore for semantic similarity search, BM25 for keyword-based retrieval, and you.com for web search.”summary, stated
• The knowledge base was expanded with three additional source families: Fully Connected Reports, Weave Examples, and W&B SDK Tests.
  “include Fully Connected Reports, Weave Examples, and W&B SDK Tests, providing more diverse sources for retrieval.”summary, stated

Alternatives considered (roads not taken)

Use default automated evaluator prompts

Rejected because: Default prompts for correctness, faithfulness, and relevance did not align well with manual human assessments.

Rely on manual evaluation for every iteration

Rejected because: Manual assessments were accurate but too time-consuming and tedious for rapid pipeline iteration.

Keep the default MarkdownNodeParser path unchanged

Rejected because: Docusaurus-specific MarkdownX artifacts produced chunks that were too short or too long for effective retrieval.

Retrieve only from the original documentation index

Rejected because: Some user questions required reports, examples, SDK tests, or external web knowledge outside the initial docs.

Use a single retrieval strategy

Rejected because: The system needed semantic retrieval, keyword matching, and web snippets to handle diverse real-world documentation queries.

Patterns

eval-driven-development (quality)

W&B used curated human labels, GPT-4 auto-evaluation, LlamaIndex/RAGAS metrics, and model/pipeline comparisons to guide Wandbot v1.1 improvements.

llm-as-judge (evaluation)

The team used GPT-4 as a W&B support expert evaluator, calibrated by human annotations and few-shot examples.

rag-retrieval (retrieval)

Wandbot retrieves contexts from an expanded documentation corpus to answer W&B platform questions and debugging issues.

hybrid-search (retrieval)

The final retriever combines FAISS semantic search, BM25 keyword search, and you.com web search.

query-rewriting (retrieval)

The query enhancement stage removes mentions, detects language, classifies intent, and adds keywords/sub-queries before retrieval and synthesis.

data-quality-loop (data)

Manual annotation exposed Markdown parsing and missing-corpus issues, which fed data-ingestion cleanup and knowledge-base expansion.

Tradeoffs

Calibrate auto-evaluation with human labels

Chose GPT-4 evaluator plus Argilla-curated few-shot examples over Default evaluator prompts or repeated manual-only evaluation.

Rationale: This made evaluation fast enough for iteration while improving alignment with human judgment.

Use hybrid retrieval

Chose FAISS, BM25, and you.com web snippets over A single internal vector retriever.

Rationale: Different query types needed semantic matching, keyword matching, and sometimes external web knowledge.

Add query enhancement before retrieval

Chose Mention cleanup, language detection, intent classification, keywords, and sub-queries over Passing raw user messages directly into retrieval.

Rationale: Enhanced queries were more concise, relevant, multilingual-aware, and informative for retrieval and synthesis.

Clean Markdown before parsing

Chose Preprocess Docusaurus artifacts over Letting the default parser handle MarkdownX directly.

Rationale: Parser artifacts created poor chunk sizes, so cleaning improved the retrieval substrate.

Use multiple evaluation frameworks

Chose Customized LlamaIndex evaluators plus RAGAS metrics over A single metric family.

Rationale: Response quality and context quality needed different measurements.

Gotchas (what bit them)

Default auto-evaluation prompts did not match human judgment.

Root cause: Generic correctness, faithfulness, and relevance prompts were not calibrated to W&B support expectations.

Resolution: Curate human annotations in Argilla and add few-shot correct/incorrect examples to the GPT-4 evaluator prompt.

Manual evaluation was too slow for iterative development.

Root cause: Repeated human annotation did not scale to every pipeline change.

Resolution: Build a GPT-4 powered auto-evaluation framework aligned with curated annotations.

Docusaurus MarkdownX parsing degraded retrieval chunks.

Root cause: The default MarkdownNodeParser did not handle JavaScript components, plugins, Tabs, Frontmatter, and Admonitions.

Resolution: Handle these artifacts before parsing to produce more consistent chunks for the vector store.

The original index missed answerable queries.

Root cause: Relevant documents such as reports, examples, and SDK tests were outside the knowledge base.

Resolution: Expand the index with Fully Connected Reports, Weave Examples, and W&B SDK Tests.

The evaluation dataset was small enough to limit confidence in broad generalization.

Root cause: The reported benchmark used only 98 question-answer pairs.

Resolution: Treat improvements as validated on the benchmark but needing broader production validation.

Evaluation

W&B evaluated Wandbot with GPT-4 acting as a W&B support expert, Argilla-curated human/GPT-4 annotations, few-shot examples, customized LlamaIndex evaluators, RAGAS metrics, and comparative runs across GPT-3.5, GPT-4, GPT-4-1106-preview, and the v1.1 pipeline.

Metrics: Answer Correctness · Answer Factfulness / Faithfulness · Answer Relevancy · Answer Similarity · Context Precision · Context Recall · 1/2/3 ordinal correctness-faithfulness-relevance scores

Outcomes

• The v1.1 pipeline with gpt-4-1106-preview outperformed other configurations across most reported metrics.
• Answer Correctness, Answer Relevancy, and Context Recall improved compared with previous versions.
• Answer Faithfulness stayed comparatively grouped across models, suggesting earlier models could still align answers with provided context.
• Hybrid retrieval and expanded knowledge base were credited with improved Context Recall.

The evaluation used 98 QA pairs and compared four model configurations; the results were positive but still called for broader production validation.