Evaluation-Driven Refactoring: How W&B Improved Their LLM Documentation Assistant Through Systematic Testing

weights_&_biases · Tech · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM documentation assistant reliability and latency

Weights & Biases needed to refactor Wandbot, a RAG documentation assistant, because document parsing, retrieval speed, and redundant LLM calls made the production assistant too slow and inefficient.

Constraints:

• The fetched article source is unavailable, so this extraction relies on the ZenML summary.
• Wandbot served multiple client applications including Slack, Discord, and Zendesk.
• The refactor needed to improve latency without sacrificing answer correctness.
• The system had document ingestion and parsing issues affecting retrieval quality.
• Evaluation coverage had to catch regressions during a substantial architecture refactor.

Architecture

W&B refactored Wandbot's RAG pipeline by moving vector storage from FAISS to ChromaDB, optimizing chunk sizes, introducing LangChain Expression Language for asynchronous processing, and parallelizing API calls. The team drove the refactor with approximately 50 evaluations so changes to parsing, retrieval, and response synthesis could be judged against correctness and latency rather than intuition.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  documentation-corpus["Documentation corpus"] --> parsing-and-chunking-pipeline["Parsing and chunking pipeline"]
  parsing-and-chunking-pipeline --> chromadb-vector-store["ChromaDB vector store"]
  chromadb-vector-store --> rag-response-synthesizer["RAG response synthesizer"]
  rag-response-synthesizer --> client-application-surfaces["Client application surfaces"]
  rag-response-synthesizer --> evaluation-suite["Evaluation suite"]

Components

Data flow

Documentation is parsed, chunked, embedded, and stored in a vector database. User questions from Slack, Discord, Zendesk, or other client surfaces trigger retrieval, response synthesis, and evaluation of correctness and speed. Refactor changes were tested against the evaluation set before being accepted.

Why this architecture

The refactor targeted latency and efficiency bottlenecks while using evaluations to protect correctness. Moving vector storage, optimizing chunks, and parallelizing/asynchronizing calls addressed the known slow retrieval and redundant LLM-call problems in the RAG pipeline.

Models used: Wandbot · RAG · FAISS · ChromaDB · LangChain Expression Language · LLM-based response synthesis

Scale

• The refactor used approximately 50 evaluations.
  “The refactoring effort, which involved approximately 50 evaluations and significant debugging”summary, stated
• Correctness improved from 72% to 81%.
  “improved answer correctness from 72% to 81%”summary, stated
• Latency dropped 84%, from about 492 seconds to about 80 seconds per response.
  “reducing response latency by 84% (from ~492 seconds to ~80 seconds per response).”summary, stated

Alternatives considered (roads not taken)

Keep the original FAISS-based RAG pipeline.

Rejected because: The existing pipeline suffered from parsing quality issues, slow retrieval, and redundant LLM calls.

Refactor without systematic evaluation.

Rejected because: The team needed to verify that latency improvements did not damage answer correctness.

Patterns

rag-retrieval (documentation assistant)

Wandbot retrieves W&B documentation from a vector database and synthesizes answers with an LLM.

eval-driven-development (refactoring control)

The team used about 50 evaluations to compare correctness and latency through the refactor.

latency-optimization (production assistant)

Asynchronous processing and parallel API calls reduced response latency substantially.

retrieval-ranking (chunking and vector retrieval)

Chunk-size optimization and vector-store migration were used to improve retrieval quality and speed.

Tradeoffs

Migrate vector storage.

Chose ChromaDB over FAISS.

Rationale: The migration was part of a broader effort to improve retrieval speed and production maintainability.

Spend effort on evaluation before and during refactor.

Chose Approximately 50 evaluations over Performance tuning by intuition alone.

Rationale: The team needed correctness and latency evidence before accepting architectural changes.

Gotchas (what bit them)

Refactoring a RAG assistant exposed hidden complexity.

Root cause: Parsing, retrieval, and LLM-call inefficiencies interacted in ways that were not solved by a single code change.

Resolution: The team used systematic evaluation and debugging across vector storage, chunking, async processing, and parallel calls.

Document parsing problems reduced answer quality.

Root cause: The ingestion pipeline produced low-quality parsed content for retrieval.

Resolution: The refactor addressed the parsing and ingestion pipeline as part of the broader RAG redesign.

Redundant LLM calls hurt response latency.

Root cause: The original response-generation path involved iterative, redundant calls.

Resolution: W&B parallelized API calls and used asynchronous processing.

Evaluation

W&B used systematic evaluation-driven refactoring, with about 50 evaluations and significant debugging, to measure answer correctness and response latency before accepting the new Wandbot architecture.

Metrics: Answer correctness · Response latency · Evaluation count · Retrieval speed · Document parsing quality

Outcomes

• Answer correctness improved from 72% to 81%.
• Response latency dropped from about 492 seconds to about 80 seconds.
• The RAG assistant moved to ChromaDB, async processing, optimized chunks, and parallel API calls.

This is summary-only due to missing source cache, but the summary gives concrete before/after metrics and evaluation count.