LLMOps Evolution: Scaling Wandbot from Monolith to Production-Ready Microservices

weights_&_biases · Tech · 2023 · source Agentic: no · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production RAG documentation assistant

Weights & Biases needed to take Wandbot from a monolithic Discord/Slack documentation assistant into a maintainable production RAG system with shared services, reliable model behavior, multilingual support, persistent conversation history, cost controls, and evaluation.

Constraints:

Discord and Slack deployments duplicated code and infrastructure with only minor configuration differences.
Unsynchronized deployments created bugs and inconsistencies across platforms.
Duplicated resources such as vector stores inflated operational cost.
New features like conversation history increased complexity in the monolith.
The assistant needed to retrieve documentation, code examples, and notebook content across multiple formats.
Production reliability required fallback behavior when GPT-4 was unavailable.

Architecture

W&B refactored Wandbot into microservices with separate ingestion, chat, database, and API modules. Ingestion syncs repositories, parses Markdown, notebooks, and code, chunks documents, embeds them with OpenAI ada-002, caches embeddings in SQLite, and stores FAISS indexes as versioned W&B artifacts. The chat service moved from LangChain to LlamaIndex, uses Cohere reranking, supports multilingual responses, and falls back from GPT-4 to GPT-3.5-turbo. The database stores conversation history, feedback, cached query results, and periodic backups to W&B Tables, while a central API exposes chat, feedback, history, and Q&A endpoints to clients.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Ingestion module"] --> B["Embedding cache"]
  A --> C["FAISS vector store"]
  A --> D["W&B artifact store"]
  C --> E["Cohere reranker"]
  C --> F["Chat module"]
  E --> F
  F --> G["Primary GPT-4 model"]
  F --> H["Fallback GPT-3.5 model"]
  F --> I["Database module"]
  I --> D
  J["Client applications"] --> K["API module"]
  K --> F
  K --> I

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Ingestion module`	Syncs repositories, parses Markdown, notebooks, and code, creates embeddings, and builds the FAISS vector store.	`FAISS vector store`, `Embedding cache`, `W&B artifact store`
`FAISS vector store`	Stores embedded document chunks and metadata for retrieval.	`Ingestion module`, `Chat module`, `Cohere reranker`
`Embedding cache`	Caches OpenAI ada-002 embedding calls in SQLite to reduce redundant model calls.	`Ingestion module`
`W&B artifact store`	Versions parsed documents, vector stores, ingestion reports, and database backups.	`Ingestion module`, `Database module`
`Chat module`	Handles query retrieval, reranking, response synthesis, multilingual prompting, and model fallback through LlamaIndex.	`FAISS vector store`, `Cohere reranker`, `Primary GPT-4 model`, `Fallback GPT-3.5 model`, `Database module`, `API module`
`Cohere reranker`	Reranks retrieved chunks before they are passed to the LLM.	`FAISS vector store`, `Chat module`
`Primary GPT-4 model`	Primary response-generation model for Wandbot.	`Chat module`
`Fallback GPT-3.5 model`	Backup LLM used when GPT-4 is unavailable.	`Chat module`
`Database module`	Stores conversation history, feedback, cached query results, and periodic backups.	`Chat module`, `API module`, `W&B artifact store`
`API module`	Exposes question-answer, chat-thread, query, and feedback endpoints to client applications.	`Chat module`, `Database module`, `Client applications`
`Client applications`	Discord, Slack, and future interfaces that call the centralized API.	`API module`

Data flow

Documentation and code are synced from GitHub, parsed into chunks with metadata, embedded, cached, and indexed into FAISS. User queries flow through the API into the chat module, which retrieves relevant chunks, reranks them with Cohere, synthesizes grounded responses through LlamaIndex and GPT models, stores conversation history and feedback in SQLite, backs up to W&B Tables, and exposes results back to Discord, Slack, or future client applications.

4 supporting quotes

“The document parsing pipeline begins with syncing the latest updates from GitHub repositories.”summary, stated

“The module creates embedding vectors for document chunks and indexes these documents into a FAISS vector store with relevant metadata.”summary, stated

“A notable integration is Cohere's rerank-v2 endpoint, which allows Wandbot to sift through retriever results more effectively.”summary, stated

“The database module serves as Wandbot's memory bank, storing conversational history, providing conversational context for future queries”summary, stated

Why this architecture

The microservices split directly addresses the monolith's duplication, maintenance, and scaling problems: ingestion, chat, database, and API concerns can evolve independently, clients share one backend, retrieval quality can be tuned separately with reranking, and reliability/cost controls can be handled at the model and cache layers.

4 supporting quotes

“The team resolved to transition to a microservices-oriented architecture, breaking down the bot into smaller, manageable components.”summary, stated

“centralizing core services and models for use across applications.”summary, stated

“This migration decision reflects the evolving landscape of LLM frameworks and the importance of choosing tools that provide the right level of abstraction for production use cases.”summary, stated

“The centralized API approach provides several advantages: loose coupling between frontend applications and backend services”summary, stated

Models used: OpenAI ada-002 · GPT-4 · GPT-3.5-turbo · Cohere rerank-v2

Scale

Wandbot supported separate Discord and Slack applications before the refactor.
“The original Wandbot was deployed as a monolithic application with separate instances for Discord and Slack”summary, stated
The assistant added English and Japanese support.
“The new architecture enabled multilingual support (English and Japanese)”summary, stated
SQLite backups were scheduled every 10 minutes to W&B Tables.
“The team implements periodic backups (every 10 minutes) to W&B Tables”summary, stated
The API module exposes four named endpoints.
“with key endpoints including /question_answer for storing Q&A pairs, /chat_thread for retrieving conversation history, /query as the primary chat endpoint, and /feedback for storing user feedback.”summary, stated

Alternatives considered (roads not taken)

Continue separate monolithic Discord and Slack bots

Rejected because: That duplicated code, resources, deployments, vector stores, and maintenance effort.

“The Discord and Slack applications were deployed separately, resulting in duplicated code with only minor configuration differences.”summary, stated

“The operational costs were inflated because the team was essentially running two distinct bots”summary, stated

Stay on LangChain for chat orchestration

Rejected because: The team wanted more control over retrieval methods, response synthesis, and customizations.

“The team migrated from LangChain to LlamaIndex, which gave them better control over underlying functionality including retrieval methods, response synthesis pipeline, and other customizations.”summary, stated

Use GPT-4 as the only response model

Rejected because: Production reliability required a backup model when GPT-4 had downtime.

“If the primary model (GPT-4) experiences downtime, the system seamlessly switches to a backup LLM (GPT-3.5-turbo).”summary, stated

Expose ingestion, chat, and database services directly to clients

Rejected because: A centralized API provided loose coupling, abstraction, independent scaling, and better security.

“improved developer productivity through abstraction, independent horizontal scaling of individual API services, and enhanced security by avoiding direct exposure of core modules.”summary, stated

Patterns

`rag-retrieval` (retrieval)

Wandbot indexes documentation, notebooks, and code chunks into FAISS and retrieves them as context for documentation answers.

“The Ingestion Module represents one of the most critical components in any RAG system”summary, stated

“The module creates embedding vectors for document chunks and indexes these documents into a FAISS vector store with relevant metadata.”summary, stated

`retrieval-ranking` (retrieval)

Cohere rerank-v2 is used to improve relevance among retriever results before response synthesis.

“Cohere's rerank-v2 endpoint, which allows Wandbot to sift through retriever results more effectively.”summary, stated

`fallback-degradation` (reliability)

The chat module falls back from GPT-4 to GPT-3.5-turbo when the primary model is unavailable.

“This failover mechanism is managed within the LlamaIndex service context and adds a layer of resilience against potential downtimes”summary, stated

`semantic-caching` (cost)

Wandbot caches embeddings and LLM query results to reduce redundant model calls and operational cost.

“A crucial operational optimization is the use of SQLite caching (part of LangChain) to minimize redundant model calls”summary, stated

“Caching of LLM query results at the database level reduces the need for repetitive queries, cutting down operational costs.”summary, stated

`observability-tracing` (operations)

Ingestion reports, W&B artifacts, backups, feedback, and evaluation reports provide visibility into data, retrieval, and response behavior.

“The team also generates comprehensive reports outlining GitHub repository revision numbers, the volume of documents ingested, and artifacts comprising parsed documents and vector stores.”summary, stated

`eval-driven-development` (evaluation)

Wandbot is evaluated manually and automatically for retrieval accuracy and response relevance across a diverse test set.

“The team conducted both manual and automated evaluation of Wandbot, measuring retrieval accuracy and response relevance across a custom test set with diverse query types.”summary, stated

Tradeoffs

Refactor into microservices

Chose Ingestion, chat, database, and API modules over One monolithic bot per platform.

Rationale: The split reduced duplication, improved maintainability, and let services change independently.

“The modular design also enabled dedicated APIs for seamless integration with existing and potential future platforms, and allowed independent modification of each service”summary, stated

Use SQLite for the database module

Chose Embedded SQLite with backups to W&B Tables over Running a separate database server.

Rationale: SQLite fit the operational goal of a serverless, embeddable, portable conversation-history store.

“The choice of SQLite as the database was driven by its serverless architecture (no need for a separate database server)”summary, stated

Migrate from LangChain to LlamaIndex

Chose LlamaIndex for retrieval and response-synthesis control over Continuing with LangChain's abstractions for the chat module.

Rationale: Production use required more control over retrieval methods, synthesis, and service-context fallback behavior.

“gave them better control over underlying functionality including retrieval methods, response synthesis pipeline, and other customizations.”summary, stated

Cache embeddings and query results

Chose SQLite caching for embeddings and database-level LLM query caching over Recomputing embeddings and responses for repeated content.

Rationale: Caching reduced redundant model calls and operational costs.

“which is essential for cost and operational efficiency.”summary, stated

Gotchas (what bit them)

Duplicated platform-specific bots drift and become costly.

Root cause: Discord and Slack instances were deployed separately with duplicate code, vector stores, resources, and unsynchronized releases.

Resolution: Centralize ingestion, chat, database, and API services behind shared modules.

“This often resulted in bugs and inconsistencies due to unsynchronized deployments between the two platforms.”summary, stated

“organize the system into distinct components for ingestion, chat, and database services while centralizing core services and models for use across applications.”summary, stated

RAG ingestion needs format-aware parsing, not just text splitting.

Root cause: Wandbot had to process Markdown, Python code, and Jupyter Notebooks, each requiring different chunking logic.

Resolution: Use MarkdownNodeParser, nbconvert, and CST-based code parsing to create metadata-rich chunks.

“handling the parsing and processing of raw documentation in diverse formats including Markdown, Python code, and Jupyter Notebooks.”summary, stated

“Code blocks receive special treatment, being parsed and chunked using Concrete Syntax Trees (CST)”summary, stated

Primary LLM downtime can break an otherwise healthy assistant.

Root cause: The chat workflow originally depended on a primary model for answer generation.

Resolution: Add LlamaIndex-managed fallback from GPT-4 to GPT-3.5-turbo.

“If the primary model (GPT-4) experiences downtime, the system seamlessly switches to a backup LLM (GPT-3.5-turbo).”summary, stated

Evaluation of RAG systems has to inspect components and the whole pipeline.

Root cause: Retrieval quality and response relevance can fail independently, and whole-system answer quality is not enough to diagnose the cause.

Resolution: Use manual and automated evaluation over diverse query types and detailed evaluation reports.

“They acknowledge that evaluating RAG systems is complex, requiring examination of each component both individually and as a whole.”summary, stated

Evaluation

W&B evaluated Wandbot manually and automatically using a custom diverse query set, measuring retrieval accuracy and response relevance while acknowledging that RAG evaluation requires component-level and whole-system inspection.

Metrics: Retrieval accuracy · Response relevance · Diverse query-type coverage · Ingestion revision and document-volume reporting · User feedback persistence

3 supporting quotes

“measuring retrieval accuracy and response relevance across a custom test set with diverse query types.”summary, stated

“recognizing that comprehensive LLM evaluation is a substantial undertaking in its own right.”summary, stated

“persisting user feedback for continuous improvement.”summary, stated

Outcomes

Wandbot moved from duplicated monolithic Slack and Discord deployments to a shared microservices architecture.
The new architecture supported English and Japanese interactions.
GPT-4 fallback to GPT-3.5-turbo improved model-availability resilience.
Embedding and query caching reduced redundant model calls and cost.
Manual and automated evaluations measured retrieval accuracy and response relevance.

This is summary-only because the source fetch failed terminally. Quantified details available in the summary include Discord/Slack deployments, English/Japanese support, 10-minute backups, four API endpoints, and named model/service choices.