Dynamic Knowledge and Instruction RAG System for Production Chatbots

wix · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Adaptive website chatbots

Wix needed AI Site-Chat to answer site visitors with business-specific preferences and unwritten owner knowledge that static RAG documents and fixed system prompts cannot capture or update easily.

Constraints:

• Business owners need to teach chatbots preferred responses without technical work.
• Some important business knowledge exists only in owners' minds, not in site content.
• Traditional RAG knowledge bases can become stale unless manually updated or reindexed.
• Static system prompts cannot adapt behavior to new owner feedback.
• Direct LLM text classification was not treated as efficient or interpretable enough for feedback routing.

Architecture

Wix AI Site-Chat combines feedback classification with Dynamic Domain Knowledge and Instruction RAG. Owner feedback is first converted into yes/no/don't-know feature answers by LLM calls, then a CatBoost classifier maps those features to the correct feedback class. The system creates class-specific RAG documents and updates the knowledge base and prompts through DDKI-RAG, so new owner feedback can become retrievable domain knowledge or dynamic instructions for future chatbot answers.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  site-owner-feedback[Site Owner Feedback] --> llm-feature-extractor[LLM feature extractor]
  llm-feature-extractor --> catboost-classifier[CatBoost classifier]
  catboost-classifier --> class-specific-rag-document[Class-specific RAG document]
  class-specific-rag-document --> ddki-rag-layer[DDKI RAG layer]
  ddki-rag-layer --> ai-site-chat[AI Site Chat]

Components

Data flow

A site owner interacts with AI Site-Chat and provides feedback. The system asks predefined yes/no/don't-know questions to extract features from the feedback, feeds those features into CatBoost for class selection, creates a RAG document for that class, and updates the DDKI-RAG knowledge/prompt layer. Future visitor questions then retrieve this owner-provided knowledge or instruction and generate answers aligned with the business context.

Why this architecture

Wix separates feedback understanding from final chatbot generation because LLMs are more useful as feature extractors than direct classifiers in this design. DDKI-RAG then solves the static-RAG problem by turning owner feedback into dynamic knowledge and prompt updates, capturing information such as preferences or unwritten facts that were not present on the website.

Models used: LLM · CatBoost · DDKI-RAG · AI Site-Chat

Scale

• The article frames the system for Wix websites and site-owner feedback, but does not provide deployment traffic numbers.
  “This chatbot helps visitors interact with websites more effectively.”article, stated

Alternatives considered (roads not taken)

Use LLMs directly for feedback text classification.

Rejected because: Wix says LLMs are better at precise yes/no feature extraction than direct classification.

Use conventional static RAG.

Rejected because: Static knowledge bases do not automatically update from feedback and require manual updates or reindexing.

Keep fixed system prompts.

Rejected because: Static prompts produce generic responses and cannot adapt dynamically to user or owner needs.

Patterns

rag-retrieval (retrieval)

DDKI-RAG updates the knowledge base and prompt context from owner feedback so future answers retrieve business-specific instructions and knowledge.

structured-output-contract (data)

LLM feature extraction is constrained to yes/no/don't-know answers that feed a classifier.

data-quality-loop (data)

Site-owner feedback becomes new knowledge or instructions, closing a loop from observed bad answers to improved chatbot behavior.

Tradeoffs

Feedback classification architecture

Chose LLM feature extraction plus CatBoost classification. over Direct LLM classification..

Rationale: Yes/no features are more interpretable and efficient for CatBoost, while LLMs handle language understanding.

Knowledge update strategy

Chose Dynamic DDKI-RAG updates. over Manual reindexing and static prompts..

Rationale: Owner feedback needs to affect knowledge and behavior without time-consuming manual updates.

Gotchas (what bit them)

Important business knowledge may not exist in the website corpus.

Root cause: Some knowledge exists only in the owner's mind or is intended only for chatbot conversations.

Resolution: Wix captures owner feedback and inserts it into dynamic domain knowledge.

Traditional RAG can serve stale or irrelevant knowledge.

Root cause: The knowledge base does not update automatically after new feedback.

Resolution: DDKI-RAG dynamically updates the knowledge base and prompts from feedback.

Fixed prompts make chatbot behavior generic.

Root cause: Traditional RAG prompts are the same regardless of context or owner preference.

Resolution: Wix uses DDKI-RAG to adapt prompts dynamically.

Evaluation

The article explains the architecture and expected behavior but does not report rigorous production metrics; evaluation is mainly design rationale for classification and adaptive feedback.

Outcomes

• Introduced a Site Owner's Feedback mechanism for AI Site-Chat.
• Designed a hybrid LLM plus CatBoost feedback classifier.
• Defined DDKI-RAG for dynamic knowledge and prompt updates.
• Allowed owner feedback and unwritten knowledge to shape future chatbot answers.

The source does not provide traffic or deployment-volume metrics; it describes a production chatbot feature and architecture.