Domain Adaptation of LLMs for Enterprise Use Through Multi-Task Fine-Tuning

wix · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise domain adaptation for customer-service LLMs

Wix needed a single customized LLM to handle multiple Wix-specific customer-service tasks, but common customization techniques such as prompt engineering, RAG, and task-specific fine-tuning were too costly, latent, hallucination-prone, or too narrow for full domain adaptation.

Constraints:

A single domain model needed to handle Q&A, intent classification, sentiment, segmentation, summarization, and other Wix tasks.
Prompt engineering and RAG add organization context through long prompts, increasing cost and latency.
Task-specific fine-tuning does not provide cross-domain multitasking capability.
DAPT requires large domain-specific datasets and full-weight training resources.
Evaluation had to cover both knowledge and task capabilities.

Architecture

Wix combined multi-task supervised fine-tuning using adapters with domain-adaptive full-weight fine-tuning. The team organized the work around training data, modeling, and evaluation; built a custom Wix Q&A benchmark from customer-service live chats and FAQs; used LLM-as-a-judge prompts created by domain experts; added task benchmarks for intent, segmentation, summarization, and sentiment; synthetically generated Q&As and reading-comprehension tasks from organizational data; and trained/evaluated LLaMA2 7B on AWS P5 GPU infrastructure.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  wix-domain-data["Wix domain data"] --> synthetic-data-generation["Synthetic data generation"]
  synthetic-data-generation --> sft-adapter-training["SFT adapter training"]
  synthetic-data-generation --> dapt-full-weight-training["DAPT full-weight training"]
  aws-p5-training-machine["AWS P5 training machine"] --> sft-adapter-training["SFT adapter training"]
  aws-p5-training-machine --> dapt-full-weight-training["DAPT full-weight training"]
  sft-adapter-training --> customized-wix-llm["Customized Wix LLM"]
  dapt-full-weight-training --> customized-wix-llm["Customized Wix LLM"]
  customized-wix-llm --> q-a-judge-benchmark["Q&A judge benchmark"]
  customized-wix-llm --> task-capability-benchmarks["Task capability benchmarks"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Wix domain data`	Customer-service chats, FAQs, documentation, internal reports, and labeled NLP project data.	`Synthetic data generation`
`Synthetic data generation`	Creates Q&A and reading-comprehension examples to expand limited manual data.	`SFT adapter training`, `DAPT full-weight training`
`SFT adapter training`	Multi-task supervised fine-tuning for instructive domain tasks.	`Customized Wix LLM`
`DAPT full-weight training`	Domain-adaptive pretraining/fine-tuning over domain-specific and public data.	`Customized Wix LLM`
`AWS P5 training machine`	High-power GPU infrastructure for full-scale fine-tuning and high-rank LoRA experiments.	`DAPT full-weight training`, `SFT adapter training`
`Customized Wix LLM`	Domain-adapted LLaMA2 7B-style model for Wix task families.	`Q&A judge benchmark`, `Task capability benchmarks`
`Q&A judge benchmark`	LLM-as-judge evaluation comparing generated answers to ground truth answers.	`Customized Wix LLM`
`Task capability benchmarks`	Domain-specific tasks such as intent, segmentation, summarization, and sentiment.	`Customized Wix LLM`

Data flow

Wix gathered domain data from customer-service chats, FAQs, knowledge-base articles, support chats, technical documentation, internal reports, and existing labeled NLP projects. That data fed synthetic Q&A/reading-comprehension generation, SFT adapters, and DAPT/full-weight fine-tuning. Custom Q&A and task-capability benchmarks then evaluated the customized model with expert-built LLM-as-judge prompts.

“To achieve full domain adaptation of an LLM at Wix, we applied multi-task supervised fine-tuning (SFT) using adapters and domain adaptation using full weights fine-tuning (DAPT).”article, stated

“To estimate the knowledge of the LLM we built a custom Wix Question and Answers (Q&A) dataset.”article, stated

Why this architecture

Wix chose full domain adaptation because easier techniques handle context or single tasks but do not create one model that can multitask across Wix-specific knowledge and task families. The combination of SFT, DAPT, synthetic data, and custom benchmarks reflects the need to adapt knowledge, task ability, and evaluation together.

“Full domain adaptation means that a model can tackle multiple domain-specific tasks at the same time.”article, stated

“The 3 fundamental components of every data science project are training data, modeling and evaluation.”article, stated

Models used: LLaMA2 7B · LoRA adapters · LLM-as-a-judge prompt · SFT · DAPT

Scale

Wix increased the domain-specific data ratio target to 2% versus 0.00000001% in the pretrained model.
“increasing the ratio of domain-specific data in the LLM’s training to 2% as opposed to the 0.00000001% in the pre-trained model.”article, stated
The infrastructure used one high-power AWS P5 GPU for full-scale fine-tuning and LoRA experiments on LLaMA2 7B.
“one high-power GPU running on AWS P5 that allowed us to experiment with full-scale fine-tuning and LoRA with high ranks on LLaMa2 7b.”article, stated

Alternatives considered (roads not taken)

Prompt engineering and RAG

Rejected because: They require long context-heavy prompts, increasing cost and latency, and still lack full multitasking domain adaptation.

“Prompt engineering and RAG require detailed prompts that include a lot of organization context. Longer prompts mean more tokens which increases cost.”article, stated

Task-specific fine-tuning

Rejected because: It improves one task but does not achieve cross-domain multitasking capability.

“Some AI engineers also fine-tune LLMs to improve performance on a specific task such as classifying user intents. However, these techniques are limited when trying to achieve full domain adaptation”article, stated

Use proprietary LLMs for domain adaptation

Rejected because: Wix says proprietary models are not a good fit because their training recipes are not public.

“Proprietary LLMs are not a good fit for this since their training recipes are not publicly available..”article, stated

Patterns

`fine-tuning-adaptation` (model-ops)

Wix used SFT adapters and DAPT full-weight fine-tuning for full domain adaptation.

“we applied multi-task supervised fine-tuning (SFT) using adapters and domain adaptation using full weights fine-tuning (DAPT).”article, stated

`synthetic-data-generation` (data)

The team generated synthetic Q&As and reading-comprehension tasks from organizational data to compensate for limited manual data.

“we synthetically generated Q&As using organizational data such as knowledge base articles, customer support chats, technical documentation, and internal reports.”article, stated

`llm-as-judge` (evaluation)

A custom judge prompt compared generated answers to ground-truth Wix Q&A answers.

“The “judge” is a prompt that compares answers suggested by an LLM to the ground-truth one.”article, stated

`eval-driven-development` (evaluation)

Wix built dedicated Q&A and task-capability benchmarks before judging model customization success.

“Having a solid, reliable metric is a must! Otherwise you are just shooting in the air.”article, stated

Tradeoffs

Pursue full domain adaptation

Chose Multi-task SFT plus DAPT over Prompt engineering, RAG, or single-task fine-tuning alone.

Rationale: The target was one model for a family of Wix-specific tasks, not a chain of one-off prompt/context tricks.

“it’s worth having a customized model for a family of tasks such as code generation, Q&A, classification, knowledge extraction, and creative content generation.”article, stated

Use synthetic data

Chose Synthetic Q&As and reading-comprehension tasks over Only manually created data.

Rationale: Manual data was very limited, so synthetic data expanded the training/evaluation set.

“The amount of manually created data was still very limited. Therefore, in order to increase the dataset, we synthetically generated Q&As”article, stated

Choose models already strong on the domain

Chose Base LLMs evaluated against custom benchmarks over Fine-tuning from scratch by default.

Rationale: If a model already contains relevant domain knowledge, fewer domain-specific tokens may be needed.

“Good performance may indicate that it already holds some domain information.”article, stated

Gotchas (what bit them)

Easy customization methods do not create full domain adaptation.

Root cause: Prompt engineering, RAG, and task fine-tuning handle context or one task at a time.

Resolution: Combine multi-task SFT, DAPT, data development, and domain benchmarks.

“Multi-task supervised fine-tuning combined with full-weight fine-tuning allow better domain adaptation, but are limited to specific LLM families and require significant development efforts and extensive training datasets.”article, stated

Long domain prompts can hurt both cost and accuracy.

Root cause: Prompt/RAG approaches add many tokens and detailed guidelines, which increases cost and may overfit or harm accuracy.

Resolution: Use a smaller domain-fine-tuned model when multitask domain behavior is required.

“too detailed guidelines in the prompt can cause overfitting and harm accuracy.”article, stated

Domain adaptation needs public/common data as well as domain data.

Root cause: Fine-tuning only on domain data can degrade common-knowledge performance.

Resolution: Train with public data to maintain pretrained general capability.

“in order to maintain a pre-trained LLM’s performance on common knowledge, the model must also train on public data.”article, stated

Evaluation

Wix used a custom Q&A benchmark with LLM-as-a-judge and domain-specific task benchmarks for intent classification, segmentation, summarization, and sentiment analysis.

Metrics: Q&A answer quality judged against ground truth · Intent classification · Customer segmentation · Custom domain summarization · Sentiment analysis · Benchmark performance of candidate base models

“For task capabilities estimation, we used common, domain-specific, text-based learning tasks.”article, stated

“In Wix’s case, these included customer’s intent classification, customer segmentation, custom domain summarization, and sentiment analysis.”article, stated

Outcomes

Defined a full-domain-adaptation strategy for Wix-specific multitask LLM use.
Built custom Q&A and task-capability benchmarks.
Created synthetic Q&A and reading-comprehension data from Wix organizational sources.
Established that customized models are worthwhile for families of knowledge-based Wix tasks.

The source is an overview post and does not publish final production traffic; concrete scale details include one AWS P5 training machine and a target domain-data ratio of 2%.

“Through our journey , we realized that it’s worth having a customized model for a family of tasks”article, stated

“The main tasks that truly require customization in our field are knowledge-based tasks such as Q&A with/without context.”article, stated