Fine-tuning LLMs for Toxic Speech Classification in Gaming

large_gaming_company · Media & Entertainment · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: online gaming trust and safety moderation

A large gaming company needed an English-language toxic speech detector that could automate a manual moderation process by first identifying toxic language in voice and text excerpts and then classifying the toxicity into customer-defined categories.

Constraints:

• the customer initially supplied only about 100 labeled samples
• the target labels were customer-defined toxicity categories such as profanity or abusive language
• training a custom language model from scratch was not viable under cost and time constraints
• the production solution needed to be scalable and easy to maintain
• the initial two-stage PoC created monitoring, cost, and inference-speed maintenance problems

Architecture

AWS GAIIC first ran a PoC by fine-tuning Hugging Face BERTweet-family models for toxic language classification. The PoC tested three BERTweet models, augmented the small customer-labeled dataset with mapped Jigsaw Toxicity examples, and used an 80/20 train-test split. The best PoC was a two-stage flow: binary toxic/not-toxic classification followed by fine-grained customer toxicity classification. For production, AWS ProServe MLDT requested about 5,000 more labeled examples, combined toxic and non-toxic labels into a single classification head, fine-tuned one-stage BERTweet models, and selected the one-stage bertweet-base-offensive model as the production tradeoff.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  game-voice-and-text-excerpts[Game voice and text excerpts]
  customer-labeled-dataset[Customer labeled dataset]
  public-toxicity-augmentation-data[Public toxicity augmentation data]
  sagemaker-fine-tuning-experiments[SageMaker fine-tuning experiments]
  bertweet-model-candidates[BERTweet model candidates]
  two-stage-poc-classifier[Two-stage PoC classifier]
  one-stage-bertweet-toxicity-classifier[One-stage BERTweet toxicity classifier]
  hugging-face-transformers-trainer[Hugging Face Transformers trainer]
  game-voice-and-text-excerpts --> customer-labeled-dataset
  customer-labeled-dataset --> public-toxicity-augmentation-data
  customer-labeled-dataset --> sagemaker-fine-tuning-experiments
  public-toxicity-augmentation-data --> sagemaker-fine-tuning-experiments
  bertweet-model-candidates --> sagemaker-fine-tuning-experiments
  sagemaker-fine-tuning-experiments --> two-stage-poc-classifier
  sagemaker-fine-tuning-experiments --> one-stage-bertweet-toxicity-classifier
  hugging-face-transformers-trainer --> one-stage-bertweet-toxicity-classifier
  one-stage-bertweet-toxicity-classifier --> game-voice-and-text-excerpts

Components

Data flow

Game voice and text excerpts are labeled into customer-defined toxic-language categories. The PoC augments the customer samples with mapped public Jigsaw Toxicity data, trains and validates BERTweet models in SageMaker notebooks, and compares binary plus fine-grained two-stage metrics. The production path adds about 5,000 more customer labels, combines non-toxic and toxic categories into one multi-label classification head, fine-tunes Hugging Face models with train and validation datasets, and deploys the selected one-stage model for at-scale toxic speech classification.

Why this architecture

AWS chose transfer learning on BERTweet-family language models because the customer lacked enough labeled data for training from scratch, game chat resembled tweet-length toxic-language data, and production requirements favored a maintainable one-stage classifier over the higher-cost two-stage PoC.

Models used: vinai/bertweet-base · cardiffnlp/bertweet-base-offensive · cardiffnlp/bertweet-base-hate

Scale

• The gaming industry context involved an estimated user base of more than 3 billion worldwide.
  “The video gaming industry has an estimated user base of over 3 billion worldwide1.”article, stated
• AWS initially received about 100 labeled customer samples, far below the roughly 1,000 samples recommended for fine-tuning.
  “AWS received about 100 samples of labeled data from the customer, which is a lot less than the 1,000 samples recommended for fine-tuning an LLM”article, stated
• The BERTweet base model was pre-trained on 850 million English tweets.
  “The bertweet-base model uses the preceding pre-training procedure from the RoBERTa study to pre-train the original BERT architecture using 850 million English tweets.”article, stated
• The offensive and hate variants used 14,100 offensive tweets and 19,600 hate-speech tweets respectively.
  “the base BertTweet FM and is further pre-trained on 14,100 annotated tweets that were deemed as offensive7”article, stated
• The expanded production dataset added about 5,000 labeled samples, split as 3,000 non-toxic and 2,000 toxic.
  “AWS ProServe MLDT received approximately 5,000 more labeled data samples, 3,000 being non-toxic and 2,000 being toxic”article, stated

Alternatives considered (roads not taken)

train a custom language model from scratch

Rejected because: cost and time constraints made full custom training unviable with the available labeled data.

use a two-stage binary-plus-fine-grained classifier in production

Rejected because: it doubled monitoring, increased cost, slowed inference, and complicated retraining timing.

"Increased costs of running two models as opposed to one.

• The speed of inference slows because inference goes through two models."

— article, stated

fine-tune only on the initial customer-provided labels

Rejected because: the customer initially had too few labels, so the PoC augmented data and production requested more customer samples.

Patterns

fine-tuning-adaptation (model-operations)

AWS fine-tuned pre-trained BERTweet models on customer toxicity categories rather than training a toxic-language model from scratch.

eval-driven-development (evaluation)

The team selected and revised the model architecture using validation metrics such as precision, recall, F1, AUC, and 80/20 train-test splits.

cost-aware-routing (serving)

The production team collapsed a two-model PoC into a one-model classifier to reduce monitoring, cost, and inference latency.

Tradeoffs

Use transfer learning with BERTweet rather than training from scratch.

Chose fine-tuning pre-trained tweet-oriented language models over building a custom game-vocabulary model from scratch.

Rationale: The customer had too little labeled data and too little time/cost budget for full custom training.

Use tweet-pretrained models for game-chat toxicity.

Chose BERTweet-family foundation models over generic language models without social-text pretraining.

Rationale: Tweet length and user diversity were considered similar to online game chat toxicity inputs.

Productionize a one-stage model despite slightly lower precision.

Chose single one-stage bertweet-base-offensive classifier over higher-scoring two-stage classifier.

Rationale: The one-stage design improved cost and maintainability while decreasing precision by only three percentage points.

Gotchas (what bit them)

The customer had far too few labeled examples for straightforward model training.

Root cause: Only about 100 labeled samples were available at first, while fine-tuning recommendations were closer to 1,000 samples.

Resolution: AWS used transfer learning from BERTweet-family models and augmented the PoC with mapped public Jigsaw toxicity data.

A custom model trained only on game vocabulary would have been more tailored but impractical.

Root cause: Classical NLP classifiers are costly to train and need a large vocabulary corpus for accurate predictions.

Resolution: The team used transfer learning from pre-trained foundation models instead of training from scratch.

The two-stage PoC architecture was harder to productionize.

Root cause: Two models required separate monitoring, could retrain on different schedules, cost more, and slowed inference.

Resolution: AWS ProServe MLDT converted the design into a single multi-class classifier with an added non-toxic class.

Evaluation

AWS evaluated BERTweet-family toxic-speech classifiers with 80/20 train-test splits and tracked precision, recall, F1, and AUC for binary, fine-grained, two-stage, and one-stage variants.

Metrics: binary PoC precision .92, recall .90, F1 .91, AUC .92 · fine-grained PoC precision .81, recall .80, F1 .81, AUC .89 · one-stage bertweet-base-offensive precision .88, recall .83, F1 .86, AUC .89 · two-stage bertweet-base-offensive precision .91, recall .90, F1 .90, AUC .92 · one-stage precision decreased by 3 percentage points while improving cost and maintenance

Outcomes

• AWS GAIIC built a PoC toxic-language detector by fine-tuning BERTweet-family models.
• AWS ProServe MLDT converted the PoC from a two-stage model architecture into a one-stage production architecture.
• The selected one-stage bertweet-base-offensive model reached precision .88, recall .83, F1 .86, and AUC .89.
• The one-stage model reduced cost and maintenance complexity while decreasing precision by only 3%.
• The productionized model was intended to automate detection of inappropriate player interactions at scale.

The case is framed around online gaming interactions in an industry with over 3 billion users worldwide; the production model used about 5,000 additional labeled samples beyond the PoC to support a scalable moderation classifier.