Building a Delicate Text Detection System for Content Safety

grammarly · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI content safety and delicate-text detection

Grammarly needed a way to detect emotionally charged or potentially triggering text that existing toxicity detectors miss, because harmful user or model interactions are broader than profanity or explicit abuse.

Constraints:

• delicate text is subjective and may overlap with, but is not identical to, toxic text
• the benchmark contains sensitive categories such as self-harm, mental health, hate speech, sexuality, and threats
• annotation requires expert judgment and risk-level labeling
• the artifacts require privacy, security, sensitivity, legal, and compliance due diligence before use

Architecture

Grammarly created DeTexD as both a training dataset and benchmark. The team sourced candidate text through domain specification and delicate-keyword matching, annotated it with expert linguists using a two-step process, fine-tuned a RoBERTa-based classifier, and evaluated it against existing toxicity-oriented methods on the DeTexD Benchmark.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  candidate-text-sources[Candidate text sources] --> keyword-dictionary[Keyword dictionary]
  candidate-text-sources --> annotation-process[Annotation process]
  keyword-dictionary --> annotation-process
  annotation-process --> detexd-training-dataset[DeTexD Training dataset]
  annotation-process --> detexd-benchmark-dataset[DeTexD Benchmark dataset]
  detexd-training-dataset --> roberta-based-classifier[RoBERTa-based classifier]
  roberta-based-classifier --> detexd-benchmark-dataset
  external-toxicity-methods[External toxicity methods] --> detexd-benchmark-dataset

Components

Data flow

Candidate text came from targeted domains and delicate-keyword matching. Expert linguists labeled whether each text was delicate and rated risk. The labeled DeTexD Training dataset was used to fine-tune a RoBERTa-based classifier, and the DeTexD Benchmark measured how well that model and other toxicity detectors captured delicate text.

Why this architecture

Grammarly treated delicate text as a separate detection problem instead of stretching toxicity classifiers to cover it. A dedicated dataset, expert annotation, and task-specific classifier were necessary because broad toxicity definitions either missed sensitive coverage or misclassified offensive-but-not-delicate examples.

Models used: RoBERTa-based classifier · Google Perspective API · OpenAI moderation API

Scale

• The DeTexD Training dataset contains 40,000 samples.
  “The result was the labeled DeTexD Training dataset of 40,000 samples and the labeled DeTexD Benchmark dataset of 1,023 paragraphs.”article, stated
• The DeTexD Benchmark contains 1,023 paragraphs.
  “The result was the labeled DeTexD Training dataset of 40,000 samples and the labeled DeTexD Benchmark dataset of 1,023 paragraphs.”article, stated
• The baseline achieved 79.3 percent F1 on the benchmark.
  “79.3% Performance of various methods on the DeTexD Benchmark evaluation dataset. After receiving valuable feedback from reviewers, we calibrated optimal thresholds for F-score for several methods (marked with #).”article, stated

Alternatives considered (roads not taken)

Reuse toxic-text detection as the safety boundary

Rejected because: Delicate text can be non-toxic but still harmful, and toxicity detectors missed important coverage.

Rely on Google Perspective API or OpenAI moderation API

Rejected because: The reported benchmark scores were much lower than the DeTexD baseline for this task.

Use keyword matching alone

Rejected because: Keyword matching helped source candidates, but final labels required expert annotation and majority vote.

Patterns

safety-policy-enforcement (safety)

Grammarly broadened safety detection beyond toxicity by defining and detecting delicate text that could harm users or agents.

data-quality-loop (data)

The benchmark depended on targeted data sourcing, expert instructions, two-step annotation, and majority-vote labels.

fine-tuning-adaptation (modeling)

Grammarly fine-tuned a RoBERTa-based classifier on the DeTexD Training dataset for the delicate-text task.

eval-driven-development (evaluation)

The DeTexD Benchmark compared the dedicated classifier against canonical toxicity and moderation baselines.

Tradeoffs

Build a dedicated delicate-text benchmark

Chose DeTexD as a separate safety dataset and benchmark over using existing toxicity datasets as-is.

Rationale: Existing toxic-text definitions missed or mischaracterized delicate text.

Use keyword matching for candidate mining but expert annotation for labels

Chose keyword-based sourcing plus human linguistic judgment over fully automated keyword labeling.

Rationale: Keywords can identify candidate topics, but delicacy and risk require subjective expert labeling.

Publish artifacts with explicit caution

Chose public benchmark/model availability with due-diligence warnings over frictionless reuse as a drop-in production filter.

Rationale: The benchmark includes sensitive content and needs privacy, security, legal, and compliance review before operational use.

Gotchas (what bit them)

Toxicity detection is not a sufficient proxy for delicate text.

Root cause: Delicate text includes emotional or triggering topics that may not contain vulgar or explicitly offensive language.

Resolution: Define delicate text separately and build a task-specific benchmark and classifier.

Subjective safety categories are hard to annotate consistently.

Root cause: Delicate text depends on sensitive topics and risk judgments rather than a simple objective label.

Resolution: Use detailed annotation instructions, expert linguists, risk-level labeling, and majority vote.

Released safety datasets can themselves contain harmful sensitive content.

Root cause: A realistic benchmark includes uncensored examples across hate speech, violence, threats, self-harm, mental health, sexuality, and profanity.

Resolution: Warn users not to use the artifacts without privacy, security, sensitivity, legal, and compliance due diligence.

Evaluation

Grammarly evaluated the fine-tuned baseline and several toxicity methods on the DeTexD Benchmark using precision, recall, and F1 score.

Metrics: 79.3% F1 for the DeTexD baseline · 42.3% F1 for Google Perspective API · 31.1% F1 for OpenAI moderation API

Outcomes

• Grammarly produced the DeTexD Training and Benchmark datasets
• the RoBERTa-based baseline outperformed evaluated toxicity methods on DeTexD
• annotation guidelines, benchmark data, and baseline model were made publicly available

The key scale numbers are dataset and benchmark sizes: 40,000 training samples and 1,023 benchmark paragraphs.